Image processing method, image processing device, storage medium and electronic device

Abstract

The embodiments of the present disclosure disclose an image processing method, an image processing device, a storage medium, and an electronic device. The embodiments of the present disclosure extract a time sequence including original key point information from an original image sequence, obtain an audio sequence including acoustic features, and adjust at least one original key point information according to the audio sequence to obtain a target image sequence. Through the method of the embodiments of the present disclosure, the image can be adjusted relatively quickly and accurately, thereby improving the matching degree between the image and the audio and enhancing the viewing experience.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, an image processing device, a storage medium, and an electronic device. Background Art

[0002] With the continuous development of computer technology, the application field of image processing has become more and more extensive. For industries involving images, such as the film and television industry, the animation industry, etc., when post-processing the filmed or produced video, it may be necessary to adjust the image and / or audio in the video, which may cause images with the same timestamp, especially the lip shape of the characters in the image and the audio to not match. Therefore, for industries involving images, such as the film and television industry, the animation industry, etc., it is necessary to adjust the image more accurately to enhance the matching degree between the image and the audio. Summary of the invention

[0003] In view of this, the purpose of the embodiments of the present disclosure is to provide an image processing method, an image processing device, a storage medium and an electronic device for adjusting the image more quickly and accurately, thereby improving the matching degree between the image and the audio and enhancing the viewing experience.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0005] Acquire an original image sequence, where the original image sequence is a time sequence including original key point information;

[0006] Acquire an audio sequence, where the audio sequence is a time series including acoustic features;

[0007] The original key point information in the original image sequence is adjusted according to the audio sequence to form a target image sequence.

[0008] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0009] An image acquisition unit, used to acquire an original image sequence, wherein the original image sequence is a time sequence including original key point information;

[0010] An audio acquisition unit, used to acquire an audio sequence, where the audio sequence is a time series including acoustic features;

[0011] An adjustment unit is used to adjust the original key point information in the original image sequence according to the audio sequence to form a target image sequence.

[0012] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which computer program instructions are stored, wherein the computer program instructions implement the method as described in any one of the first aspects when executed by a processor.

[0013] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, and wherein the one or more computer program instructions are executed by the processor to implement a method as described in any one of the first aspects.

[0014] The disclosed embodiment extracts a time sequence including original key point information from the original image sequence, obtains an audio sequence including acoustic features, and adjusts at least one original key point information according to the audio sequence to obtain a target image sequence. The method of the disclosed embodiment can adjust the image relatively quickly and accurately, thereby improving the matching degree between the image and the audio and enhancing the viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0016] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure;

[0017] Figure 2 is a flow chart of a method for acquiring a target image sequence according to an exemplary embodiment of the present disclosure;

[0018] Figure 3 is a schematic diagram of acquiring a target image in an exemplary embodiment of the present disclosure;

[0019] Figure 4 is a schematic diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0020] Figure 5 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, known methods, processes, flows, components and circuits are not described in detail.

[0022] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0023] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0024] In the description of the present disclosure, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0025] For industries involving images, such as the film and television industry, the animation industry, etc., when post-processing the filmed or produced videos, it may be necessary to adjust the images and / or audio in the video. For example, due to a slip of the tongue by a voice actor during the audio recording process, a sentence in the audio is re-recorded later, which may result in images with the same timestamp, especially the lip shape of the characters in the image and the audio not matching. Therefore, for industries involving images, such as the film and television industry, the animation industry, etc., it is necessary to adjust the images more accurately to enhance the matching degree between the images and the audio.

[0026] In the embodiment of the present invention, a human face is used as an example for description, but those skilled in the art can easily understand that when the face is other types of faces, such as a cartoon character face, an animal face, etc., the method of this embodiment is also applicable.

[0027] Figure 1 FIG. 1 is a flowchart of an image processing method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method of this embodiment may include the following steps:

[0028] Step S100, obtaining an original image sequence.

[0029] In this embodiment, the original image sequence can be an image sequence obtained by taking an image acquisition device (such as a camera, a video camera, etc.), or an image sequence obtained by manual drawing, which is not specifically limited in this embodiment. The original image sequence includes original key point information, wherein the original key point information is used to characterize the information of key parts of the face that have a greater impact on expression changes, which can be specifically the contour shape, coordinates, etc. of the key parts, and the key parts can specifically be eyes, lips, etc. In this embodiment, the lip key information is selected as the original key point information, so that the server can perform face detection on each original image in the original image sequence, obtain the face area information of each original image, and determine the original key point information of each original image based on the face area information.

[0030] In this embodiment, face detection can be achieved by various existing image processing algorithms, such as reference template method, face rule method, feature sub-face method and sample recognition method. The acquired face region information can be represented by the data structure R (X, Y, W, H) of the face region. Among them, R (X, Y, W, H) defines a rectangular region including the main part of the face in the image, wherein X and Y define the coordinates of two different end points of the rectangular region, and W and H define the width and height of the rectangular region respectively.

[0031] In an optional implementation of the present embodiment, Dlib can be used to perform face detection and obtain lip key point information. Dlib is a C++ open source toolkit that includes machine learning algorithms. Dlib can identify the facial features and contours of a person through 68 key points. Among them, the contour of the lips can be defined by multiple key points. Thus, the server can extract the lip key point information from each original image based on Dlib, and determine the original key point information in the original image sequence based on the extracted lip key point information and the timestamp information of each original image.

[0032] Step S200, obtaining an audio sequence.

[0033] In this embodiment, the audio sequence may be an audio sequence recorded synchronously with the image sequence, or may be an audio sequence recorded later according to the image sequence, or may be an audio sequence obtained by converting each character (or word) in the predetermined text into speech according to the time axis order of the original image sequence, that is, at least one audio sequence may be an audio sequence recorded synchronously, recorded later, or processed later according to the predetermined text. The time axis correspondence between the image sequence and each audio sequence may be predetermined, or may be determined by various existing methods, such as by synchronizing the image sequence and the audio sequence in time axis by the method described in "Qi Chengming. Research and Implementation of Audio and Video Synchronization Problems. Harbin Institute of Technology. Master's Thesis 2009", thereby determining the time axis correspondence between the image sequence and each audio sequence, which is not specifically limited in this embodiment.

[0034] The audio sequence may specifically be a time series including acoustic features. The acoustic features may be used to characterize at least one of the audio signal strength and the audio signal frequency, wherein the audio signal strength may reflect the volume, and the audio signal frequency may reflect the pitch. According to the audio signal strength and the audio signal frequency, the server may effectively distinguish whether the audio signal at the current timestamp is a human voice or an environmental sound, thereby accurately adjusting at least one original image subsequently.

[0035] Step S300: adjusting original key point information in an original image sequence according to an audio sequence to form a target image sequence.

[0036] Figure 2 FIG. 1 is a flowchart of a method for acquiring a target image sequence according to an exemplary embodiment of the present disclosure. Figure 2 As shown, in an optional implementation of this embodiment, step S300 may include the following steps:

[0037] Step S310, obtaining lip features corresponding to the acoustic features.

[0038] In some embodiments, the server may obtain lip features corresponding to each acoustic feature. In some embodiments, it also includes: determining whether the audio signal of the current timestamp is a human voice, and if it is a human voice, obtaining the lip features corresponding to the acoustic feature. Among them, the way to determine whether the audio signal is a human voice includes: determining whether the frequency of the audio signal belongs to the frequency interval of the human voice, or determining whether the intensity of the audio signal belongs to the intensity interval of the human voice, etc., which is not limited in this embodiment.

[0039] Optionally, the lip features may include lip width and lip height. Usually, the intensity of human speech is positively correlated with the degree of opening of the human mouth. That is, the wider the mouth is opened, the stronger the audio signal is. Therefore, the server can determine the lip features corresponding to at least one acoustic feature of the human voice based on the correspondence between the predetermined acoustic features and the lip features.

[0040] Optionally, the mouth shape can also be used as a lip feature, thereby effectively improving the accuracy of subsequent adjustment of the original key point information. The server can use various existing methods to perform speech recognition on each audio sequence, for example, through the speech recognition system recorded in "Cui Tianyu. Research and Implementation of Speech Recognition System Based on HMM. Jilin University. 2016 Master's Thesis", to obtain the recognition result corresponding to the acoustic feature, and obtain the lip feature corresponding to the acoustic feature according to each recognition result. Among them, the recognition result is used to characterize characters or phonemes. Usually, characters or phonemes have a corresponding relationship with lip features, and the lip features of characters or phonemes with similar pronunciations are relatively close, for example, the lip features of "win" and "sound" are relatively close, so the server can determine the lip features corresponding to each recognition result according to the corresponding relationship between the recognition result and the lip feature. It is easy to understand that the server can also determine the lip features corresponding to the recognition result according to the language corresponding to the recognition result and the corresponding relationship between the recognition result and the lip feature corresponding to the language.

[0041] Optionally, the correspondence between the acoustic features and the lip features may also be adjusted according to the ratio of the size of the face in the original image to the size of the actual face.

[0042] Step S320: adjust the original key point information according to the lip features to obtain target key point information.

[0043] In some embodiments, the server may adjust each original key point information according to each lip feature to obtain target key point information corresponding to each original key point information.

[0044] In some embodiments, when the acoustic feature of any timestamp is a human voice, it can be considered that the mouth of the person in the original image corresponding to the timestamp needs to be opened to a certain extent, so the server can adjust the original key point information according to the lip features to obtain the target key point information.

[0045] For example, the lip features obtained by the server based on the acoustic features are 1.5 cm in width and 1 cm in height. The server can determine the center point of the person's mouth at the current timestamp based on the original key point information, and then adjust the coordinates of the key points used to represent the two lip corners, the upper end of the lip, and the lower end of the lip in the original key point information according to the coordinates of the center point, and adaptively adjust the coordinates of multiple key points adjacent to the above key points, so that the distance between the two lip corners meets 1.5 cm, and the distance between the upper end of the lip and the lower end of the lip meets 1 cm, thereby obtaining the target key point information.

[0046] For another example, if the mouth shape obtained by the server based on the acoustic features is shape 1, the server can determine the first center point of the person's mouth at the current timestamp based on the original key point information, determine the second center point based on shape 1, and overlap the second center point with the first center point, and then adjust the original key point information based on shape 1 to obtain the target key point information.

[0047] Optionally, the server may further determine whether to adjust the original key point information.

[0048] Step S3A, determining whether the difference between the original key point information and the lip feature is less than a first threshold.

[0049] The first threshold is used to determine whether the difference between the original key point information and the lip features is small. Specifically, it can be determined whether the difference between the distance between any two original key points in the original key point information and the distance between the corresponding two key points in the lip features is less than the first threshold, etc. This embodiment does not make specific limitations.

[0050] If the difference between the original key point information and the lip feature is less than the first threshold, the server may not adjust the original key point information; if the difference between the original key point information and the lip feature is greater than or equal to the first threshold, the server may execute step S320. In other words, step S3A may be executed before step S320.

[0051] Through step S3A, when the difference between the original key point information and the lip features is small, the server does not need to obtain the target key point information, thereby effectively reducing the amount of server calculations and improving the efficiency of key point adjustment.

[0052] Step S330: determining a key point sequence according to the target key point information.

[0053] Specifically, after adjusting the original key point information corresponding to at least one acoustic feature to obtain the corresponding at least one target key point information, the server may determine the key point sequence according to the at least one target key point information and the unadjusted original key point information.

[0054] Step S340: adjusting the original image sequence according to the key point sequence to obtain a target image sequence.

[0055] In this step, the server can replace the original key point information in at least one original image according to at least one target key point information based on the time axis correspondence between the key point sequence and the original image sequence to obtain at least one target image, thereby determining the target image sequence based on each target image and the original image corresponding to each acoustic feature when it is not a human voice.

[0056] In some embodiments, the acquisition of target key point information also integrates the emotion coefficient corresponding to the acoustic feature. As a method of acquiring key point information, specifically, step S300 also includes:

[0057] Step S3B, obtaining the emotion coefficient corresponding to the acoustic feature.

[0058] The emotion coefficient is used to characterize the intensity of emotion. Generally, the greater the audio signal strength and / or the higher the audio signal frequency, the stronger the emotion of the person when speaking, so the server can determine the emotion coefficient corresponding to the acoustic feature according to at least one of the audio signal strength and the audio signal frequency. Optionally, the server can determine the emotion coefficient corresponding to any acoustic feature of human voice according to the corresponding relationship between the acoustic feature and the emotion coefficient.

[0059] For example, if the audio signal strength is between 61-65 decibels, the emotion coefficient may be 1; if the audio signal strength is between 66-70 decibels, the emotion coefficient may be 1.5.

[0060] Among them, step S3B is performed before step S320. Specifically, step S310 may be performed first, and then step S3B; or step S3B may be performed first, and then step S310. As long as the emotion coefficient corresponding to the acoustic feature is obtained by step S3B before step S320, it belongs to the protection scope of the present disclosure.

[0061] At this time, step S320 includes: adjusting the original key point information according to the lip features or the lip features and the emotion coefficient to obtain the target key point information.

[0062] When adjusting the original key point information according to the lip features and the emotion coefficient, the server can adjust the original key point information according to the product of the lip features and the emotion coefficient. For example, if the acoustic feature of any timestamp is a human voice, and the emotion coefficient corresponding to the acoustic feature is 1.5, and the lip feature is a width of 1.5 cm and a height of 1 cm, then the server can determine that the product of the width and the emotion feature is 1.5*1.5=2.25, and the product of the height and the emotion feature is 1*1.5=1.5, thereby adjusting the original key point information according to the above two products to obtain the target key point information.

[0063] Through the above steps S3B and S320, the target key point information can more accurately reflect different emotions in the audio, further improving the accuracy of key point adjustment.

[0064] As another method of obtaining target key point information, specifically, step S310 also includes: obtaining the emotion coefficient corresponding to the acoustic feature. In this step, the server can obtain the emotion coefficient corresponding to the acoustic feature when it is a human voice, and then determine the corresponding lip feature according to the acoustic feature and the emotion coefficient based on the pre-trained classification model. In this step, the method of determining the emotion coefficient is similar to the method of determining the emotion coefficient in step S3B, and will not be repeated here.

[0065] In this embodiment, the classification model can be a decision tree, a neural network, a support vector machine (SVM, Support Vector Machine), etc., which is not specifically limited in this embodiment. Taking a neural network as an example, the full name of the neural network is an artificial neural network (ANN), which is an information processing model formed by interconnecting a large number of processing units. Common artificial neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), etc. ANN has the characteristics of nonlinearity (suitable for processing nonlinear information), non-limitation (that is, the overall behavior of a system depends on the interaction between processing units), very qualitative (that is, it has adaptive, self-organizing, self-learning capabilities, and can continuously self-learn in the process of processing information) and non-convexity (the activation function of the model has multiple extreme values, which makes the model have multiple relatively stable equilibrium states, so that the changes of the model are diverse), so it can be widely used in various fields for more accurate data prediction.

[0066] When people speak, the mouth shapes formed by characters or phonemes with similar pronunciations are relatively close, that is, the number of lip features is limited. The server can obtain category labels of multiple lip features in advance, and then convert the problem of predicting lip features into a classification problem.

[0067] The server can train the classification model based on historical data. The historical data may include multiple acoustic features and the emotion coefficient and category label corresponding to each acoustic feature. During the training of the classification model, the server can train the classification model with each acoustic feature and the corresponding emotion coefficient as input and the category label as output. After the classification model is trained, the server can use each acoustic feature and the corresponding emotion coefficient of the human voice as input, obtain the category label corresponding to each acoustic feature based on the trained (i.e., pre-trained) classification model, and then determine the corresponding lip feature according to the category label.

[0068] Through the above step S310, key point information can be obtained according to lip features and emotion coefficients through model prediction, thereby further improving the accuracy of key point adjustment.

[0069] In the above optional implementation, since the human voice is continuous, when the timestamp interval between two adjacent acoustic features is less than a predetermined threshold, it means that although there is a blank period between the two acoustic features, the lip feature is not interrupted due to this blank period. Therefore, the server can adjust the original key point information corresponding to the timestamp interval to the target key point information of the adjacent acoustic feature to enhance the continuity of the target image sequence.

[0070] Among them, the original key point information corresponding to the timestamp interval can be adjusted to the target key point information of the acoustic feature before the timestamp interval, and the original key point information corresponding to the timestamp interval can also be adjusted to the target key point information of the acoustic feature after the timestamp interval.

[0071] For example, a general video has 25 frames per second. The first word occupies frames 1 to 6, the second word occupies frames 7 to 13, and the third word occupies frames 16 to 23. There are three blank frames between the second and third words without any sound. Due to the inertia of lip features during speaking, the three blank frames in the middle also need to be processed. Accordingly, the lip features corresponding to the three blank frames in the middle are adjusted to be consistent with the lip features of the 13th frame.

[0072] Figure 3 FIG. 1 is a schematic diagram of obtaining a target image in an exemplary embodiment of the present disclosure. Figure 3As shown, image 31 is the original image with a timestamp of 0 minutes and 33 seconds in the original image sequence. The server can obtain the acoustic feature with a timestamp of 0 minutes and 33 seconds in the audio sequence. If the acoustic feature is a human voice, the server can obtain the lip feature and then determine whether the original key point information is less than the first threshold. When the original key point information is greater than or equal to the first threshold, the server can adjust the original key point information according to the lip feature to obtain the target key point information. Then, the original key point information in image 31 is replaced with the target key point information, thereby obtaining the target image with a timestamp of 0 minutes and 33 seconds, that is, image 32.

[0073] Optionally, after determining the target image sequence, since the target image sequence is obtained from the original image sequence, the server may use the time axis sequence of the original image sequence as the time axis sequence of the target image sequence. Thus, the server may synthesize the target image sequence and the at least one audio sequence according to the time axis correspondence between the target image sequence and the at least one audio sequence, thereby obtaining the target video clip.

[0074] This embodiment extracts a time sequence including original key point information from the original image sequence, obtains an audio sequence including acoustic features, and adjusts at least one original key point information according to the audio sequence to obtain a target image sequence. Through the method of this embodiment, the image can be adjusted relatively quickly and accurately, thereby improving the matching degree between the image and the audio and enhancing the viewing experience.

[0075] Figure 4 FIG. 1 is a schematic diagram of an image processing device according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the device of this embodiment includes an image acquisition unit 41, an audio acquisition unit 42, and an adjustment unit 43.

[0076] The image acquisition unit 41 is used to acquire an original image sequence, which is a time sequence including original key point information. The audio acquisition unit 42 is used to acquire an audio sequence, which is a time sequence including acoustic features. The adjustment unit 43 is used to adjust the original key point information in the original image sequence according to the audio sequence to form a target image sequence.

[0077] This embodiment extracts a time sequence including original key point information from the original image sequence, obtains an audio sequence including acoustic features, and adjusts at least one original key point information according to the audio sequence to obtain a target image sequence. Through the method of the embodiment of the present disclosure, the image can be adjusted relatively quickly and accurately, thereby improving the matching degree between the image and the audio and enhancing the viewing experience.

[0078] Figure 5is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. Figure 5 The electronic device shown is a general data processing device, which includes a general computer hardware structure, which includes at least a processor 51 and a memory 52. ​​The processor 51 and the memory 52 are connected via a bus 53. The memory 52 is suitable for storing instructions or programs executable by the processor 51. The processor 51 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 51 executes the commands stored in the memory 52, thereby executing the method flow of the embodiment of the present disclosure as described above to realize the processing of data and the control of other devices. The bus 53 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 54 and the display device and the input / output (I / O) device 55. The input / output (I / O) device 55 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output (I / O) device 55 is connected to the system through an input / output (I / O) controller 56.

[0079] The memory 52 may store software components such as an operating system, a communication module, an interaction module, and an application program. Each of the modules and applications described above corresponds to a set of executable program instructions that implement one or more functions and methods described in the disclosed embodiments.

[0080] The above-mentioned flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure describes various aspects of the present disclosure. It should be understood that each block of the flowchart and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that (via the processor of a computer or other programmable data processing device) instructions are created to implement the device for the function / action specified in the flowchart and / or block diagram block or block.

[0081] At the same time, as will be appreciated by those skilled in the art, various aspects of the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, aspects of the present disclosure may take the form of a computer program product implemented in one or more computer-readable media, the computer-readable media having a computer-readable program code implemented thereon.

[0082] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the embodiments of the present disclosure, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0083] A computer readable signal medium may include a propagated data signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0084] The computer program code for performing operations for various aspects of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, PHP, Python, etc.; and conventional procedural programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0085] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an original image sequence, where the original image sequence is a time sequence including original key point information; Acquire an audio sequence, wherein the audio sequence is a time sequence including acoustic features, and the audio sequence is an audio sequence recorded synchronously with the original image sequence, an audio sequence recorded later according to the original image sequence, or an audio sequence obtained by converting each word in a predetermined text into speech according to the time axis order of the original image sequence; Adjusting the original key point information in the original image sequence according to the audio sequence to form a target image sequence; The adjusting the original key point information in the original image sequence according to the audio sequence to form a target image sequence includes: Acquire lip features corresponding to the acoustic features, where the lip features include lip width and lip height; Adjusting the original key point information according to the lip features to obtain target key point information; Determine a key point sequence according to the target key point information, wherein the key point sequence is a time sequence including at least one target key point information; The original image sequence is adjusted according to the key point sequence to obtain the target image sequence.

2. The method according to claim 1, characterized in that The original key point information includes lip key point information; The obtaining of the original image sequence comprises: Performing face detection on original images in the original image sequence to obtain face region information of each original image; The lip key point information of each of the original images is obtained according to the facial region information.

3. The method according to claim 1, characterized in that The adjusting the original key point information in the original image sequence according to the audio sequence to form a target image sequence further includes: Before adjusting the original key point information according to the lip feature to obtain the target key point information, obtaining an emotion coefficient corresponding to the acoustic feature, wherein the emotion coefficient is used to represent the intensity of the emotion; Furthermore, adjusting the original key point information according to the lip features to obtain target key point information includes: The original key point information is adjusted according to the lip feature and the emotion coefficient to obtain the target key point information.

4. The method according to claim 1, characterized in that The adjusting the original key point information according to the lip feature to obtain the target key point information includes: When the time stamp interval between adjacent acoustic features is smaller than a predetermined threshold, the original key point information corresponding to the time stamp interval is adjusted to the target key point information of the adjacent acoustic features.

5. The method according to claim 1, characterized in that The acquiring of the lip feature corresponding to the acoustic feature comprises: Performing speech recognition on the audio sequence to obtain a recognition result corresponding to the acoustic feature, wherein the recognition result is used to represent a character or a phoneme; The lip features corresponding to the acoustic features are obtained according to the recognition results.

6. The method according to claim 1, characterized in that The acquiring of the lip feature corresponding to the acoustic feature comprises: Acquiring an emotion coefficient corresponding to the acoustic feature, where the emotion coefficient is used to represent the intensity of the emotion; Based on a pre-trained classification model, each of the lip features is determined according to the acoustic features and the emotion coefficient, and the classification model is obtained by training based on historical data.

7. The method according to claim 1, characterized in that The adjusting the original image sequence according to the key point sequence to obtain the target image sequence comprises: Replacing the original key point information in at least one original image in the original image sequence according to at least one target key point information to obtain a target image; The target image sequence is determined according to each of the target images.

8. The method according to any one of claims 1 to 7, characterized in that The acoustic feature is used to characterize at least one of an audio signal strength and an audio signal frequency.

9. An image processing device, characterized in that: The device comprises: An image acquisition unit, used to acquire an original image sequence, wherein the original image sequence is a time sequence including original key point information; an audio acquisition unit, configured to acquire an audio sequence, wherein the audio sequence is a time sequence including acoustic features, and the audio sequence is an audio sequence recorded synchronously with the original image sequence, an audio sequence recorded later according to the original image sequence, or an audio sequence obtained by converting each word in a predetermined text into speech according to the time axis order of the original image sequence; an adjusting unit, configured to adjust the original key point information in the original image sequence according to the audio sequence to form a target image sequence; Wherein, the adjustment unit is further used for: Acquire lip features corresponding to the acoustic features, where the lip features include lip width and lip height; Adjusting the original key point information according to the lip features to obtain target key point information; Determine a key point sequence according to the target key point information, wherein the key point sequence is a time sequence including at least one target key point information; The original image sequence is adjusted according to the key point sequence to obtain the target image sequence.

10. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 8 when executed by a processor.

11. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 8.

Images