Voice driving method and device of image and electronic equipment

Abstract

The invention provides an image voice driving method and device and electronic equipment, and the method comprises the steps: carrying out the three-dimensional face reconstruction of a target object according to at least one frame of first original image containing the target object in an original video, and obtaining a three-dimensional face model of the target object, after the target voice used for driving the second original image different from the first original image in the original video is obtained, the three-dimensional face model is directly driven through the target voice, and the target face image for the target object is rendered according to the driven three-dimensional face model. And replacing the original face image of the target object in the second original image with the target face image to obtain a driving result of the second original image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence applications, and more specifically to a method, device, and electronic device for voice-driven image processing. Background Art

[0002] With the continuous development of science and technology, artificial intelligence technology is increasingly used in fields such as film and television production, education and training, and game development. A voice-driven facial animation generation technology is proposed, which drives the original facial image through voice and generates a new facial image with the mouth shape synchronized with the voice to meet the animation processing needs of the corresponding field.

[0003] However, in the practical application of the above-mentioned speech-driven facial animation generation method, there is still the problem that the facial movements presented by the new facial images are unnatural (poor realism), which reduces the quality of facial animation. Summary of the Invention

[0004] In view of the above problems, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for driving an image through voice, the method comprising:

[0006] Acquire at least one first original image frame containing a target object in the original video;

[0007] Performing three-dimensional facial reconstruction on the target object based on the at least one frame of the first original image to obtain a three-dimensional facial model of the target object;

[0008] obtaining a target voice, where the target voice is used to drive a second original image in the original video, where the second original image is different from the first original image;

[0009] driving the three-dimensional facial model by the target voice, and rendering a target facial image for the target object based on the driven three-dimensional facial model;

[0010] The target facial image is used to replace the original facial image of the target object in the second original image, thereby obtaining a driving result for the second original image.

[0011] In a possible implementation, using the target facial image to replace the original facial image of the target object in the second original image to obtain a driving result for the second original image includes:

[0012] Pasting the target facial image to the facial position of the target object in the second original image to obtain an updated image;

[0013] If it is determined that the updated image meets the correction condition, the updated image is corrected to obtain a driving result for the second original image;

[0014] If it is determined that the updated image does not satisfy the correction condition, the updated image is determined as a result of driving the second original image.

[0015] In a possible implementation, the correction condition includes: the updated image contains a partial original facial image of the target object in the second original image;

[0016] The step of correcting the updated image to obtain a driving result for the second original image includes any one of the following implementations:

[0017] Using the non-face image in the updated image, correcting a portion of the original face image of the target object in the updated image to obtain a driving result for the second original image;

[0018] The image filling model is used to perform pixel filling on a portion of the original facial image of the target object where the updated image exists, to obtain a driving result for the second original image.

[0019] In one possible implementation, using the non-facial region image in the updated image to correct a portion of the original facial image of the target object in the updated image includes at least one of the following implementations:

[0020] Extracting background image data from the updated image, and filling the background image data into a target background area in the partial original facial image of the target object in the updated image;

[0021] Based on the body image data or image filling model of the target object in the updated image, pixel filling is performed on the target body area in the partial original facial image of the target object in the updated image.

[0022] In one possible implementation, the at least one first original image frame meets a three-dimensional facial modeling condition; and performing three-dimensional facial reconstruction on the target object based on the at least one first original image frame to obtain a three-dimensional facial model of the target object includes:

[0023] Extracting facial key point features and facial texture features of the target object from the at least one frame of the first original image;

[0024] A three-dimensional facial model of the target object is constructed using the facial key point features and the facial texture features.

[0025] In a possible implementation, the three-dimensional facial modeling condition includes: a frame of the first original image or a combination of multiple frames of the first original image that can enhance the complete facial features of the target object; the facial features include facial key point features and facial texture features of the target object;

[0026] The step of obtaining at least one frame of a first original image containing a target object in the original video includes:

[0027] Performing mouth movement tracking detection on each frame of the original image from the original video to obtain the mouth opening and closing angle of the target object in the original image containing the target object;

[0028] A first original image corresponding to the mouth opening and closing angle being less than an angle threshold is obtained.

[0029] In a possible implementation, driving the three-dimensional facial model by the target voice includes:

[0030] Determining a second original image driven by the target voice in the original video;

[0031] extracting target facial perspective data of the target object in the second original image;

[0032] According to the target voice and the target facial perspective data, the three-dimensional facial model is driven to perform corresponding actions to obtain the driven three-dimensional facial model of the target object.

[0033] In one possible implementation, the step of driving the three-dimensional facial model to perform corresponding actions based on the target voice and the target facial perspective data to obtain the driven three-dimensional facial model of the target object includes:

[0034] Determining, based on an original image and original speech of a target object in the original video, a correspondence between mouth movement data of the target object in the original image and original phoneme features of the original speech;

[0035] Extracting features of the target speech to obtain target phoneme features;

[0036] Determining target mouth movement data corresponding to the target phoneme feature based on the corresponding relationship;

[0037] The three-dimensional facial model is driven to perform corresponding mouth movements according to the target mouth movement data, and the three-dimensional facial model is driven to perform corresponding facial movements according to the target facial perspective data, thereby obtaining the driven three-dimensional facial model of the target object.

[0038] A second aspect of the present application provides a voice-driven device for an image, the device comprising:

[0039] A first original image acquisition module is used to acquire at least one first original image frame containing a target object in an original video;

[0040] a three-dimensional facial reconstruction module, configured to perform three-dimensional facial reconstruction on the target object based on the at least one frame of the first original image to obtain a three-dimensional facial model of the target object;

[0041] a target speech acquisition module, configured to acquire a target speech; the target speech is used to drive a second original image in the original video, the second original image being different from the first original image;

[0042] A driving module, configured to drive the three-dimensional facial model through the target voice;

[0043] A rendering module, configured to render a target facial image of the target object based on the driven three-dimensional facial model;

[0044] The replacement module is used to replace the original facial image of the target object in the second original image with the target facial image to obtain a driving result for the second original image.

[0045] A third aspect of the present application provides an electronic device, comprising: at least one communication element, at least one memory, and at least one processor, wherein:

[0046] The communication element is used to obtain the original video and the target voice;

[0047] The processor is configured to load and execute the plurality of computer instructions stored in the memory to implement the following steps:

[0048] Acquire at least one first original image frame containing a target object in the original video;

[0049] Performing three-dimensional facial reconstruction on the target object based on the at least one frame of the first original image to obtain a three-dimensional facial model of the target object;

[0050] obtaining a target voice, where the target voice is used to drive a second original image in the original video, where the second original image is different from the first original image;

[0051] driving the three-dimensional facial model by the target voice, and rendering a target facial image for the target object based on the driven three-dimensional facial model;

[0052] The target facial image is used to replace the original facial image of the target object in the second original image, thereby obtaining a driving result for the second original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0054] Figure 1 This is a flowchart of the voice-driven method for images proposed in Example 1 of the present application;

[0055] Figure 2 This is a flow chart of the voice-driven method for images proposed in Example 2 of the present application;

[0056] Figure 3 This is a flowchart of the voice-driven method for images proposed in Example 3 of the present application;

[0057] Figure 4 This is a flowchart of the voice-driven method for images proposed in Example 4 of the present application;

[0058] Figure 5 A schematic diagram of an optional application scenario of the voice-driven image method proposed in an embodiment of the present application;

[0059] Figure 6 This is a flowchart of the voice-driven method for images proposed in Example 5 of the present application;

[0060] Figure 7 This is a schematic diagram of another optional application scenario of the voice-driven image method proposed in the embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of an image-based voice-driven device provided in an embodiment of the present application;

[0062] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation methods section of this application are only used to explain the specific embodiments of the present application and are not intended to limit this application. The embodiments of the present application are described below in conjunction with the drawings. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0064] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0065] In order to solve the above problems, the embodiment of the present application provides a method for driving an image by voice. The method for driving an image by voice according to the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0066] Reference Figure 1 , is a flow chart of the voice-driven method for images proposed in the first embodiment of the present application. This method can be applied to electronic devices, which can be terminal devices or servers with data processing capabilities, or the terminal device and the server can cooperate to implement the method provided in the embodiment of the present application. Based on this, Figure 1 As shown, the voice-driven method for images proposed in this embodiment may include but is not limited to the following steps:

[0067] Step S11, obtaining at least one first original image frame containing a target object in the original video;

[0068] Step S12, performing three-dimensional facial reconstruction on the target object based on the at least one frame of the first original image to obtain a three-dimensional facial model of the target object;

[0069] In an embodiment of the present application, the original video is a video that needs to be voice-driven and contains the original image of the target object (such as a specified person or animated animal, which can be a real or virtual object). It can be a video file downloaded or filmed by the electronic device, or a video accessed through the browser of the electronic device, or a video accessed using video software installed on the electronic device (such as proprietary video apps of different companies), etc. This application does not restrict the source of the original video, which can be determined according to the circumstances.

[0070] When a voice is required to drive one or more frames of original images containing a target object, so that the target object's mouth shape moves synchronously with the voice and retains the target object's original facial expressions, postures and other movements, in the voice-driven process of the image, if the original image to be driven containing the target object in the original video is used to construct a three-dimensional facial model of the target object, and then the three-dimensional facial model is driven by voice to render a new image of the target object, it is necessary to construct a corresponding three-dimensional facial model for each frame of the original image to be driven, and then perform voice-driven and image rendering on it. However, since the differences between the original images to be driven for each frame of the target object are often small, there is a large amount of repeated calculation between the three-dimensional facial reconstruction processes for different frames of the original images to be driven, which increases the resource consumption of the electronic device and reduces the voice-driven efficiency of the image.

[0071] To address the aforementioned issues and reduce the number of 3D facial reconstructions, embodiments of the present application propose utilizing other more suitable original images from the original video (typically different from the original image to be driven, and thus referred to as the first original image), such as a first original image that more stably presents the facial features of the target object, to reconstruct the 3D facial model of the target object, thereby improving the accuracy of the facial features of the 3D facial model. Furthermore, when voice actuation is subsequently required for each frame of the original image to be driven containing the target object in the original video, this single 3D facial model can be directly actuated, eliminating the need for repeated 3D facial reconstruction of the target object. This eliminates the resource waste of electronic devices caused by repeatedly constructing the 3D facial model of the target object, and shortens the duration of non-initial voice actuation of the second original image of the target object, thereby improving efficiency.

[0072] It should be noted that this application does not restrict the method for constructing a three-dimensional facial model of a target object based on a first original image containing the target object. In conjunction with the principles of three-dimensional reconstruction technology, software modeling (three-dimensional facial modeling using computer software) can be employed. Specifically, based on the facial features of the target object in the first original image, mathematical, computer graphics, and other techniques are used to convert the facial features into a three-dimensional model.

[0073] In another possible implementation, an image-based modeling method, that is, a three-dimensional face modeling method using a two-dimensional image (a first original image), can also be used. For example, computer vision technology is used to match feature points of the original facial image of the target object in at least one frame of the first original image. A stereo reconstruction method is used to deduce the three-dimensional shape of the target object's face based on the position and geometric relationship of the feature points on the target object's face. Then, through texture mapping, the facial texture information of the target object in the first original image is mapped to the three-dimensional model, thereby improving the authenticity and reliability of the three-dimensional facial model of the target object.

[0074] It should be understood that in order to ensure the reliability and accuracy of the three-dimensional facial model constructed above and to be able to fully and accurately represent the three-dimensional facial features and texture features of the target object, the first original image obtained in step S11 can contain or represent the complete three-dimensional facial features and texture features of the target object. This can be based on the facial features of the target object actually contained in the first original image of each frame to determine which frames from the original video contain the original image of the target object as the first original image.

[0075] Step S13, obtaining a target voice; the target voice is used to drive a second original image in the original video, where the second original image is different from the first original image;

[0076] Following the above analysis, the target voice is the voice used to drive the lip shape of the original image to be driven in the original video (recorded as the second original image). It can be a voice recorded by an electronic device, or a voice file downloaded from a web page or other third-party platform, or one or more frames of voice in a voice text obtained by converting specified text content based on text-to-speech technology. This application does not impose any restrictions on the source of the target voice, which can be determined according to the circumstances.

[0077] In an embodiment of the present application, after determining the second original voice to be driven in the original video (i.e., the above-mentioned original voice to be driven), a matching target voice can be obtained based on factors such as the identity characteristics of the target object contained in the second original voice, the plot described in the original video, and the plot role requirements of the target object, to ensure that the voice-driven result meets actual needs. Of course, after obtaining a voice used to drive the original video as the target voice, combined with the content of each frame of the target voice, the original image to be driven containing the target object is selected from the original video and determined as the second original image, etc. This application does not limit the method and order for obtaining the target voice and the second original image, and can be determined based on the actual voice-driven control needs.

[0078] Step S14, driving the three-dimensional facial model by the target voice, and rendering a target facial image of the target object based on the driven three-dimensional facial model;

[0079] Step S15 , using the target facial image to replace the original facial image of the target object in the second original image, to obtain a driving result for the second original image.

[0080] In conjunction with the above description of the implementation process for obtaining a three-dimensional facial model of the target object, the embodiment of the present application can drive the three-dimensional facial model of the target object constructed based on the first original image, regardless of which frame of the target speech is obtained for driving the second original image. The driven three-dimensional facial model has the complete and accurate three-dimensional facial shape, texture, and other features of the target object, and the facial perspective is consistent with the perspective presented by the second original image to be driven, ensuring that the rendered target facial image meets the driving requirements. The rendering process of the target facial image can be implemented using, but is not limited to, rendering engines such as UE4 and Unity. The rendering method of step S14 is not described in detail in this application.

[0081] Since the present application drives the facial movements of the target object in the second original image through the target voice, other area images of the second original image do not need to be updated. In this way, the rendered target facial image of the target object can be directly used to replace the original facial image corresponding to the second original image, so that the replaced image differs from the second original image before replacement only in the facial area of ​​the target object, ensuring that the facial movement of the replaced image matches the mouth movement after driving, thereby improving the naturalness and authenticity of the voice drive.

[0082] In this way, in a scenario where different frames of target voice need to be used to drive the second original image of the corresponding frame containing the target object in the original video, after obtaining the three-dimensional facial model of the target object according to the implementation method of step S11 and step S12, the three-dimensional facial model can be directly driven by each frame of target voice, so that the rendering engine renders the target facial image of the target object that matches the corresponding frame of target voice based on the driven three-dimensional facial model. There is no need to construct the three-dimensional facial model of the target object each time a frame of target voice is obtained, which reduces the time and electronic equipment resource cost of repeatedly constructing the three-dimensional facial model of the target object and improves the efficiency of obtaining the target facial image.

[0083] Moreover, since the present application renders the entire facial image of the target object and replaces the original facial image in the second original image, it ensures that the facial movement of the target object in the resulting driving result is natural. This solves the problem that only the mouth animation of the target object is driven, and the facial movement of the target object in the second original image is omitted, which easily leads to the new mouth animation not matching the original facial movement, thereby resulting in unnatural facial movement in the new image. This improves the naturalness and authenticity of voice driving.

[0084] It should be understood that the 3D facial model of the target object constructed in steps S11 and S12 above may have been constructed while the electronic device was performing other tasks. When voice-activated driving of the second original image in the original video is required, the constructed 3D facial model of the target object can be directly obtained, and the subsequent target voice-activated driving steps can be performed. Furthermore, the voice-activated driving of the second original image in the original video does not affect the activation of the original image containing other objects in the original video (which may be referred to as the third original image) by the target voice or other voices. The voice-activated driving process for the third original image is similar, and this application does not provide detailed examples one by one.

[0085] Reference Figure 2 , is a flow chart of the voice-driven method for an image proposed in the second embodiment of the present application. This embodiment can provide a detailed description of an optional implementation process of how to drive the second original image through the target voice in the voice-driven process of the above-mentioned image to obtain its driving result, such as Figure 2 As shown, the voice-driven method for images proposed in this embodiment may include:

[0086] Step S21, obtaining at least one first original image frame containing a target object in the original video;

[0087] Step S22, performing three-dimensional facial reconstruction on the target object based on the at least one frame of the first original image to obtain a three-dimensional facial model of the target object;

[0088] Step S23, obtaining a target voice; the target voice is used to drive a second original image in the original video, where the second original image is different from the first original image;

[0089] Step S24, driving the three-dimensional facial model by the target voice, and rendering a target facial image of the target object based on the driven three-dimensional facial model;

[0090] Regarding the implementation process of steps S21 to S24, reference may be made to the description of the corresponding part of the context embodiment, such as the implementation method described in steps S11 to S14.

[0091] Step S25, pasting the target facial image to the facial position of the target object in the second original image to obtain an updated image;

[0092] In an embodiment of the present application, a map replacement method can be used to paste the rendered target facial image of the target object, that is, the facial image that matches the target voice, back to the facial position of the target object in the second original image to obtain an updated image for the second original image, so that the updated image is aligned with the target object face in the second original image. When the target voice is different from the original voice in the original video corresponding to the second original image, the present application can ensure that the texture features, facial expressions, facial postures and other features between the target facial image of the target object in the updated image and the original facial image of the target object in the second original image are consistent, thereby changing the original mouth shape (that is, mouth movements) of the target object and realizing the synchronous driving of the target voice on the mouth shape of the target object.

[0093] It should be noted that this application does not limit the implementation method of the mapping in step S25. It can be implemented using artificial intelligence AI video face-changing technology, such as using a tracking algorithm to realize the facial position of the target object in the original video in the second original image, and using a facial alignment algorithm to align and fuse the target facial image with the original facial image in the second original image in the original video to ensure seamless visual replacement and more consistent with the surrounding environment of the non-target object facial area in the second original image. In this fusion process, the image processing method described in the following steps is adopted but not limited to meet the video face-changing effect. Of course, for the above-mentioned mapping method, other AI video editing tools can also be called to implement it, such as MimicBrush, LivePortrait, Opus Clip, WinkStudio, Chillin, etc., which can be flexibly selected according to actual task processing requirements and preferences. The working principles of each AI video editing tool are not described in detail in this application.

[0094] Step S26, determining whether the updated image meets the correction condition, if so, proceeding to step S27; if not, executing step S28;

[0095] Step S27, correcting the updated image to obtain a driving result for the second original image;

[0096] Step S28: determining the updated image as the driving result for the second original image.

[0097] Since this application requires that the facial area of ​​the target object in the voice-driven image be integrated with the non-facial area to present a natural appearance, however, in actual situations, different facial movements will be generated when executing different mouth shapes, and the facial movements will change with the changes in the mouth movements, and the facial contours will change synchronously with the changes in the facial movements, resulting in the facial contour shape of the target facial image being different from the facial contour shape of the original facial image. In this way, when the facial contour position of a part of the target facial image is reduced, the updated image will retain the facial contour of that part in the original facial image, so that the facial contour actually presented in the updated image is different from the facial contour of the target facial image, thereby making the facial movement presented in the updated image unnatural.

[0098] For example, the target speech drives the target object to close its mouth (i.e., a mouth movement in which the mouth opening and closing angle is less than a first angle threshold), and the target object in the second original image performs a mouth opening (a mouth movement in which the mouth opening and closing angle is greater than a second angle threshold, the second angle threshold is greater than the first angle threshold, such as a larger second angle threshold such as 90 and a smaller first angle threshold such as 0 or 5, and this application does not limit the values ​​of the angle thresholds). Relative to the facial contour position of the target object when the mouth is open in the original facial image, the chin contour position of the target object when the mouth is closed will move toward the center of the face, resulting in a decrease in the distance between the top and bottom of the face (the lowest point of the chin contour) in the target facial image. The target facial image is directly pasted onto the location of the original facial image, and the facial contour of that portion in the resulting updated image retains that portion of the facial contour of the original facial image, so that the facial movement presented thereby does not match the target mouth movement driven by the speech. That is, when the target object closes its mouth, the chin still exhibits a downward and protruding visual effect that only occurs when the mouth is opened. At this time, the facial movement presented by the target object in the updated image is unnatural.

[0099] To avoid the aforementioned issues, modification conditions can be pre-configured to indicate unnatural facial mapping results, based on various mapping results that may result in unnatural facial movement during facial mapping replacement. For example, in the example above, when the target facial image is applied to the corresponding original facial image, it is not possible to completely cover the original facial image, resulting in the updated image still showing a portion of the original facial image. Further modification is required to avoid showing the original facial image. In this case, a correction condition describing the mapping result can be configured, but this is not limited to this content. The modification condition can be flexibly configured or adjusted based on actual circumstances, and this application does not impose any restrictions on the content of the modification condition.

[0100] Based on this, the present application determines that the target facial image is directly pasted back onto the target subject's facial position in the second original image through a mapping method. After obtaining the direct mapping result, i.e., the updated image, to ensure smooth and natural facial movement of the target subject in the voice-driven image, further verification can be performed to determine whether the updated image meets the correction conditions. If so, it indicates that the facial movement of the target subject in the updated image is unnatural. If there is any part of the original facial image left after the mapping that causes the target subject's face to be unnatural, it needs to be corrected to make the facial movement natural. Otherwise, it indicates that the facial movement of the target subject in the updated image is natural and the target subject's mouth shape and facial movement match. No further correction is required and the updated image can be directly output as the driving result or used to execute subsequent tasks.

[0101] During the correction process for the updated image, portions of the updated image that actually meet the correction conditions can be corrected so that they no longer meet the correction conditions, such as by making the area no longer represent the corresponding original facial image, while ensuring that the facial image in the corrected image is consistent with the surrounding environment. In some embodiments, the electronic device can launch appropriate image editing software or an image processing model to correct the updated image, ensuring that the facial movement in the corrected image is natural. This application does not limit the method for implementing the correction of the updated image.

[0102] In practical applications, according to the method described above, after the second original image of the corresponding frame in the original video is driven by multiple consecutive frames of target voice, a voice-driven video for the target object can be formed by the target voice and the driving results of the corresponding frames, such as voice-driven graphics, or the target voice replaces the original voice corresponding to the target object in the original video, and the driving results of the second original image of each frame are combined to obtain the driven target video, ensuring that the background and appearance of the target object driven by the target voice in the target video remain unchanged throughout the video, and that the image after voice driving presents a natural, smooth and continuous effect with the original image that has not been driven. This application does not limit the processing tasks and applications after obtaining the driving results of the target voice on the second original image, and can be determined according to the circumstances.

[0103] In some embodiments, combined with the above analysis, if the above correction condition includes that the updated image contains part of the original facial image of the target object in the second original image, it means that in the process of aligning the target facial image with the original facial image of the target object in the second original image and pasting them back, the updated image directly obtained after pasting back does not completely cover the original facial image. In this case, when modifying this type of updated image, in a possible implementation, the facial position of the uncovered original facial image in the updated image can be directly identified, and the non-facial image in the updated image (such as the background environment image or the non-facial body part image of the target object, etc., as appropriate) is used to correct the part of the original facial image of the target object that exists in the updated image (that is, modify the image of the identified facial position) to obtain a driving result for the second original image, ensuring that the driving result no longer contains any part of the original facial image of the target object.

[0104] In this process, the context information of the facial area in the updated image (i.e., the image content of the adjacent area) can be combined, or the second original image and several adjacent frames before and after it in the original video can be contextually analyzed to determine what content image the remaining part of the original facial image should be, so as to utilize the same content image in the updated image to correct this part of the original facial image, ensure that the modified image content matches the surrounding content features, and that the facial movement of the target object is natural.

[0105] Based on the above analysis, refer to Figure 3 , is a flow chart of the voice-driven method for images proposed in the third embodiment of the present application. This embodiment describes an optional implementation process of how to replace the original facial image of the target object in the second original image with the rendered target facial image. Regarding the process of obtaining the target facial image, please refer to the corresponding part of the above embodiment, such as the implementation method described in the above steps S11 to S14, which will not be repeated in this embodiment. Based on this, Figure 3 As shown, the implementation method of executing the content in step S15 may include:

[0106] Step S31, pasting the target facial image to the facial position of the target object in the second original image to obtain an updated image;

[0107] Step S32, determining that the updated image satisfies the correction condition, extracting background image data from the updated image, and filling the background image data into a target background area in the partial original facial image of the target object in which the updated image exists;

[0108] Following the above analysis, through a patch replacement method, it is determined that the target facial image is directly pasted back to the facial position of the target object in the second original image. The resulting updated image contains a portion of the target object's original facial image, indicating that the updated image meets the correction condition. This portion of the original facial image is unnecessary, and the facial movement presented in the updated image needs to be natural, with the facial movements and mouth shape matching. In this way, it is determined that the area containing the portion of the original facial image in the updated image should be the target background area, that is, the portion of the original facial image should be the background image of the non-target object. Background image data can be directly extracted from the updated image, such as from a background image adjacent to the target background area, and then filled into the target background area in the updated image so that it can be visually presented as the background image.

[0109] In another possible implementation, combined with the above analysis, the present application can also determine the background area from the changes in the original images of the adjacent video frames of the second original image, and extract background image data therefrom to fill in the target background area of ​​the uncovered portion of the original image that should be the background image. In addition to directly extracting from the adjacent original image or the updated image or the second original image, or adaptively adjusting the extracted background image data so that the transition between the filled target background area image and its adjacent background image is natural and smooth, the change rules of the model or background image data can also be used to directly generate the required background image data, etc. The present application does not limit the source or generation method of the background image data used to cover the uncovered portion of the original facial image that should be the target background area.

[0110] For example, the mouth shape of the target object in the second original image is closed, and the mouth shape of the target facial image rendered after the target voice drives the three-dimensional facial model is open. As the target mouth opening and closing angle increases, its facial features such as the mandibular angle are pushed back and lowered, and the lateral face at the corresponding position becomes narrower. Pasting the target facial image back into the original facial image in the second original image causes the mandibular angle portion of the original facial image to be unable to be covered, so that the updated image presents the two mandibular angles of the target object or the mandibular angle contour is unnatural, which does not conform to the visual effect that the mandibular angle should have when the target object opens its mouth. When it is determined that the uncovered original mandibular angle image should be the background image based on the image of the mandibular angle adjacent area of ​​the target object in the second original image, the background image data for covering the portion of the original facial image can be obtained according to, but not limited to, the method described above, and then filled into the uncovered original mandibular angle area to present the background image.

[0111] It should be understood that for the updated image that should be the background image, but it actually contains other types of uncovered original facial images, such as uncovered original cheek images, etc., you can directly refer to the correction method of the original mandibular angle image described above, and this application will not give detailed examples one by one.

[0112] Step S33, performing pixel filling on the target body area in the partial original facial image of the target object in the updated image according to the body image data or the image filling model of the target object in the updated image;

[0113] Step S34 : determining that the updated image after filling does not meet the correction condition, and determining it as the driving result of the second original image.

[0114] Following the above analysis, when it is determined that the area where part of the original facial image in the updated image is located should be the body area of ​​the target object, as in the above case where the target voice drives the target object to update from an open mouth to a closed mouth in the second original image, the lowest position of the target object's chin contour becomes higher with the change of the temporomandibular joint (change in the opening and closing angle). Pasting the target facial image back to the facial position of the target object in the corresponding second original image will cause the lowest position of the chin contour of the target object in the updated image when the mouth is closed to be the same as the lowest position of the chin contour when the mouth is open. This is obviously unreasonable and will cause the facial shape of the target object in the updated image to be different from the facial shape of the target object in the original image, affecting the image realism and even causing the facial contour in the updated image to be unsmooth and the facial movement to be unnatural.

[0115] In this case, the portion of the original facial image not covered by the target facial image in the updated image (such as the original chin image in the above example) should be the neck image of the target object, that is, the target body area image. Based on the recognition result, this application can extract the body image data of the target object's corresponding target body area from the updated image or the second original image, and fill it into the target body area where the portion of the original facial image is located. For example, using the neck image to cover the original chin image solves the problem of unnatural facial movement caused by not covering the original chin image. It should be understood that other original facial images not covered in the updated image, such as ear images, can be corrected according to the method described above. This application does not provide detailed examples of the implementation process.

[0116] In some other embodiments, in addition to extracting the corresponding body image data for pixel filling when correcting the target body area in the updated image, a trained image filling model can also be used, combined with the updated image or the second original image or its adjacent frame original image, to generate body image data for the uncovered original facial image in the updated image, and fill it into the corresponding target body area in the updated image so that the filled image no longer contains the original facial image. The image filling model can be an image restoration project trained with a Generative Adversarial Network (GAN), such as AOT-GAN-for-Inpainting; or a model built based on an image filling algorithm such as imfill; or a model supporting image correction functions built based on image correction technology; or some AI image processing applications, etc. This application does not elaborate on the structure and working principle of the image filling model, and the required type of image filling model can be flexibly selected based on actual conditions.

[0117] It should be noted that, in the process of correcting the updated image that meets the correction conditions, if the part of the original facial image in the updated image that is not covered by the target facial image only contains the target background area or the target body area, step S32 or step S33 can be executed accordingly to complete the correction of the updated image and obtain the driving result of the second original image.

[0118] Of course, after completing the correction according to the method described in step S32 and step S33 (here referring to the method of extracting body image data for filling), if there is still an uncovered part of the original facial image in the updated image after filling, it is still possible to continue to fill the pixels according to the image filling model to ensure that the updated image after filling does not contain the original facial image, that is, the correction condition is not met, so as to determine that it is the result of the target voice driving the second original image.

[0119] In addition, for the updated image that meets the correction conditions, the pre-trained image filling model can also be directly used to perform pixel filling on part of the original facial image of the target object in the updated image to obtain the driving result of the second original image. The implementation process can refer to but is not limited to the pixel filling method described above, and will not be described in detail in this embodiment.

[0120] To summarize, the present application drives the three-dimensional facial image of the target object through the obtained target voice, obtains the target facial image of the target object that matches the target voice, aligns it with the face of the target object in the second original image and fits it to the original video. In the process, if the target facial image completely covers the original facial image, the original facial image does not exist in the obtained updated image, that is, it does not meet the correction conditions. For example, if the target facial image is the facial image of the target object with its mouth open, and the corresponding original facial image is the facial image of the target object with its mouth closed, the target facial image can be pasted back into the second original image to completely cover the original facial image, and there is no need to fill and correct the updated image.

[0121] However, as in the examples above, the target facial image generated when other mouth shape changes occur cannot completely cover the original facial image, resulting in the resulting updated image containing part of the original facial image. This causes the facial movements visually presented in the updated image to not match the mouth shape, and the facial movements to be unnatural. To address this issue, the area of ​​the original facial image that is not covered in the updated image can be located. By extracting the target image data (such as the background image data or one or more of the body image data) or image filling model required to cover the area, the pixel filling of the original image that is not covered in the updated image can be performed so that the corresponding area visually presents the correct image (corresponding to the background image or the body image). This ensures that the facial movements in the updated image after filling are natural, and that the original facial image is absent, thereby improving the naturalness and authenticity of the original image of the speech-driven target object.

[0122] Reference Figure 4 , is a flow chart of the voice-driven method for images proposed in the fourth embodiment of this application. This embodiment can describe another optional refinement of the voice-driven method for images described above, and can describe an optional refinement of the construction process of a three-dimensional facial model and the driving process. Figure 4 As shown, the method may include:

[0123] Step S41, obtaining at least one first original image frame containing a target object in an original video that meets the three-dimensional facial modeling conditions;

[0124] In an embodiment of the present application, in order to ensure that the three-dimensional facial model constructed for the target object based on the three-dimensional reconstruction technology is consistent with the three-dimensional facial shape and texture features presented by the target object in the original video, the target object image used to construct the three-dimensional facial model is required to be the original image from the original video. The visually presented face of the target object is stable, and the complete three-dimensional facial shape and texture features can be extracted from it. The present application can pre-configure the three-dimensional facial modeling conditions accordingly.

[0125] Thus, when it is necessary to construct a 3D facial model for a target object in an original video, one or more frames of original images that meet the 3D facial modeling conditions are selected from the original image frames of the original video containing the target object. In other words, one or more frames of original images alone can meet the 3D facial modeling conditions, or multiple frames of original images can meet the 3D facial modeling conditions. The 3D facial models constructed in these two cases are the same. These two cases can be determined based on the facial features of the target object contained in each frame of original image. This application does not limit the implementation method of step S41.

[0126] Step S42, extracting facial key point features and facial texture features of the target object from the at least one frame of the first original image;

[0127] After determining the first original image in the original video that meets the three-dimensional facial modeling conditions, facial key point features and facial texture features of the target object that meet the three-dimensional facial modeling conditions can be extracted from it, where the facial key point features can be relatively stable features in the original video, such as the target object's eyes, nose, mouth, face contour (such as chin), etc. The facial key point features of the target object in the original video are basically unchanged (remain consistent) between different frames.

[0128] In practical applications, landmark detection algorithms can be used, but are not limited to them. The landmarks in this application refer to key points on the face, and facial key point features in the first original image are detected based on these landmarks. Facial key point detection can also be achieved through any one or more combinations of methods based on deep learning, methods based on cascaded shape regression, methods based on AAM (Active Appearance Models, a statistical model that locates facial key points by combining facial shape and texture information) / ASM (Active Shape Models, a model based on shape information), and methods based on graph models. This application does not limit the method for extracting facial key point features.

[0129] The extraction of facial texture features of the target object in the first original image may be achieved by using one or more of deep learning techniques, traditional texture feature extraction methods, wavelet transforms, bilateral convolutional networks, multi-scale and texture feature enhancement methods, and detection methods based on color space + texture features. In some embodiments, deep learning techniques and other methods may also be used to extract facial key point features and texture features of the target object in the first original image. The implementation process of step S42 is not described in detail in this application.

[0130] Step S43, constructing a three-dimensional facial model of the target object using facial key point features and facial texture features;

[0131] In practical applications, the facial shape of the target object is represented by the features of each target key point. Based on this, the target object's face is geometrically modeled, such as using polygonal mesh modeling technology, to obtain a three-dimensional facial shape model of the target object. Texture features are mapped to the three-dimensional facial shape model, such as by attaching a two-dimensional image to the surface of the corresponding three-dimensional image in the three-dimensional facial shape model to obtain a three-dimensional facial model of the target object, thereby representing the three-dimensional facial shape and texture of the target object. Of course, the present application can also use other methods to reconstruct the target object's face in the first original image, such as a reconstruction method based on stereo vision matching, a three-dimensional face modeling method based on deep learning, or a NeRF (Neural Radiance Fields) reconstruction technology to construct a three-dimensional facial model of the target object.

[0132] Step S44, obtaining a target voice, and determining a second original image driven by the target voice in the original video;

[0133] Step S45, extracting target facial perspective data of the target object in the second original image;

[0134] Step S46, driving the three-dimensional facial model to perform corresponding actions based on the target voice and the target facial perspective data, thereby obtaining a driven three-dimensional facial model of the target object;

[0135] Step S47, rendering a target facial image of the target object based on the driven three-dimensional facial model;

[0136] In such Figure 5 In the scenario shown, where the combination of multiple frames of first original images meets the 3D facial modeling conditions, the 3D facial modeling conditions include: a combination of multiple frames of first original images that can enhance the target object's complete facial features, including facial key features and facial texture features. After constructing a 3D facial model of the target object in the original video, a new speech is obtained. For any frame of the target speech, the second original image of the corresponding frame in the original video driven by the speech can be determined. For example, based on the correspondence between the target object's original image and the original speech in the original video, the second original image driven by the target speech can be determined.

[0137] Since the second original image is inconsistent with the facial posture, expression, etc. of the target object in the constructed three-dimensional facial model, driving the three-dimensional facial model only according to the target voice usually generates facial expressions with lip synchronization and its mapping, which may not match the facial expressions, postures and other features in the second original image. In order to improve the voice-driven results, the present application proposes to drive the three-dimensional facial model according to the target voice and the second original image, so that the driven three-dimensional facial model is consistent with the facial features represented by the three-dimensional facial model obtained by directly performing three-dimensional facial reconstruction on the second original image, ensuring that the target facial image rendered thereby is consistent with the facial perspective data such as facial expressions and postures in the second original image, and that the mouth shape visually presented in the target facial image matches the target voice.

[0138] Based on this, the present application can extract the target facial perspective data of the target object from the second original image, such as the direction, rotation angle, tilt angle, scaling ratio and other feature data representing the facial posture of the target object's face in the second original image, and drive the three-dimensional facial model to perform corresponding actions in combination with the target voice, such as Figure 5 As shown, the perspective data represented by the driven three-dimensional facial model is made consistent with the facial perspective data of the target object in the second original image.

[0139] Step S48: Using the target facial image, the original facial image of the target object in the second original image is replaced to obtain a driving result for the second original image.

[0140] According to the method described above, the rendered target facial image is pasted back to the corresponding target object's facial position in the second original image. However, there may still be a situation where the target facial image does not completely cover the original facial image of the target object in the second original image. That is, the updated image obtained by directly pasting meets the correction conditions. The updated image can be corrected according to but not limited to the method described above to ensure that the driving result of the second original image does not contain the original facial image, and the visually presented facial movement is natural and realistic.

[0141] It should be understood that, for the scenario where multiple frames of target speech need to be driven by the original video corresponding to multiple frames of the second original image, such as Figure 5 As shown, the implementation process of driving the corresponding second original image for each frame of target speech is similar, that is, each time a frame of target speech is obtained, the driving result corresponding to the second original image can be obtained according to but not limited to the method described in steps S44 to S48. There is no need to repeatedly construct a three-dimensional facial model of the target object, which reduces the time and resource consumption caused by repeated execution of three-dimensional facial modeling, improves the speech driving efficiency of the original face of the target object in the original video, and improves the naturalness and authenticity of the driving result.

[0142] Reference Figure 6, is a flow chart of the voice-driven method for images proposed in the fifth embodiment of the present application, which can describe another optional detailed implementation method of how to construct a three-dimensional facial model of the target object and its driving implementation process in the voice-driven method for images, such as Figure 6 As shown, the method may include:

[0143] Step S61, performing mouth movement tracking detection on each frame of the original image from the original video to obtain the mouth opening and closing angle of the target object in the original image containing the target object;

[0144] Step S62, obtaining a first original image corresponding to the mouth opening and closing angle being less than the angle threshold;

[0145] In an embodiment of the present application, the aforementioned 3D facial modeling conditions include a first original image frame capable of enhancing the complete facial features of the target subject, and a first original image frame meeting the 3D facial modeling conditions can be selected from the original video. In conjunction with the description of the corresponding portion of the above embodiment, when the target subject's face is facing outward from the screen and the mouth is open and closed at a small angle, the target subject's facial movements are small. The first original image capable of presenting this visual effect can accurately and completely represent the target subject's facial features, indicating that the first original image frame meets the 3D facial modeling conditions. Thus, only this first original image frame is required for 3D facial modeling.

[0146] Based on this, combined Figure 7 In the application scenario shown, a landmark detection algorithm can be used to locate a closed-mouth frame, i.e., a video frame where the mouth opening and closing angle is less than an angle threshold (i.e., the first angle threshold, such as a small value such as 0 or 5, which is not limited in this application), by analyzing the angle between the upper and lower lips, i.e., the mouth opening and closing angle, in the original image of the target object in the original video. This algorithm can then be used to obtain the first original image of this frame. It should be understood that other algorithms can also be used in this application to select this first original image from the original video, but this implementation process is not described in detail in this application.

[0147] Step S63, performing three-dimensional facial reconstruction on the target object based on the first original image of the frame to obtain a three-dimensional facial model of the target object;

[0148] Step S64: obtaining a target voice; the target voice is used to drive a second original image in the original video, where the second original image is different from the first original image;

[0149] Regarding the implementation process of step S63 and step S64, reference may be made to the description of the corresponding parts of the above embodiment, which will not be elaborated in this embodiment.

[0150] Step S65, determining a correspondence between the mouth movement data of the target object in the original image and the original phoneme features of the original voice based on the original image and the original voice of the target object in the original video;

[0151] Step S66, extracting features of the target speech to obtain target phoneme features;

[0152] Step S67, determining target mouth movement data corresponding to the target phoneme feature based on the corresponding relationship;

[0153] Step S68: driving the 3D facial model to perform corresponding mouth movements based on the target mouth movement data, and driving the 3D facial model to perform corresponding facial movements based on the target facial perspective data, thereby obtaining a driven 3D facial model of the target object.

[0154] In the process of implementing the three-dimensional facial model driven by the target voice, it can be achieved by driving the mouth shape by phonemes. In one possible implementation, the known different phonemes and mouth shape change rules are used to determine the mouth shape corresponding to the phoneme characteristics of the obtained target voice, that is, the target mouth movement data, and accordingly drive the three-dimensional facial model to perform the corresponding mouth movement.

[0155] In another possible implementation, in order to improve the reliability and accuracy of speech phoneme-driven mouth shapes and to better align them with the mouth shapes of the same phoneme features in the original image that was not driven in the original video, a phoneme mouth shape matching analysis can be performed on the original image containing the target object and its corresponding original speech in the original video to determine the correspondence between different original phoneme features and the mouth movement data of different mouth shapes, thereby determining the target mouth movement data corresponding to the target phoneme features of the target speech. Specifically, for the extraction of speech phoneme features, the MFCC (Mel Frequency Cepstrum Coefficient) algorithm can be used to extract the spectral features of the target speech, and the target phoneme features of the target speech of the corresponding frame can be determined accordingly. The implementation process is described in detail in this application.

[0156] It can be seen from this that the mouth shape driven by the target voice, that is, the target mouth movement data described in steps S65 to S67 of this application, is consistent with the mouth movement data of the original voice in the original video with the same phonemic features as the target voice, ensuring that the subsequent driving results of the second original image and the visual effects presented between other original images in the original video are natural and smooth.

[0157] Step S69, rendering a target facial image of the target object based on the driven three-dimensional facial model;

[0158] Step S610: Using the target facial image, the original facial image of the target object in the second original image is replaced to obtain a driving result for the second original image.

[0159] Regarding the implementation method of step S69 and step S610, reference may be made to but not limited to the description method of the corresponding parts of the above embodiment, and this embodiment will not be described in detail here.

[0160] It can be seen that the present application selects a first original image that presents the stable facial features of the target object from the original image, such as Figure 7 The closed-mouth frame original image of the target object shown is used to construct a three-dimensional facial model of the target object. Afterwards, for a new voice used to drive the original video, each frame of target voice can be combined with the facial perspective data of the corresponding second original image to drive the same three-dimensional facial model corresponding to the closed-mouth frame, so as to quickly and accurately obtain the driven three-dimensional facial model consistent with the facial features of the corresponding second original image, and paste the target facial image of the target object rendered based on this back to the target object facial position of the corresponding frame second original image to replace the original facial image, so as to ensure that the facial movement of the target object in the driving result of the second original image is natural and continuous with the facial changes of the target object in the adjacent frame images in the driven original video, thereby improving the naturalness and authenticity of the voice driving.

[0161] The above describes a method for driving an image through voice according to an embodiment of the present application. The following describes a device for executing the method for driving an image through voice.

[0162] Reference Figure 8 , is a structural diagram of an image voice driving device provided in an embodiment of the present application, such as Figure 8 As shown, the voice-driven device of the image may include:

[0163] The first original image acquisition module 810 is configured to acquire at least one first original image frame containing a target object in the original video;

[0164] a 3D facial reconstruction module 820, configured to perform 3D facial reconstruction on the target object based on the at least one first original image to obtain a 3D facial model of the target object;

[0165] A target speech acquisition module 830 is configured to acquire a target speech; the target speech is used to drive a second original image in the original video, the second original image being different from the first original image;

[0166] A driving module 840 is configured to drive the three-dimensional facial model using the target voice;

[0167] A rendering module 850 is configured to render a target facial image of the target object based on the driven three-dimensional facial model;

[0168] The replacement module 860 is configured to replace the original facial image of the target object in the second original image with the target facial image to obtain a driving result for the second original image.

[0169] In some embodiments, the replacement module 860 may include:

[0170] a mapping unit, configured to map the target facial image to the facial position of the target object in the second original image to obtain an updated image;

[0171] a correction unit, configured to correct the updated image when it is determined that the updated image satisfies a correction condition, to obtain a driving result for the second original image;

[0172] The determining unit is configured to determine the updated image as a result of driving the second original image when it is determined that the updated image does not satisfy a correction condition.

[0173] Wherein, when the correction condition includes: the updated image contains a partial original facial image of the target object in the second original image, the correction unit may include any of the following subunits:

[0174] A first correction subunit is configured to correct a portion of the original facial image of the target object in the updated image, thereby obtaining a driving result for the second original image.

[0175] The first pixel filling subunit is used to perform pixel filling on a portion of the original facial image of the target object in the updated image using an image filling model to obtain a driving result for the second original image.

[0176] In a possible implementation, the first correction subunit may include at least one of the following filling subunits:

[0177] A background pixel filling subunit is configured to extract background image data from the updated image and fill the background image data into a target background area in the partial original facial image of the target object in the updated image;

[0178] The body pixel filling subunit is used to perform pixel filling on the target body area in the partial original facial image of the target object in the updated image according to the body image data or image filling model of the target object in the updated image.

[0179] In some further embodiments, if at least one first original image frame obtained from the original video meets the 3D facial modeling conditions, the 3D facial reconstruction module 820 may include:

[0180] A feature extraction unit, configured to extract facial key point features and facial texture features of the target object from the at least one frame of the first original image;

[0181] A construction unit is used to construct a three-dimensional facial model of the target object using the facial key point features and the facial texture features.

[0182] Wherein, when the three-dimensional facial modeling conditions include: a frame of the first original image or a combination of multiple frames of the first original images capable of enhancing the complete facial features of the target object, and the facial features include facial key point features and facial texture features of the target object, the first original image acquisition module 810 may include:

[0183] a mouth opening and closing angle obtaining unit, configured to perform mouth movement tracking detection on each frame of an original image from an original video, and obtain the mouth opening and closing angle of the target object in the original image containing the target object;

[0184] The first original image obtaining unit is configured to obtain a frame of the first original image corresponding to a mouth opening and closing angle being smaller than an angle threshold.

[0185] In some further embodiments, the driving module 840 may include:

[0186] A second original image determining unit, configured to determine a second original image driven by the target voice in the original video;

[0187] a target facial perspective data extraction unit, configured to extract target facial perspective data of the target object in the second original image;

[0188] A driving unit is used to drive the three-dimensional facial model to perform corresponding actions according to the target voice and the target facial perspective data, so as to obtain the driven three-dimensional facial model of the target object.

[0189] In a possible implementation, the driving unit may include:

[0190] A first determining subunit is configured to determine, based on an original image and an original voice of a target object in the original video, a correspondence between mouth movement data of the target object in the original image and original phoneme features of the original voice;

[0191] A target phoneme feature obtaining subunit is used to extract features of the target speech to obtain target phoneme features;

[0192] a second determining subunit, configured to determine target mouth movement data corresponding to the target phoneme feature based on the corresponding relationship;

[0193] The driving subunit is used to drive the three-dimensional facial model to perform corresponding mouth movements according to the target mouth movement data, and to drive the three-dimensional facial model to perform corresponding facial movements according to the target facial perspective data, so as to obtain the driven three-dimensional facial model of the target object.

[0194] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the image voice-driven methods provided in the embodiments of the present application.

[0195] Among them, when the computer-readable instructions are loaded and executed on the computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-readable instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer-readable instructions can be transmitted from one website, computer, training equipment or data center to another website, computer, training equipment or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method to meet the data (e.g., original video or driving results, etc.) / instruction transmission requirements during the execution of the voice-driven method for images proposed in the embodiment of the present application.

[0196] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the image voice-driven methods provided in the embodiment of the present application.

[0197] The computer-readable storage medium may be any available medium capable of storing data on a computer, or a data storage device such as a training device or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, hard disk, or magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0198] Reference Figure 9, is a hardware structure diagram of an electronic device provided in an embodiment of the present application. The electronic device may be a terminal device or a server. The terminal device may include but is not limited to a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, a netbook, a robot or a business terminal, etc. The server may be one or more physical servers, or a cloud server, etc. Figure 9 As shown, the electronic device may include but is not limited to: at least one communication element 91, at least one memory 92 and at least one processor 93, wherein:

[0199] At least one communication element 91, at least one memory 92, and at least one processor 93 can communicate with each other via a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by only one bidirectional line, but this does not mean that there is only one bus or one type of bus.

[0200] The communication element 91 can be used to obtain the original video and the target voice, realize the communication between the internal components of the electronic device, and the communication between the electronic device and other devices (such as the video source providing the original video, or a server, etc.), so as to transmit the data or instructions required for the execution of the voice-driven method of the image.

[0201] It can be seen that the communication element 91 may include a communication element corresponding to wireless communication methods such as WiFi, Bluetooth, 5G / 6G and other mobile communications, so that the electronic device can realize data transmission with other devices through the communication element. It can also include one or more interfaces supporting wired communication methods, such as general-purpose input / output (GPIO) interface, USB interface, universal asynchronous receiver / transmitter (UART) interface, etc., to realize data transmission between various components within the electronic device. This application does not limit the component structure of the communication element 91 to realize this function and its corresponding communication transmission mechanism, which can be determined according to the circumstances.

[0202] The memory 92 can be used to store computer program instructions for implementing the voice-driven method for images proposed in the embodiments of the present application; the processor 93 can load and execute multiple computer instructions stored in the memory 92 to implement the various steps of the voice-driven method for images proposed in the embodiments of the present application. The implementation process can refer to the description of the corresponding part of the method embodiment above.

[0203] In the embodiment of the present application, the memory 92 may include the storage media listed above. The processor 93 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0204] It should be understood that Figure 9 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 9 More or fewer components shown, or combinations of certain components, such as microphones, speakers, displays, various sensors, antennas, power modules, RF components, external ports and other input / output components, can be determined based on processing function requirements, and this application does not provide detailed examples one by one.

[0205] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0206] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. In this specification, the various embodiments are described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the devices and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

Claims

1. A method for driving an image by voice, the method comprising: Acquire at least one frame of a first original image containing a target object in the original video; Reconstructing the three-dimensional face of the target object according to the at least one frame of the first original image to obtain a three-dimensional face model of the target object; Obtaining a target voice, wherein the target voice is used to drive a second original image in the original video, wherein the second original image is different from the first original image; Driving the three-dimensional facial model by the target voice, and rendering a target facial image for the target object according to the driven three-dimensional facial model; The target facial image is used to replace the original facial image of the target object in the second original image, so as to obtain a driving result for the second original image.

2. The method according to claim 1, wherein the step of replacing the original facial image of the target object in the second original image with the target facial image to obtain a driving result for the second original image comprises: Pasting the target facial image to the facial position of the target object in the second original image to obtain an updated image; If it is determined that the updated image satisfies the correction condition, the updated image is corrected to obtain a driving result for the second original image; If it is determined that the updated image does not satisfy the correction condition, the updated image is determined as a result of driving the second original image.

3. The method according to claim 2, wherein the correction condition comprises: The updated image contains a partial original facial image of the target object in the second original image; The step of correcting the updated image to obtain a driving result for the second original image includes any one of the following implementations: Using the non-face image in the updated image, a portion of the original face image of the target object existing in the updated image is corrected to obtain a driving result for the second original image; The image filling model is used to perform pixel filling on a portion of the original facial image of the target object where the updated image exists, to obtain a driving result for the second original image.

4. The method according to claim 3, wherein the use of the non-face area image in the updated image to correct the portion of the original face image of the target object in the updated image comprises at least one of the following implementations: Extracting background image data from the updated image, and filling the background image data into a target background area in a portion of the original facial image of the target object where the updated image exists; Based on the body image data or image filling model of the target object in the updated image, pixel filling is performed on the target body area in the partial original facial image of the target object in the updated image.

5. According to the method of any one of claims 1 to 4, the at least one first original image meets the three-dimensional facial modeling condition; the three-dimensional facial reconstruction of the target object based on the at least one first original image to obtain the three-dimensional facial model of the target object comprises: Extracting facial key point features and facial texture features of the target object from the at least one frame of the first original image; The three-dimensional facial model of the target object is constructed by using the facial key point features and the facial texture features.

6. The method according to claim 5, wherein the three-dimensional facial modeling conditions include: One frame of the first original image or a combination of multiple frames of the first original images capable of improving the complete facial features of the target object; The facial features include facial key point features and facial texture features of the target object; The step of obtaining at least one frame of a first original image containing a target object in the original video includes: Performing mouth movement tracking detection on each frame of the original image from the original video to obtain the mouth opening and closing angle of the target object in the original image containing the target object; A first original image corresponding to a frame in which the mouth opening and closing angle is less than an angle threshold is obtained.

7. The method according to any one of claims 1 to 4, wherein driving the three-dimensional facial model by the target voice comprises: Determine a second original image driven by the target voice in the original video; Extracting target facial perspective data of the target object in the second original image; According to the target voice and the target facial perspective data, the three-dimensional facial model is driven to perform corresponding actions to obtain the driven three-dimensional facial model of the target object.

8. The method according to claim 7, wherein the step of driving the three-dimensional facial model to perform corresponding actions based on the target voice and the target facial perspective data to obtain the driven three-dimensional facial model of the target object comprises: Determine, based on the original image and original speech of the target object contained in the original video, a correspondence between the mouth movement data of the target object in the original image and the original phoneme features of the original speech; Extracting features of the target speech to obtain target phoneme features; Determining target mouth movement data corresponding to the target phoneme feature according to the corresponding relationship; According to the target mouth action data, the three-dimensional facial model is driven to perform corresponding mouth actions, and according to the target facial perspective data, the three-dimensional facial model is driven to perform corresponding facial actions to obtain the driven three-dimensional facial model of the target object.

9. A voice-driven device for an image, the device comprising: A first original image acquisition module, used to acquire at least one frame of a first original image containing a target object in an original video; A three-dimensional facial reconstruction module, configured to perform three-dimensional facial reconstruction on the target object based on the at least one frame of the first original image to obtain a three-dimensional facial model of the target object; A target speech acquisition module, used to acquire a target speech; the target speech is used to drive a second original image in the original video, the second original image is different from the first original image; A driving module, used for driving the three-dimensional facial model through the target voice; A rendering module, used for rendering a target facial image for the target object according to the driven three-dimensional facial model; The replacement module is used to replace the original facial image of the target object in the second original image with the target facial image to obtain a driving result for the second original image.

10. An electronic device, comprising: at least one communication element, at least one memory, and at least one processor, wherein: The communication element is used to obtain the original video and the target voice; The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps: Acquire at least one frame of a first original image containing a target object in the original video; Reconstructing the three-dimensional face of the target object according to the at least one frame of the first original image to obtain a three-dimensional face model of the target object; Obtaining a target voice, wherein the target voice is used to drive a second original image in the original video, wherein the second original image is different from the first original image; Driving the three-dimensional facial model by the target voice, and rendering a target facial image for the target object according to the driven three-dimensional facial model; The target facial image is used to replace the original facial image of the target object in the second original image, so as to obtain a driving result for the second original image.

Images