AI virtual character and image processing method, system, electronic device and storage medium

By performing frame processing and AI attribute editing on digital human videos, combining 3D facial mask and expression model database training, the mask model is generated for video synthesis, which solves the problem that the existing AI face swap system has low processing efficiency and accuracy in virtual scenes. The problem of this achieves efficient and accurate digital image processing and video content production.

CN115423936BActive Publication Date: 2024-11-15CAMIKASI (TAICANG) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210977676.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-15
Publication Date
2024-11-15
Estimated Expiration
2042-08-15

AI Technical Summary

Technical Problem

The existing AI face swap system cannot process digital human images efficiently and accurately in virtual scenes, and the operation is cumbersome, which cannot meet the face swap and head swap requirements of Deep Fake and Deepfacelab. The detection performance is poor, resulting in AI virtual characters and image processing. The accuracy rate is not high.

Method used

By obtaining digital human videos or portrait datasets, processing and identifying face images in frames, generating AI digital human photos, performing AI attribute editing and gradient model processing of digital human photos, AI training is carried out based on 3D facial mask and expression model database, and generating Digital human mask model, used for video synthesis, achieving efficient and accurate face-changing video processing.

Benefits of technology

It realizes efficient and accurate face-changing video processing in AI virtual scenes, supports the face-changing, real-time face-changing, head-changing and person-changing functions of DeepFaceLab and Live software, and improves the production and live broadcast efficiency of digital virtual human video content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115423936B_ABST
    Figure CN115423936B_ABST
Patent Text Reader

Abstract

The present invention relates to an AI virtual character and an image processing method, system, electronic device and storage medium, and to the field of computer vision technology. The method is to obtain a target digital human source video or a target digital human portrait data set, perform AI training on a variable model adaptation result based on a loaded 3D facial makeup model database and a 3D expression model database, and generate a digital human mask model; obtain a target video, and input the target video into the digital human mask model for video synthesis, and then output the processed AI virtual scene image video. The present invention uses AI to perform attribute editing and digital human gradient model adaptation with a 3D expression model database and a 3D facial makeup model database, and performs AI training based on a 3D facial makeup model, expression parameters and posture parameters, and performs video synthesis on the target video according to the obtained digital human mask model, so as to realize efficient, accurate and fast face-changing image processing in different target video scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an AI virtual character and image processing method, system, electronic device and storage medium. Background Art

[0002] With the continuous development and research of artificial intelligence technology, artificial intelligence has achieved the recognition and understanding of human faces and natural language. It can convert the received voice into text, and can also intelligently identify facial features based on the collected facial images, laying the foundation for the rapid development of artificial intelligence technology.

[0003] Moreover, with the development of information technology and artificial intelligence technology, there are more and more technologies for AI face-changing, head-changing, and video content production for characters (virtual digital people). AI face-changing APPs are also becoming one of the technologies for short video production, and the effect of AI face-changing is getting better and better. However, as an application of AI face-changing special effects in virtual scene image processing, the current ordinary AI face-changing system can only perform facial recognition and feature identification on real people alone. Due to the lack of material library, it cannot be applied to digital people. Moreover, in the current ordinary AI face-changing system, digital people can generally only generate a photo and a 3D model. Some software integrates a small amount of expressions and postures (such as MetaHuman Creator Epic Games), which cannot meet the face-changing and head-changing realization of Deep Fake and Deepfacelab. Digital people lack materials, and the operation is cumbersome and time-consuming, which is not conducive to image processing, and thus cannot efficiently and accurately realize face-changing videos.

[0004] Currently, the public AI face-changing video detection only uses a single feature for detection, such as blinking, mouth shape, or skin edge, which results in poor detection performance. During the calculation process, most detection methods consider the entire image, which greatly increases the calculation cost, making it impossible to efficiently and accurately implement AI virtual characters and image processing, and also results in the accuracy of AI virtual characters and image processing being often low. Summary of the invention

[0005] The embodiments of the present application provide an AI virtual character and image processing method, system, electronic device and storage medium, which can improve the accuracy and efficiency of image processing during AI face-changing performances in AI virtual scenes.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides an AI virtual character and image processing method, including:

[0008] Obtain a target digital person source video or a target digital person portrait data set to be processed, perform frame processing on the target digital person source video or the target digital person portrait data set and recognize the face image to generate an AI digital person photo;

[0009] Performing AI attribute editing on the digital human photo, and inputting the digital human photo into the digital human gradient model;

[0010] The results of AI attribute editing and digital human gradient model processing are input into the initial digital human 3D facial model for variable model adaptation;

[0011] Based on the loaded 3D facial model database and 3D expression model database, AI training is performed on the variable model adaptation results to generate a digital human mask model;

[0012] The target video is obtained, and the target video is input into the digital human mask model for video synthesis, and then the processed AI virtual scene image video is output.

[0013] In a second aspect, the embodiment of the present application further provides an AI virtual character and image processing system, the system comprising:

[0014] A digital human generation module, used to edit the digital human photo AI attributes and input the digital human photo into the digital human gradient model;

[0015] The analysis module is used to input the results of AI attribute editing and digital human gradient model processing into the initial model of the digital human 3D face makeup for variable model adaptation;

[0016] A training module, used for performing AI training on the adaption results of the variable model based on the loaded 3D facial model database and the 3D expression model database to generate a digital human mask model; and

[0017] The synthesis module is used to obtain the target video, input the target video into the digital human mask model for video synthesis, and output the processed AI virtual scene image video.

[0018] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned AI virtual character and image processing method when executing the program.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned AI virtual character and image processing method.

[0020] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0021] The embodiment of the present application provides an AI virtual character and image processing solution, which obtains a target digital human source video or a target digital human portrait data set to be processed, performs frame processing on the target digital human source video or the target digital human portrait data set and recognizes the face image to generate an AI digital human photo; performs digital human photo AI attribute editing on the digital human photo, and inputs the digital human photo into a digital human gradient model; inputs the results of AI attribute editing and digital human gradient model processing into a digital human 3D facial makeup initial model for variable model adaptation; performs AI training on the variable model adaptation results based on a loaded 3D facial makeup model database and a 3D expression model database to generate a digital human mask model; obtains a target video, and inputs the target video into the digital human mask model for video synthesis, and then outputs the processed AI virtual scene image video.

[0022] The present invention simulates the expressions and postures of digital people through artificial intelligence to obtain "source video" materials similar to real people, and then applies them to DeepFaceLab's 3.0 and Live software to perform face-changing, real-time face-changing, head-changing, and person-changing, thereby realizing the production and live broadcast of digital virtual human video content. The present invention adopts the above technical solution, uses AI to perform attribute editing and digital human gradient model adaptation with 3D expression model database and 3D facial model database, and performs AI training based on 3D facial model, expression parameters and posture parameters, and performs video synthesis on the target video according to the obtained digital human mask model, without the need to perform facial recognition and feature identification separately for the source video and the target video, and can realize efficient, accurate and fast face-changing image processing in different target video scenes.

[0023] These and other aspects of the present application will be more concise and understandable in the following description of the embodiments. It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. In the drawings:

[0025] Figure 1 A flowchart schematically illustrates an AI virtual character and image processing method in an exemplary embodiment of the present invention;

[0026] Figure 2 Schematically illustrates the software architecture and flow chart of an AI digital human 3D face-changing performance of an AI virtual character and an image processing method in an exemplary embodiment of the present invention;

[0027] Figure 3 A flowchart schematically illustrates secondary development of an AI digital human in an AI virtual character and an image processing method in an exemplary embodiment of the present invention;

[0028] Figure 4 Schematically shown Figure 3 A schematic diagram of an application scenario of secondary development of an AI digital human in an AI virtual character and an image processing method in an exemplary embodiment of the present invention;

[0029] Figure 5 It is a structural block diagram of the AI ​​virtual character and image processing system of the present invention;

[0030] Figure 6 A hardware architecture diagram of an electronic device in some embodiments of the present invention;

[0031] Figure 7 The integrated schematic diagram of the digital human audio voice changing system in an AI virtual character and image processing system in an exemplary embodiment of the present invention is schematically shown.

[0032] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0033] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0034] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0037] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0038] In the related technology, since the current ordinary AI face-changing system can only perform facial recognition and feature identification on real people separately, it cannot be applied to digital people due to the lack of material library. Then, the video is synthesized after the parameters are covered by the training model. The operation is cumbersome, time-consuming and labor-intensive, and is not conducive to image processing, so it is impossible to efficiently and accurately realize face-changing videos.

[0039] Currently, the public AI face-changing video detection only uses a single feature for detection, such as blinking, mouth shape, or skin edge, which results in poor detection performance. During the calculation process, most detection methods consider the entire image, which greatly increases the calculation cost, making it impossible to efficiently and accurately implement AI virtual characters and image processing, and also results in the accuracy of AI virtual characters and image processing being often low.

[0040] In view of this, the present application provides an AI virtual character and image processing method, system, electronic device and storage medium, which can improve the accuracy and efficiency of image processing during AI face-changing performances in AI virtual scenes.

[0041] Figure 1 The flowchart of an AI virtual character and image processing method provided in an embodiment of the present application can be performed by an AI virtual character and image processing system, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1 As shown, the method includes:

[0042] Step S10: obtaining a target digital human source video or a target digital human portrait data set to be processed, performing frame processing on the target digital human source video or the target digital human portrait data set and recognizing the face image to generate an AI digital human photo.

[0043] The source video to be processed includes an image video containing a target user captured or recorded by a video capture camera or other device.

[0044] In the embodiment of the present application, a camera is connected to the integrated electronic device, and the source video of the target digital person to be processed or the portrait data set of the target digital person can be collected and obtained through the camera. It can also be that when the user's source video is detected to be imported into a file, the source video is read by the electronic device, and the read source video is sent to the processor through the network. It can be understood that the above-mentioned method of collecting and obtaining the source video of the target digital person to be processed or the portrait data set of the target digital person is an example rather than a limitation.

[0045] Step S20: Edit the digital human photo AI attributes and input the digital human photo into the digital human gradient model.

[0046] Among them, when AI training is performed on the variable model adaptation result based on the loaded 3D facial model database and 3D expression model database, the AI ​​training includes the 3D facial model, expression parameters and posture parameters.

[0047] Step S30: The results of AI attribute editing and digital human gradient model processing are input into the digital human 3D facial makeup initial model for variable model adaptation.

[0048] Step S40: Perform AI training on the variable model adaptation result based on the loaded 3D facial model database and 3D expression model database to generate a digital human mask model.

[0049] In some embodiments of the present application, when AI training is performed on the variable model adaptation result based on the loaded 3D facial model database and 3D expression model database, it also includes:

[0050] The AI-trained 3D facial model, expression parameters, and posture parameters are input into the digital human 3D facial AI training model to update the 3D facial model database.

[0051] In some embodiments of the present application, when AI training is performed on the variable model adaptation result based on the loaded 3D facial model database and 3D expression model database, it also includes:

[0052] The AI-trained 3D facial model, expression parameters, and posture parameters are input into the digital human 3D expression AI training model to update the 3D expression model database.

[0053] Step S50, obtaining a target video, and inputting the target video into a digital human mask model for video synthesis, and then outputting a processed AI virtual scene image video.

[0054] In some embodiments of the present application, the digital human photo AI attribute editing is performed on the digital human photo, including:

[0055] Load the AI ​​digital human photo generated after the source video is frame-processed and the face image is recognized;

[0056] Performing facial recognition on the AI ​​digital human photo in the source video to generate feature identification;

[0057] Perform facial expression recognition on the AI ​​digital human photo in the source video to generate posture parameters and expression parameters.

[0058] In some embodiments of the present application, before obtaining a target video and inputting the target video into a digital human mask model for video synthesis, it also includes performing expression recognition on the target video. The expression recognition method includes the following steps:

[0059] Extracting facial expression features from the acquired target video;

[0060] Based on the expression model database and the extracted expression features, the expression analysis network is input to perform posture extraction and label extraction;

[0061] The posture parameters and expression parameters corresponding to the target video are generated according to the results of posture extraction and label extraction.

[0062] The AI ​​virtual person and image processing method of the embodiment of the present invention obtains the target digital human source video or the portrait data set of the target digital human to be processed, performs frame processing on the target digital human source video or the portrait data set of the target digital human and recognizes the face image to generate an AI digital human photo; performs AI attribute editing on the digital human photo, and inputs the digital human photo into the digital human gradient model; the result after AI attribute editing and digital human gradient model processing is input into the digital human 3D facial makeup initial model for variable model adaptation; AI training is performed on the variable model adaptation result based on the loaded 3D facial makeup model database and 3D expression model database to generate a digital human mask model; obtains the target video, and inputs the target video into the digital human mask model for video synthesis, and then outputs the processed AI virtual scene image video. The present invention simulates the expression, posture, etc. of the digital human through artificial intelligence to obtain the "source video" material similar to that of a real person, and then applies it to DeepFaceLab's 3.0 and Live software to perform face change, real-time face change, head change, and person change, thereby realizing the production and live broadcast of digital virtual human video content. The present invention adopts the above-mentioned technical scheme, performs attribute editing through AI and adapts the digital human gradient model to the 3D expression model database and the 3D facial model database, and performs AI training based on the 3D facial model, expression parameters and posture parameters. The target video is synthesized according to the obtained digital human mask model. There is no need to perform facial recognition and feature identification separately for the source video and the target video, and face-changing image processing in different target video scenes can be realized efficiently, accurately and quickly.

[0063] It should be understood that, although described in a certain order, these steps are not necessarily performed in sequence in the above order. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a part of the steps or stages in other steps or other steps.

[0064] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0065] Figure 5 1 is a schematic diagram of the structure of an AI virtual character and image processing system provided in an embodiment of the present application. The system can be implemented by software and / or hardware and is generally integrated in an electronic device. Figure 2 and Figure 5 As shown, the system may include:

[0066] A digital human generation module 100 is used to edit the digital human photo AI attributes and input the digital human photo into a digital human gradient model;

[0067] The analysis module 200 is used to input the results of AI attribute editing and digital human gradient model processing into the initial model of the digital human 3D face makeup to perform variable model adaptation;

[0068] A training module 300 is used to perform AI training on the variable model adaptation result based on the loaded 3D facial model database and 3D expression model database to generate a digital human mask model; and

[0069] The synthesis module 400 is used to obtain a target video, input the target video into a digital human mask model for video synthesis, and output a processed AI virtual scene image video.

[0070] The technical solution of the embodiment of the present application provides an AI virtual character and image processing system, which uses AI to perform attribute editing and digital human gradient model adaptation with 3D expression model database and 3D facial model database, and performs AI training based on 3D facial model, expression parameters and posture parameters. The target video is synthesized according to the obtained digital human mask model, without the need to perform facial recognition and feature identification for the source video and the target video respectively, and can achieve efficient, accurate and fast face-changing image processing in different target video scenes.

[0071] In some embodiments, Figure 3 , Figure 4 and Figure 5 As shown, the AI ​​virtual character and image processing system also includes an expression recognition module 500, which is used to extract expression features from the acquired target video, input the expression analysis network to perform posture extraction and label extraction based on the expression model database and the extracted expression features, and generate posture parameters and expression parameters corresponding to the target video according to the results of the posture extraction and label extraction.

[0072] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on at least two network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application. Those of ordinary skill in the art may understand and implement them without creative effort.

[0073] In some embodiments, Figure 7 As shown, the AI ​​virtual character and image processing system is also integrated with a digital human audio voice changing system, which is a voice changing system based on AI generation and learning of digital virtual character images and audio. The source sound audio is processed by combining the AI ​​audio generation system with the text voice system, and the source sound audio is processed based on the target video of a real person performing according to the script. The audio required for the target video is output after processing by the voice changing system, thereby realizing the AI ​​digital human 3D face-changing performance and voice changing processing at the same time.

[0074] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0075] This embodiment also provides an electronic device, such as Figure 6 As shown, the electronic device includes multiple electronic devices 1000. In the embodiment, the components of the AI ​​virtual character and the image processing system can be dispersed in different electronic devices 1000. The electronic device 1000 can be a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server, or a server cluster composed of multiple servers) that executes the program. The electronic device 1000 of this embodiment includes at least but is not limited to: a memory 1001 and a processor 1002 that can communicate with each other through a system bus. It should be pointed out that Figure 6 Only the electronic device 1000 having components memory 1001 and processor 1002 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0076] In this embodiment, the memory 1001 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 1001 may be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. In other embodiments, the memory 1001 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1000. Of course, the memory 1001 may also include both the internal storage unit of the electronic device 1000 and its external storage device. In this embodiment, the memory 1001 is generally used to store operating systems and various application software installed on electronic devices, such as the AI ​​virtual characters and image processing systems of the embodiments. In addition, the memory 1001 can also be used to temporarily store various types of data that have been output or are to be output.

[0077] In some embodiments, the processor 1002 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1002 is generally used to control the overall operation of the electronic device 1000. In this embodiment, the processor 1002 is used to run the program code stored in the memory 1001 or process data. When the processors 1002 of multiple electronic devices 1000 of the electronic device of this embodiment jointly execute the computer program, the AI ​​virtual character and image processing method of the embodiment is implemented, and the method includes:

[0078] Obtain a target digital person source video or a target digital person portrait data set to be processed, perform frame processing on the target digital person source video or the target digital person portrait data set and recognize the face image to generate an AI digital person photo;

[0079] Performing AI attribute editing on the digital human photo, and inputting the digital human photo into the digital human gradient model;

[0080] The results of AI attribute editing and digital human gradient model processing are input into the initial digital human 3D facial model for variable model adaptation;

[0081] Based on the loaded 3D facial model database and 3D expression model database, AI training is performed on the variable model adaptation results to generate a digital human mask model;

[0082] The target video is obtained, and the target video is input into the digital human mask model for video synthesis, and then the processed AI virtual scene image video is output.

[0083] Through the description of the above embodiments, ordinary technicians in this field can clearly understand that each embodiment can be implemented by software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above embodiment methods can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-matchable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods.

[0084] The embodiments of the present application also provide a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and the program implements the corresponding function when executed by the processor. The computer-readable storage medium of this embodiment stores the AI ​​virtual character and image processing system 10 of the embodiment, and when executed by the processor, implements the AI ​​virtual character and image processing method of the embodiment, which includes:

[0085] Obtain a target digital person source video or a target digital person portrait data set to be processed, perform frame processing on the target digital person source video or the target digital person portrait data set and recognize the face image to generate an AI digital person photo;

[0086] Performing AI attribute editing on the digital human photo, and inputting the digital human photo into the digital human gradient model;

[0087] The results of AI attribute editing and digital human gradient model processing are input into the initial digital human 3D facial model for variable model adaptation;

[0088] Based on the loaded 3D facial model database and 3D expression model database, AI training is performed on the variable model adaptation results to generate a digital human mask model;

[0089] The target video is obtained, and the target video is input into the digital human mask model for video synthesis, and then the processed AI virtual scene image video is output.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0091] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDORAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0092] Of course, the storage medium containing computer executable instructions provided in the embodiments of the present application is not limited to the AI ​​virtual character and image processing operations described above, but can also execute related operations in the AI ​​virtual character and image processing method provided in any embodiment of the present application.

[0093] The AI ​​virtual character and image processing method, system, electronic device and storage medium provided by the present invention obtain a target digital human source video to be processed or a target digital human portrait data set, perform frame processing on the target digital human source video or the target digital human portrait data set and recognize the face image to generate an AI digital human photo; perform digital human photo AI attribute editing on the digital human photo, and input the digital human photo into a digital human gradient model; input the result after AI attribute editing and digital human gradient model processing into a digital human 3D facial makeup initial model for variable model adaptation; perform AI training on the variable model adaptation result based on a loaded 3D facial makeup model database and a 3D expression model database to generate a digital human mask model; obtain a target video, and input the target video into the digital human mask model for video synthesis, and then output the processed AI virtual scene image video.

[0094] The present invention simulates the expressions and postures of digital people through artificial intelligence to obtain "source video" materials similar to real people, and then applies them to DeepFaceLab's 3.0 and Live software to perform face-changing, real-time face-changing, head-changing, and person-changing, thereby realizing the production and live broadcast of digital virtual human video content. The present invention adopts the above technical solution, uses AI to perform attribute editing and digital human gradient model adaptation with 3D expression model database and 3D facial model database, and performs AI training based on 3D facial model, expression parameters and posture parameters, and performs video synthesis on the target video according to the obtained digital human mask model, without the need to perform facial recognition and feature identification separately for the source video and the target video, and can realize efficient, accurate and fast face-changing image processing in different target video scenes.

[0095] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An AI virtual character and image processing method, characterized in that: include: Obtain a target digital person source video or a target digital person portrait data set to be processed, perform frame processing on the target digital person source video or the target digital person portrait data set and recognize the face image to generate an AI digital person photo; Performing AI attribute editing on the digital human photo, and inputting the digital human photo into the digital human gradient model; The results of AI attribute editing and digital human gradient model processing are input into the initial digital human 3D facial model for variable model adaptation; Based on the loaded 3D facial model database and 3D expression model database, AI training is performed on the variable model adaptation results to generate a digital human mask model; The target video is obtained, and the target video is input into the digital human mask model for video synthesis, and then the processed AI virtual scene image video is output.

2. The AI ​​virtual character and image processing method according to claim 1, characterized in that: When AI training is performed on the variable model adaptation result based on the loaded 3D facial model database and the 3D expression model database, the AI ​​training includes the 3D facial model, expression parameters, and posture parameters.

3. The AI ​​virtual character and image processing method according to claim 2, characterized in that: When AI training is performed on the variable model adaptation results based on the loaded 3D facial model database and 3D expression model database, it also includes: The AI-trained 3D facial model, expression parameters, and posture parameters are input into the digital human 3D facial AI training model to update the 3D facial model database.

4. The AI ​​virtual character and image processing method according to claim 2, characterized in that: When AI training is performed on the variable model adaptation results based on the loaded 3D facial model database and 3D expression model database, it also includes: The AI-trained 3D facial model, expression parameters, and posture parameters are input into the digital human 3D expression AI training model to update the 3D expression model database.

5. The AI ​​virtual character and image processing method according to claim 1, characterized in that: Editing the digital human photo AI attributes of the digital human photo includes: Load the AI ​​digital human photo generated after the source video is frame-processed and the face image is recognized; Performing facial recognition on the AI ​​digital human photo in the source video to generate feature identification; Perform facial expression recognition on the AI ​​digital human photo in the source video to generate posture parameters and expression parameters.

6. The AI ​​virtual character and image processing method according to claim 1 or 5, characterized in that: The target video is obtained, and before the target video is input into the digital human mask model for video synthesis, expression recognition is performed on the target video. The expression recognition method includes the following steps: Extracting facial expression features from the acquired target video; Based on the expression model database and the extracted expression features, the expression analysis network is input to perform posture extraction and label extraction; The posture parameters and expression parameters corresponding to the target video are generated according to the results of posture extraction and label extraction.

7. An AI virtual character and image processing system, characterized in that: Based on the AI ​​virtual character and image processing described in any one of claims 1 to 6, AI digital human 3D face-changing performance processing is performed, and the AI ​​virtual character and image processing system includes: A digital human generation module, used to edit the digital human photo AI attributes and input the digital human photo into the digital human gradient model; The analysis module is used to input the results of AI attribute editing and digital human gradient model processing into the initial model of the digital human 3D face makeup for variable model adaptation; A training module, used for performing AI training on the adaption results of the variable model based on the loaded 3D facial model database and the 3D expression model database to generate a digital human mask model; and The synthesis module is used to obtain the target video, input the target video into the digital human mask model for video synthesis, and output the processed AI virtual scene image video.

8. The AI ​​virtual character and image processing system according to claim 7, characterized in that: Also includes: The expression recognition module is used to extract expression features from the acquired target video, input the expression analysis network to perform posture extraction and label extraction based on the expression model database and the extracted expression features, and generate posture parameters and expression parameters corresponding to the target video according to the results of posture extraction and label extraction.

9. An electronic device, characterized in that: The electronic device includes multiple electronic devices, each of which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. It is characterized in that the processors of the multiple electronic devices jointly implement the steps of the AI ​​virtual character and image processing method described in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program stored in the storage medium implements the steps of the AI ​​virtual character and image processing method described in any one of claims 1 to 6 when executed by a processor.