An AI matting method, device and equipment

Abstract

The application discloses an AI matting method, device and equipment, and relates to the technical field of image processing, and is used for solving the problems of low matting precision and poor matting efficiency in the prior art. The application comprises the following steps: acquiring to-be-recognized video image information; the to-be-recognized video image information comprises image information collected by a camera; a trained face recognition model is used to recognize the to-be-recognized video image information, so that foreground portrait information in the to-be-recognized video image information is obtained; the foreground portrait information is provided with an alpha channel; the foreground portrait information provided with the alpha channel is drawn to a rendering target and is transmitted to a UE engine; and the UE engine displays the foreground portrait through a dynamic material instance. The AI matting method provided by the application can realize video portrait matting, and through a deep network with an adversarial loss and a background switching block codec, AI matting technology is realized in an Unreal engine, so that the matting precision and the matting efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an AI cutout method, device and equipment. BACKGROUND

[0002] Portrait cutout refers to identifying the portrait area in an image, including head, half body and full body positions, accurately predicting the opacity between the portrait and the background, and then matching different background pictures and effects for fusion. Cutout is used to extract the foreground object of interest in an image, and is a basic technology in the field of image and video processing. Its technical achievements are mainly used in image processing, video and film industry, and become a crucial technical link.

[0003] In the field of image processing, foreground cutout is a commonly used processing method. Foreground cutout refers to extracting the region of interest (foreground) in an image, obtaining a fine transparency mask, and using the transparency mask to extract the cutout object from the image or video, so as to apply the cutout object to photo editing and film re-creation. For any background, the cutout has always been a very tricky problem in the video industry. The first is the accuracy of the cutout, and the second is the efficiency of the cutout. This technology works well in scenes where the portrait background is relatively monotonous, but it performs poorly in places where the background color is similar to the portrait.

[0004] Therefore, it is urgent to provide a more reliable AI cutout method. SUMMARY

[0005] The present application aims to provide an AI cutout method, device and equipment, which solves the problems of low cutout accuracy and poor cutout efficiency in the prior art.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] In a first aspect, the present application provides an AI cutout method, comprising:

[0008] Obtaining to-be-identified video image information; the to-be-identified video image information includes image information collected by a camera;

[0009] Using a trained face recognition model to identify the to-be-identified video image information, to obtain foreground portrait information in the to-be-identified video image information; the foreground portrait information has an alpha channel;

[0010] Drawing the foreground portrait information with the alpha channel to a rendering target, and transmitting it to a UE engine; the UE engine displays the foreground portrait through a dynamic material instance.

[0011] Optionally, before the step of using the trained face recognition model to recognize the to-be-recognized video image information to obtain foreground portrait information in the to-be-recognized video image information, the method further comprises:

[0012] obtaining video image information containing a face image and a background image, portrait soft segmentation information, and motion prior information;

[0013] inputting the video image information, the portrait soft segmentation information, and the motion prior information into an initial face recognition model to obtain a mask with an alpha channel and a foreground image;

[0014] comparing the mask with the alpha channel and the foreground image with known information to obtain a comparison result;

[0015] adjusting parameters of the initial face recognition model using the comparison result to obtain a trained face recognition model.

[0016] Optionally, the trained face recognition model contains four different encoders, respectively corresponding to the video image information containing the face image, the video image information containing the background image, the portrait soft segmentation information, and the motion prior information.

[0017] Optionally, each encoder generates a feature map of 256 channels;

[0018] image features in the video image information containing the face image are combined with the video image information containing the background image, the portrait soft segmentation information, and the motion prior information through 1x1 convolution, a BN layer, and a ReLU layer, and each pair of combinations generates a 64-channel feature;

[0019] the three 64-channel features are combined with original 256-channel image features to generate encoded features and are transmitted to the remaining network composed of residual blocks and encoders.

[0020] Optionally, before the step of drawing the foreground portrait information with the alpha channel to a rendering target and transmitting the foreground portrait information to a UE engine, the method further comprises:

[0021] creating a rendering target of a color language protocol;

[0022] asynchronously sending the collected video image information into the trained face recognition model;

[0023] using parallel computing technology of a parallel computing architecture to perform GPU acceleration on the trained face recognition model.

[0024] Optionally, the step of drawing the foreground portrait information with the alpha channel to a rendering target and transmitting the foreground portrait information to a UE engine specifically comprises:

[0025] Determine target background information;

[0026] Superimpose the foreground portrait information and the target background information to obtain an AI synthesized image.

[0027] Optionally, superimposing the foreground portrait information and the target background information to obtain an AI synthesized image specifically includes:

[0028] Superimposing the foreground portrait information and the target background information, and performing a blurring operation on the figure outline and the hair part to obtain an AI synthesized image.

[0029] Compared with the prior art, the AI matting method provided by the present application can realize video portrait matting by using a deep network with an adversarial loss and a background switching block codec to improve the matting accuracy and efficiency.

[0030] In the second aspect, the present application provides an AI matting device, which comprises:

[0031] The video image information to be identified is obtained by the video image information to be identified acquisition module, and the video image information to be identified includes image information collected by a camera.

[0032] The foreground portrait information is determined by the foreground portrait information determination module by using a trained face recognition model to identify the video image information to be identified, and the foreground portrait information has an alpha channel.

[0033] The image rendering module is used to draw the foreground portrait information with the alpha channel to a rendering target and transmit it to a UE engine, and the UE engine displays the foreground portrait by using a dynamic material instance.

[0034] The technical effects of the device are the same as those of the AI matting method of the first aspect, and will not be repeated here.

[0035] In the third aspect, the present application provides an AI matting equipment, which comprises:

[0036] A communication unit / communication interface is configured to acquire to-be-identified video image information; the to-be-identified video image information comprises image information collected by a camera;

[0037] A processing unit / processor is configured to identify the to-be-identified video image information by using a trained face recognition model, to obtain foreground portrait information in the to-be-identified video image information; the foreground portrait information is provided with an alpha channel;

[0038] The foreground portrait information provided with the alpha channel is drawn to a rendering target and transmitted to a UE engine; the UE engine displays the foreground portrait by using a dynamic material instance.

[0039] The technical effect of the device is the same as that of the AI cutout method of the first aspect, which will not be repeated here.

[0040] In a fourth aspect, the present application provides a computer storage medium, wherein instructions are stored in the computer storage medium, and when the instructions are executed, the AI cutout method described above is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0042] Figure 1 A flowchart of an AI cutout method provided by the present application;

[0043] Figure 2 A portrait channel graph generated by an AI cutout image in the AI cutout method provided by the present application;

[0044] Figure 3 A structure diagram of an AI cutout device provided by the present application;

[0045] Figure 4 A structure diagram of an AI cutout device provided by the present application. DETAILED DESCRIPTION

[0046] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and the order is not limited. Those skilled in the art can understand that "first", "second", etc. do not limit the number and execution order, and "first", "second", etc. also do not necessarily mean different.

[0047] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the exemplary or for example embodiments are intended to convey concepts in a concrete form when possible.

[0048] In the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of the associated objects is described, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, wherein a, b and c can be single or multiple.

[0049] For any background matting, it has always been a very tricky problem in the video industry. The first is the problem of matting accuracy, and the second is the problem of matting efficiency. The technology works well in scenes with relatively monotonous portrait backgrounds, but works poorly in places where the background and portrait colors are similar.

[0050] Unlike the traditional arbitrary background matting technology, which uses light processing (i.e., using a predefined Trimap), AI matting uses a lightweight target decomposition network MODNet (Matting Objective Decomposition Network), which can smoothly process dynamic portraits from single input images with different backgrounds. In general, the function of the present application is video portrait matting, which is an AI matting technology implemented in Unreal Engine through a deep network with adversarial loss plus a background switching block codec.

[0051] To this end, the present application provides a reliable AI matting scheme.

[0052] Next, the scheme provided by the embodiments of the present application will be described in conjunction with the accompanying drawings:

[0053] Figure 1 An AI matting method flowchart provided by the present application is shown in Figure 1 The flowchart can include the following steps:

[0054] Step 110: obtaining to-be-recognized video image information; the to-be-recognized video image information includes image information collected by a camera.

[0055] The to-be-recognized video image information can be a picture of an external camera in a media player or a camera video picture obtained through an SDI signal.

[0056] Step 120: recognizing the to-be-recognized video image information by using a trained face recognition model to obtain foreground portrait information in the to-be-recognized video image information; the foreground portrait information has an alpha channel.

[0057] The trained face recognition model can recognize a face portrait in an image and distinguish the portrait from background information.

[0058] The face recognition model can be a face deep learning model, and formats such as ONNX and TMX can be used.

[0059] Step 130: drawing the foreground portrait information with the alpha channel to a rendering target and transmitting the foreground portrait information to a UE engine; the UE engine displays the foreground portrait by using a dynamic material instance.

[0060] The portrait picture with the alpha channel is transmitted to the UE engine, and the UE creates a dynamic material instance for the model by using the channel to display a matting effect in real time.

[0061] Figure 1 The method in the background can realize AI matting by using a lightweight Matting Objective Decomposition Network (MODNet), which can smoothly process dynamic portraits from a single input image with different backgrounds. The scheme of the present application can realize video portrait matting, which is an AI matting technology realized in an Unreal engine by using a deep network with an adversarial loss and a background switching block codec. The AI matting technology can obtain satisfactory AI synthesis effects in real time (30 FPS), improve matting accuracy and efficiency.

[0062] Based on the method in the background, Figure 1 The embodiments of the present application also provide some specific implementations of the method, which are described below.

[0063] Optionally, before the to-be-recognized video image information is recognized by using the trained face recognition model to obtain the foreground portrait information in the to-be-recognized video image information, the method can further include the following steps.

[0064] Obtaining video image information containing a face image and a background image, portrait soft segmentation information and motion prior information;

[0065] inputting the video image information, the portrait soft segmentation information and the motion prior information into an initial face recognition model to obtain a mask with an alpha channel and a foreground image;

[0066] comparing the mask with the alpha channel and the foreground image with known information to obtain a comparison result;

[0067] adjusting parameters of the initial face recognition model by using the comparison result to obtain a trained face recognition model.

[0068] For a video image, a deep neural network G can be supervised trained on a non-transparent object subset in an Adobe Matting dataset. The input is a photo I with a portrait and a background B' in the photo, and portrait soft segmentation S and motion prior M. According to the input, the network can predict an alpha mask a and a foreground image F.

[0069] A background switching block replaces a residual block-based encoder-decoder: When a part of a person matches the background, the network can pay more attention to the subdivision clues in the area. The G network has four different encoders, respectively for the four inputs of I, B', S and M. Each encoder generates a 256-channel feature map. Through 1*1 convolution, BatchNorm and ReLU, the image features in I are combined with B', S and M respectively, and each pair of combinations generates a 64-channel feature. Finally, the three 64-channel features are combined with the original 256-channel image features to generate encoded features, which are transmitted to the remaining network composed of residual blocks and encoders.

[0070] Optionally, the trained face recognition model includes four different encoders, respectively corresponding to the video image information containing a face image, the video image information containing a background image, the portrait soft segmentation information and the motion prior information.

[0071] Optionally, each encoder generates a 256-channel feature map;

[0072] Through 1*1 convolution, BN layer and ReLU layer, the image features in the video image information containing a face image are combined with the video image information containing a background image, the portrait soft segmentation information and the motion prior information respectively, and each pair of combinations generates a 64-channel feature. The three 64-channel features are combined with the original 256-channel image features to generate encoded features, which are transmitted to the remaining network composed of residual blocks and encoders.

[0073] Optionally, before drawing the foreground portrait information with the alpha channel to a rendering target and transmitting it to a UE engine, the following steps can be further included:

[0074] creating a render target of a color language protocol;

[0075] The collected video image information is sent into the trained face recognition model asynchronously;

[0076] The trained face recognition model is accelerated by GPU using parallel computing technology of parallel computing architecture.

[0077] sRGB (standard Red Green Blue) is a color language protocol jointly developed by Microsoft image giants, and sRGB represents standard red, green and blue, that is, three basic pigments used for color reproduction in CRT displays, LCD panels, projectors, printers and other devices. The color space of sRGB is based on independent color coordinates, which can make colors correspond to the same color coordinate system in different device transmission, and is not affected by the different color coordinates of these devices.

[0078] Optionally, the foreground portrait information with the alpha channel is drawn to the render target and transmitted to the UE engine, which can specifically include:

[0079] determining target background information;

[0080] superimposing the foreground portrait information and the target background information to obtain an AI synthesis image.

[0081] Optionally, superimposing the foreground portrait information and the target background information to obtain an AI synthesis image can specifically include:

[0082] superimposing the foreground portrait information and the target background information, and performing a blurring operation on the character outline and the hair part to obtain an AI synthesis image.

[0083] First, an sRGB render target is created, and then data streams are sent into the above deep learning API asynchronously. At the same time, the entire deep learning framework is accelerated by GPU using the Cuda parallel computing technology of Nvidia, and the matting and background superposition are performed, so that a satisfactory AI synthesis effect can be obtained in real time (30 FPS).

[0084] The screen of an external camera in a media player or the screen of a camera obtained through an SDI signal is acquired, and the screen is rendered to a render target (RenderTarget); for example, Figure 2As shown, in the AI cutout generated portrait channel, the left figure is the source image, and the right figure is the pha image. In the specific implementation, the model of face recognition learning is established, the pha of the image is analyzed through the model, and then the pha is synthesized with the source image. During the synthesis, the outline of the character, especially the hair part, is blurred, so as to achieve a relatively soft effect. The final imaging effect depends on the performance of the deep learning model.

[0085] The portrait image with the Alpha channel is transmitted to the UE engine, and the channel creates a dynamic material instance for the model to display the cutout effect in real time.

[0086] Based on the same idea, the application also provides an AI cutout device, such as Figure 3 As shown, the device can include:

[0087] The to-be-identified video image information acquisition module 310 is configured to acquire to-be-identified video image information; the to-be-identified video image information includes image information collected by a camera;

[0088] The foreground portrait information determination module 320 is configured to identify the to-be-identified video image information by using a trained face recognition model, to obtain foreground portrait information in the to-be-identified video image information; the foreground portrait information has an Alpha channel;

[0089] The image rendering module 330 is configured to draw the foreground portrait information with the Alpha channel to a rendering target and transmit the foreground portrait information to a UE engine; the UE engine displays the foreground portrait by using a dynamic material instance.

[0090] Based on the device in Figure 3 The device can further include some specific implementation units:

[0091] Optionally, the device can further include:

[0092] The input information acquisition module is configured to acquire video image information including a face image and a background image, portrait soft segmentation information, and motion prior information;

[0093] The identification module is configured to input the video image information, the portrait soft segmentation information, and the motion prior information into an initial face recognition model, to obtain a mask with an Alpha channel and a foreground image;

[0094] The comparison module is configured to compare the mask with the Alpha channel and the foreground image with known information, to obtain a comparison result;

[0095] The model training module is configured to adjust parameters of the initial face recognition model by using the comparison result, to obtain a trained face recognition model.

[0096] Optionally, the trained face recognition model can include four different encoders, respectively corresponding to video image information containing a face image, video image information containing a background image, portrait soft segmentation information, and motion prior information.

[0097] Optionally, each encoder generates a feature map of 256 channels;

[0098] Through 1x1 convolution, BN layer and ReLU layer, the image features in the video image information containing the face image are combined with the video image information containing the background image, the portrait soft segmentation information, and the motion prior information, respectively. Each pair of combinations generates a 64-channel feature.

[0099] The three 64-channel features are combined with the original 256-channel image features to generate encoded features, which are passed into the remaining network composed of residual blocks and encoders.

[0100] Optionally, the device can further include:

[0101] A rendering target creation module for creating a rendering target of a color language protocol;

[0102] A video image information sending-in module for asynchronously sending the collected video image information into the trained face recognition model;

[0103] An acceleration module for using parallel computing technology of a parallel computing architecture to perform GPU acceleration on the trained face recognition model.

[0104] Optionally, the image rendering module can specifically include:

[0105] A target background information determination unit for determining target background information;

[0106] An image superimposition unit for superimposing the foreground portrait information and the target background information to obtain an AI synthesized image.

[0107] Optionally, the image superimposition unit can specifically be configured to:

[0108] Superimpose the foreground portrait information and the target background information, and perform a blurring operation on the human outline and the hair part to obtain an AI synthesized image.

[0109] Based on the same idea, the embodiments of the present specification also provide an AI cutout device. As shown in Figure 4 The AI cutout device can include:

[0110] A communication unit / communication interface for obtaining to-be-recognized video image information; the to-be-recognized video image information includes image information collected by a camera;

[0111] a processing unit / processor configured to recognize the foreground portrait information in the to-be-recognized video image information by using the trained face recognition model, the foreground portrait information being provided with an alpha channel;

[0112] draw the foreground portrait information provided with the alpha channel to a rendering target and transmit the foreground portrait information to a UE engine, the UE engine displaying the foreground portrait by using a dynamic material instance.

[0113] As shown in Figure 4 , the terminal device can further include a communication line. The communication line can include a path for transmitting information between the components.

[0114] Optionally, as shown in Figure 4 , the terminal device can further include a memory. The memory is configured to store computer-executable instructions for implementing the solutions of the present application, and the processor is configured to execute the computer-executable instructions stored in the memory to implement the method provided by the embodiments of the present application.

[0115] As shown in Figure 4 , the memory can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magneto-optical disk, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through the communication line. The memory can also be integrated with the processor.

[0116] Optionally, the computer-executable instructions in the embodiments of the present application can also be referred to as application program codes, and the embodiments of the present application do not make specific limitations on this.

[0117] In a specific implementation, as an embodiment, as shown in Figure 4 , the processor can include one or more CPUs, such as CPU0 and CPU1 in Figure 4 .

[0118] In a particular implementation, as one example, a terminal device can include multiple processors, such as the processors in Figure 4 ​ Each of these processors can be a single core processor or a multiple core processor.

[0119] The above mainly introduces the scheme provided by the embodiments of the present application from the perspective of interaction between various modules. It can be understood that each module contains a hardware structure and / or software unit for executing each function in order to achieve the above functions. Those skilled in the art should easily realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0120] The embodiments of the present application can divide the functional modules according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be realized in the form of hardware or software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division. There can be another division method when actually implemented.

[0121] The processor in the specification can also have the function of a memory. The memory is used to store computer execution instructions for executing the scheme of the present application, and is controlled by the processor to execute. The processor is used to execute the computer execution instructions stored in the memory, thereby realizing the method provided by the embodiments of the present application.

[0122] ​The memory can be read-only memory (ROM) or other type of static storage devices that can store static information and instructions, random access memory (RAM) or other type of dynamic storage device that can store information and instructions, electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), Blu-ray discs, etc.), magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by the computer, but is not limited to this. The memory can exist independently and be connected to the processor through a communication line. The memory can also be integrated with the processor.

[0123] Optionally, the computer-executed instructions in the embodiments of the present application can also be referred to as application codes, and the embodiments of the present application do not make specific limitations thereto.

[0124] The method disclosed in the embodiments of the present application can be applied to a processor or implemented by the processor. The processor can be an integrated circuit chip with a signal processing capability. In the implementation process, the steps of the above method can be completed by the integrated logic electric circuit or the software form of instructions in the processor. The processor mentioned above can be a general processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0125] In a possible implementation, a computer readable storage medium is provided, and the computer readable storage medium stores instructions. When the instructions are executed, the method in the above embodiments is implemented.

[0126] In the above embodiments, the method can be implemented by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, the method can be implemented in the form of a computer program product, in whole or in part. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; an optical medium, such as a digital video disc (digital video disc, DVD); and a semiconductor medium, such as a solid state drive (solid state drive, SSD).

[0127] Although the present application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. The word "comprising" does not exclude other components or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to an advantage.

[0128] Although the present application has been described in connection with the preferred embodiments thereof with reference to the specific content thereof, it will be apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the spirit and scope of the application. Accordingly, it is intended that the present application cover all such modifications and changes as fall within the scope of the application, along with all equivalents thereof. It will be understood by those within the art that, in general, terms used herein, and especially to the immediately preceding description and claims attached hereto, are intended to be given their broadest interpretation consistent with the specification and the patent statutes.

Claims

1. An AI matting method, characterized in that, The method comprises the following steps: obtaining to-be-recognized video image information; the to-be-recognized video image information comprises image information collected by a camera; using a trained face recognition model to recognize the to-be-recognized video image information, to obtain foreground portrait information in the to-be-recognized video image information; the foreground portrait information has an alpha channel; drawing the foreground portrait information with the alpha channel to a rendering target and transmitting the foreground portrait information with the alpha channel to a UE engine; the UE engine displays the foreground portrait through a dynamic material instance; before using the trained face recognition model to recognize the to-be-recognized video image information, to obtain the foreground portrait information in the to-be-recognized video image information, the method further comprises the following steps: obtaining video image information containing a face image and a background image, portrait soft segmentation information and motion prior information; inputting the video image information, the portrait soft segmentation information and the motion prior information into an initial face recognition model, to obtain a mask with an alpha channel and a foreground image; comparing the mask with the alpha channel and the foreground image with known information, to obtain a comparison result; using the comparison result, adjusting parameters of the initial face recognition model, to obtain the trained face recognition model; the trained face recognition model contains four different encoders, which correspond to the video image information containing the face image, the video image information containing the background image, the portrait soft segmentation information and the motion prior information respectively; each encoder generates a feature map with 256 channels; through 1x1 convolution, a BN layer and a ReLU layer, combining image features in the video image information containing the face image with the video image information containing the background image, the portrait soft segmentation information and the motion prior information respectively; each pair of combinations generates a 64-channel feature; combining the three 64-channel features with the original 256-channel image features, to generate encoded features, and transmitting the encoded features to the remaining network composed of residual blocks and encoders.

2. The method of claim 1, wherein, before drawing the foreground portrait information with the alpha channel to the rendering target and transmitting the foreground portrait information with the alpha channel to the UE engine, the method further comprises the following steps: creating a rendering target of a color language protocol; asynchronously sending the collected video image information into the trained face recognition model; using parallel computing technology of a parallel computing architecture, performing GPU acceleration on the trained face recognition model.

3. The method of claim 1, wherein, drawing the foreground portrait information with the alpha channel to the rendering target and transmitting the foreground portrait information with the alpha channel to the UE engine specifically comprises the following steps: determining target background information; superimposing the foreground portrait information and the target background information, to obtain an AI synthesized image.

4. The method of claim 3, wherein, superimposing the foreground portrait information and the target background information, to obtain the AI synthesized image specifically comprises the following steps: superimposing the foreground portrait information and the target background information, and performing a virtualization operation on a figure contour and a hair part, to obtain the AI synthesized image.

5. An AI matting device, comprising: The device comprises: a to-be-recognized video image information acquisition module, configured to acquire to-be-recognized video image information; the to-be-recognized video image information comprises image information collected by a camera; The foreground portrait information determination module is configured to identify the to-be-identified video image information by using the trained face recognition model to obtain foreground portrait information in the to-be-identified video image information. The foreground portrait information has an alpha channel. The image rendering module is configured to draw the foreground portrait information with the alpha channel to a rendering target and transmit the foreground portrait information to a UE engine. The UE engine displays the foreground portrait by using a dynamic material instance. The input information acquisition module is configured to acquire video image information containing a face image and a background image, portrait soft segmentation information, and motion prior information. The identification module is configured to input the video image information, the portrait soft segmentation information, and the motion prior information into an initial face recognition model to obtain a mask with an alpha channel and a foreground image. The comparison module is configured to compare the mask with the alpha channel and the foreground image with known information to obtain a comparison result. The model training module is configured to adjust parameters of the initial face recognition model by using the comparison result to obtain a trained face recognition model. The trained face recognition model contains four different encoders corresponding to the video image information containing the face image, the video image information containing the background image, the portrait soft segmentation information, and the motion prior information, respectively. Each encoder generates a feature map with 256 channels. The image features in the video image information containing the face image are combined with the video image information containing the background image, the portrait soft segmentation information, and the motion prior information by using 1*1 convolution, a BN layer, and a ReLU layer, and each pair of combinations generates a 64-channel feature. The three 64-channel features and the original 256-channel image features are combined to generate encoded features and transmit the encoded features to the remaining network composed of residual blocks and encoders.

6. An AI matting device, comprising: a device It includes: A communication unit / communication interface is configured to acquire to-be-identified video image information. The to-be-identified video image information includes image information collected by a camera. A processing unit / processor is configured to identify the to-be-identified video image information by using a trained face recognition model to obtain foreground portrait information in the to-be-identified video image information. The foreground portrait information has an alpha channel. The foreground portrait information with the alpha channel is drawn to a rendering target and transmitted to a UE engine. The UE engine displays the foreground portrait by using a dynamic material instance. Before the to-be-identified video image information is identified by using the trained face recognition model to obtain the foreground portrait information in the to-be-identified video image information, the following steps are further included: Video image information containing a face image and a background image, portrait soft segmentation information, and motion prior information are acquired. The video image information, the portrait soft segmentation information, and the motion prior information are input into an initial face recognition model to obtain a mask with an alpha channel and a foreground image. The mask with the alpha channel and the foreground image are compared with known information to obtain a comparison result. The parameters of the initial face recognition model are adjusted by using the comparison result to obtain a trained face recognition model. The trained face recognition model comprises four different encoders corresponding to video image information containing face images, video image information containing background images, portrait soft segmentation information, and motion prior information respectively; each encoder generates a feature map of 256 channels; Through 1*1 convolution, a BN layer and a ReLU layer, image features in the video image information containing face images are combined with the video image information containing background images, the portrait soft segmentation information, and the motion prior information respectively, and each pair of combinations generates a 64-channel feature; The three 64-channel features are combined with the original 256-channel image features to generate encoded features, which are transmitted to the remaining network composed of residual blocks and encoders.

7. A computer storage medium, characterized in that The computer storage medium stores instructions, and when the instructions are executed, the AI matting method of any one of claims 1-4 is implemented.

Images