Neural network training method, image matting processing method, device and equipment

By training the target neural network and combining manual adjustment and data enhancement technology, the shortcomings of traditional green screen keying algorithms in complex scenes and texture processing are solved, and a higher quality and stable keying effect is achieved, avoiding jagged edges and Keying error.

CN118982553APending Publication Date: 2024-11-19SHANGHAI BILIBILI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410989359.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The traditional green screen keying algorithm performs poorly when dealing with complex scenes, the edge processing is not fine, and the adaptability is poor. It requires manual settings, which are sensitive to light and shadows, which are prone to pinching errors and overflowing, and are also suitable for complex backgrounds. and texture processing is difficult.

Method used

By training the target neural network, combining manual fine adjustment parameters, overflow suppression algorithm and data enhancement technology, the initial training data is corrected and processed, and trained using backpropagation algorithm and pre-built loss function to reduce jagged edges and pinching Error, improve the quality and stability of the keying effect.

Benefits of technology

It realizes better processing of complex scenes and textures, avoids jagged edges and keying errors of traditional algorithms, improves the quality and stability of the keying effect, and reduces the impact on lighting changes and overflow phenomena.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982553A_ABST
    Figure CN118982553A_ABST
Patent Text Reader

Abstract

The invention provides a neural network training method for image matting processing, an image matting processing method, an image matting processing device, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: acquiring initial training data, wherein the initial training data comprises an image matting result image obtained by performing image matting processing on an image sample by using a predetermined image matting algorithm; performing effect correction processing on an image matting result image in the initial training data to obtain a corresponding corrected image matting image, thereby obtaining target training data based on the corrected image matting image; and using the target training data to train a target neural network through a back propagation algorithm and a pre-constructed loss function, the target neural network being used for matting an image. According to the method and the device, the problem of degradation of the image matting effect caused by phenomena such as color overflow and illumination condition change in the image matting process is effectively inhibited, so that the quality and the stability of the image matting effect are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a neural network training method, a matting processing method, a device, an electronic device, a computer-readable medium and a computer program product for matting processing. Background Art

[0002] Green screen keying is a post-processing technology widely used in film and television production and video production. Its basic principle is to use the Chroma Key technology to remove the background of a specific color (usually green or blue) from the image or video, so that other images or video materials can be synthesized into the background in the later stage. This technology relies on the obvious difference between the background color and the main color, which makes it easier for computer software to recognize and remove the background.

[0003] Green screen keying technology is widely used and critical in digital human projects. It is mainly used to create highly realistic digital human images that are seamlessly integrated with virtual environments. In digital human projects, green screen keying technology allows the use of a green background when shooting real actors to create digital characters, so that the background can be replaced in the virtual environment later. This is crucial to creating immersive virtual scenes.

[0004] Traditional green screen keying algorithms mainly rely on Chroma Key technology. The algorithm first identifies and sets the green background range, then calculates the color difference between each pixel and the background color, classifies the pixel as foreground or background based on the threshold, generates a mask and separates the foreground. Finally, after some post-processing operations, the foreground and the new background are naturally integrated.

[0005] Traditional green screen keying algorithms have shortcomings such as imprecise edge processing, poor adaptability to complex scenes, the need for manual parameter setting, sensitivity to lighting and shadows, difficulty in processing complex backgrounds, and difficulty in dealing with complex textures. These shortcomings result in poor performance when dealing with complex situations, high technical requirements for operators, and the tendency to produce keying errors or residues. At the same time, when the green screen layout is not professional enough, the foreground after keying is prone to overflow. Summary of the invention

[0006] Multiple aspects of the present application provide a neural network training method, a matting processing method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for performing matting processing.

[0007] In one aspect of the present application, a neural network training method for performing matting processing is provided, wherein the method comprises:

[0008] Acquire initial training data, wherein the initial training data includes a matting result image obtained by matting an image sample using a predetermined matting algorithm;

[0009] By performing effect correction processing on the matting result image in the initial training data, a corresponding corrected matting image is obtained, thereby obtaining target training data based on the corrected matting image;

[0010] The target training data is used to train a target neural network through a back-propagation algorithm and a pre-constructed loss function, and the target neural network is used to perform image matting processing on the image.

[0011] In one aspect of the present application, a method for image matting is provided, wherein the method comprises:

[0012] Get the image to be processed;

[0013] Performing matting processing on the image to be processed by a target neural network to obtain a matting result image corresponding to the image to be processed;

[0014] The target neural network is obtained by training using the neural network training method for matting processing in an embodiment of the present application.

[0015] In one aspect of the present application, a device for training a neural network for matting processing is provided, wherein the device comprises:

[0016] A device for acquiring initial training data, wherein the initial training data includes a matting result image obtained by matting an image sample using a predetermined matting algorithm;

[0017] Device for obtaining a corresponding corrected matte image by performing effect correction processing on the matte result image in the initial training data, thereby obtaining target training data based on the corrected matte image;

[0018] A device for training a target neural network using the target training data through a back-propagation algorithm and a pre-constructed loss function, wherein the target neural network is used to perform image matting processing on an image.

[0019] In one aspect of the present application, an image matting processing device is provided, wherein the device comprises:

[0020] A device for acquiring an image to be processed;

[0021] A device for performing matting processing on the image to be processed through a target neural network to obtain a matting result image corresponding to the image to be processed;

[0022] The target neural network is obtained by training using the neural network training method for matting processing in an embodiment of the present application.

[0023] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of an embodiment of the present application.

[0024] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0025] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0026] In the solution provided in the embodiment of the present application, a target neural network is trained for green screen matting processing, and when training the target neural network, the training data is corrected and processed in combination with manual fine adjustment of parameters, spill suppression algorithm and / or data enhancement to improve the defects of the traditional green screen matting algorithm, so that the trained target neural network can effectively suppress the degradation of the matting effect caused by the spill phenomenon, changes in lighting conditions and the like during the matting process, thereby improving the quality and stability of the matting effect; edge refinement is performed in the process of training the neural network, so that the trained target neural network can better handle complex scenes and textures, avoiding the jagged edges and matting errors of the traditional algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0029] Figure 1 A schematic flow chart of a neural network training method for performing matting processing provided in an embodiment of the present application is shown;

[0030] Figure 2 A schematic diagram of a process flow of an image matting processing method according to an embodiment of the present application is shown;

[0031] Figure 3A schematic diagram of the structure of a device for training a neural network for matting processing provided in an embodiment of the present application is shown;

[0032] Figure 4 A schematic diagram of the structure of a device for image matting processing provided by an embodiment of the present application is shown;

[0033] Figure 5 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0034] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0037] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0038] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0039] Figure 1A schematic flow chart of a method provided in an embodiment of the present application is shown. The method at least includes step S101, step S102, step S103 and step S104.

[0040] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0041] The concepts involved in the embodiments of the present application are explained below.

[0042] Green screen: A technique used in video production and image processing that uses a green background when shooting or recording, and then removes the green background in post-production and replaces it with another background or image.

[0043] Keying: refers to the process of separating a part of an image from the background. In green screen keying, keying refers to the process of removing the green screen background and leaving the foreground person or object.

[0044] Chroma key: also known as chroma separation key or chroma keying, is a video synthesis technology that is often used in video production to remove the background of a specific color range and replace it with other images or videos. Typical chroma key technology usually uses green or blue as the background color because these colors rarely appear in human skin and are easy to distinguish from foreground objects.

[0045] Color correction: refers to adjusting the color of an image or video in post-production to match the background or achieve a desired effect.

[0046] Neural Network: In the field of image processing, it generally refers to convolutional neural networks. They are designed and trained to extract features from images and perform various visual tasks such as classification, detection, segmentation, etc.

[0047] Unet: The two main features of the UNet network structure are: U-shaped network structure and skip connection. UNet is a symmetrical network structure with downsampling on the left and upsampling on the right. According to the function, the series of downsampling operations on the left can be called an encoder, and the series of upsampling operations on the right can be called a decoder. Skip connection refers to the fusion of feature maps in the downsampling process during the upsampling process. The fusion operation used by the skip connection superimposes the channels of the feature map.

[0048] In the video processing scenario, the method can be executed by a server for performing matting processing on images such as green screen images that need to remove the background. The server trains a target neural network for green screen matting processing, and processes the training data in combination with manual fine adjustment of parameters, color suppression algorithm and / or data enhancement when training the target neural network, so as to improve the defects of the traditional green screen matting algorithm, so that the trained target neural network can effectively suppress the matting effect degradation caused by the overflow phenomenon and the change of lighting conditions during the matting process, thereby improving the quality and stability of the matting effect.

[0049] Reference Figure 1 In step S101, initial training data is obtained, and the initial training data includes a matting result image obtained by matting an image sample using a predetermined matting algorithm.

[0050] The matting process is used to perform matting on specific objects in the image sample, for example, to segment various types of objects such as people, animals or other objects in the green screen image and remove the background part outside the object in the image.

[0051] The image samples may be image samples pre-stored in a database, or image samples uploaded by a terminal.

[0052] Those skilled in the art should be familiar with the fact that image samples can be keyed out using a variety of keying algorithms, such as Luma Keying, Color Difference Keying, etc. Those skilled in the art can adopt appropriate keying algorithms based on actual needs, which will not be described in detail here.

[0053] In step S102, the matting result image in the initial training data is subjected to effect correction processing to obtain a corresponding corrected matting image, thereby obtaining target training data based on the corrected matting image.

[0054] Wherein, the target training data is used to train the target neural network.

[0055] According to one embodiment, the effect correction process includes a manual adjustment process, and the step S102 includes step S1021.

[0056] In step S1021, the corrected matte image after manual adjustment is obtained, so that the corrected matte image after manual adjustment is used as target training data.

[0057] The relevant personnel adjust one or more cutout parameters through the manual adjustment process to adjust the cutout result to better meet the cutout requirements.

[0058] According to one embodiment, the effect correction process includes data enhancement process, and the step S102 includes step S1022.

[0059] In step S1022, data enhancement processing is performed on the image samples, so that the corrected matte image obtained after the data enhancement processing is used as target training data.

[0060] The data enhancement process includes but is not limited to at least one of the following operations:

[0061] 1) Adjust background related parameters; for example, adjust the color saturation, brightness, color offset and other parameters of the green screen background;

[0062] 2) Adjust parameters related to lighting conditions; including adjusting the brightness, shadow and other parameters of the lighting.

[0063] The method enhances the training data by adjusting the color saturation, brightness, color offset and other parameters of the green screen, and introducing different lighting conditions, such as light and dark, shadow, etc., so that the matting network trained later can better adapt to matting tasks under different green screen colors and various lighting conditions, thereby improving the robustness and practicality of the algorithm. As a result, better results can be obtained without human intervention during the operation of the algorithm.

[0064] According to one embodiment, the effect correction process includes spill suppression process, and the step S102 includes step S1023.

[0065] In step S1023, overflow suppression processing is performed on the initial training data, so that the corrected matte image obtained after the overflow suppression processing is used as the target training data.

[0066] Specifically, data with overflow phenomenon and corresponding clean data are added to the initial training data.

[0067] The method of the embodiment of the present application can train the target neural network based on these data, so that the target neural network can learn the relevant features for overflow suppression without the need for manual adjustment to perform overflow suppression processing.

[0068] According to an embodiment, in step S102, the method obtains target training data by performing manual adjustment processing, data enhancement processing and overflow suppression processing on the matting results in the initial training data.

[0069] Continue to refer to Figure 1 To illustrate, in step S103, the target neural network is trained using the target training data through a back propagation algorithm and a pre-constructed loss function.

[0070] Those skilled in the art should be familiar with the back-propagation algorithm process, which will not be described in detail here.

[0071] According to one embodiment, step S103 further includes step S1031 and step S1032.

[0072] In step S1031, a target matting loss is determined based on the matting result image in the target training data and its corresponding corrected matting image.

[0073] Optionally, the target matting loss includes a minimum absolute error loss (L1 loss) of an alpha channel after matting and an L1 loss of an RGB channel after matting.

[0074] Among them, the Alpha channel is an 8-bit grayscale channel, which uses 256 levels of gray to record the transparency information in the image and define transparent, opaque and semi-transparent areas.

[0075] In step S1032, the target neural network is trained based on the target cutout loss until the target cutout loss approaches an expected value.

[0076] Optionally, the method focuses on the loss of the Alpha channel in the early stage of training the target neural network, and increases the weight of the loss of the RGB channel in the later stage of training.

[0077] According to one embodiment, when the method trains the target neural network in step S103, the cutout result obtained by the target neural network is subjected to edge refinement processing by increasing the weight of the target cutout loss of the cutout foreground edge area, so that the trained target neural network pays extra attention to the effect of the cutout edge part.

[0078] According to one embodiment, the method uses a target neural network based on a Unet network structure. UNet has an encoder-decoder structure and integrates feature information of different levels in a skip connection. In addition, in order to achieve a faster speed, the target neural network used in this embodiment subtracts two encoder-decoder groups from the Unet network structure.

[0079] Specifically, for the image to be processed that is input into the target neural network with a Unet structure, the method performs multi-level feature extraction processing on the image to be processed, and fuses and refines the edges of the extracted features in subsequent layers, so that the edges of the matting results are finer, avoiding the jagged edges and matting errors of traditional algorithms.

[0080] According to the method of the embodiment of the present application, a target neural network is trained for green screen matting processing, and the training data is corrected and processed in combination with manual fine adjustment of parameters, spill suppression algorithm and / or data enhancement when training the target neural network, so as to improve the defects of the traditional green screen matting algorithm, so that the trained target neural network can effectively suppress the degradation of the matting effect caused by the spill phenomenon, changes in lighting conditions and the like during the matting process, thereby improving the quality and stability of the matting effect; edge refinement processing is performed in the process of training the neural network, so that the trained target neural network can better handle complex scenes and textures, avoiding the jagged edges and matting errors of the traditional algorithm.

[0081] Figure 2 An image matting processing method according to an embodiment of the present application is shown, and the method comprises step S201 and step S202.

[0082] In step S201, an image to be processed is obtained.

[0083] The image to be processed contains an object that needs to be separated from the background.

[0084] In step S202, the image to be processed is subjected to matting processing by a target neural network to obtain a matting result image corresponding to the image to be processed.

[0085] The target neural network is obtained by training the neural network training method for matting processing by executing the above steps S101 to S103.

[0086] According to the method of the embodiment of the present application, the trained target neural network is used to perform matting processing on the image to be processed, so as to effectively suppress the degradation of the matting effect caused by the overflow phenomenon, the change of lighting conditions and the like during the matting process, thereby improving the quality and stability of the matting effect; edge refinement processing is performed in the process of training the neural network, so that the trained target neural network can better handle complex scenes and textures, avoiding the jagged edges and matting errors of traditional algorithms.

[0087] Figure 3 A schematic diagram of the structure of a device for training a neural network for matting processing provided in an embodiment of the present application is shown.

[0088] Figure 3 The device shown includes: a device for acquiring initial training data (hereinafter referred to as "data acquisition device 101"), a device for obtaining a corresponding corrected matte image by performing effect correction processing on the matte result image in the initial training data, thereby obtaining target training data based on the corrected matte image (hereinafter referred to as "data correction device 102"), and a device for using the target training data to train the target neural network through a back propagation algorithm and a pre-constructed loss function (hereinafter referred to as "network training device 103").

[0089] The data acquisition device 101 acquires initial training data, where the initial training data includes a matting result image obtained by matting an image sample using a predetermined matting algorithm.

[0090] The matting process is used to perform matting on specific objects in the image sample, for example, to segment various types of objects such as people, animals or other objects in the green screen image and remove the background part outside the object in the image.

[0091] The image samples may be image samples pre-stored in a database, or image samples uploaded by a terminal.

[0092] Those skilled in the art should be familiar with the fact that image samples can be keyed out using a variety of keying algorithms, such as Luma Keying, Color Difference Keying, etc. Those skilled in the art can adopt appropriate keying algorithms based on actual needs, which will not be described in detail here.

[0093] The data correction device 102 obtains a corresponding corrected matte image by performing effect correction processing on the matte result image in the initial training data, thereby obtaining target training data based on the corrected matte image.

[0094] Wherein, the target training data is used to train the target neural network.

[0095] According to one embodiment, the effect correction process includes a manual adjustment process, and the data correction device 102 includes an adjustment data acquisition device.

[0096] The adjustment data acquisition device acquires the modified matte image after manual adjustment processing, so as to use the modified matte image after manual adjustment processing as target training data.

[0097] The relevant personnel adjust one or more cutout parameters through the manual adjustment process to adjust the cutout result to better meet the cutout requirements.

[0098] According to one embodiment, the effect correction process includes data enhancement process, and the data correction device 102 includes a data enhancement device.

[0099] The data enhancement device performs data enhancement processing on the image samples, so as to use the corrected matte image obtained after the data enhancement processing as the target training data.

[0100] The data enhancement process includes but is not limited to at least one of the following operations:

[0101] 1) Adjust background related parameters; for example, adjust the color saturation, brightness, color offset and other parameters of the green screen background;

[0102] 2) Adjust parameters related to lighting conditions; including adjusting the brightness, shadow and other parameters of the lighting.

[0103] The method enhances the training data by adjusting the color saturation, brightness, color offset and other parameters of the green screen, and introducing different lighting conditions, such as light and dark, shadow, etc., so that the matting network trained later can better adapt to matting tasks under different green screen colors and various lighting conditions, thereby improving the robustness and practicality of the algorithm. As a result, better results can be obtained without human intervention during the operation of the algorithm.

[0104] According to one embodiment, the effect correction process includes a spill suppression process, and the data correction device 102 includes a spill suppression device.

[0105] The overflow suppression device performs overflow suppression processing on the initial training data, so as to use the corrected matte image obtained after the overflow suppression processing as the target training data.

[0106] Specifically, data with overflow phenomenon and corresponding clean data are added to the initial training data.

[0107] The overflow suppression device of the embodiment of the present application can train the target neural network based on these data, so that the target neural network can learn the relevant features for overflow suppression without the need for manual adjustment to perform overflow suppression processing.

[0108] According to one embodiment, the data correction device 102 obtains the target training data by performing manual adjustment processing, data enhancement processing and overflow suppression processing on the matting results in the initial training data.

[0109] Continue to refer to Figure 3 To illustrate, the network training device 103 uses the target training data to train the target neural network through a back propagation algorithm and a pre-constructed loss function.

[0110] Those skilled in the art should be familiar with the back-propagation algorithm process, which will not be described in detail here.

[0111] According to one embodiment, the network training device 103 further includes a loss determination device and a sub-network training device.

[0112] The loss determination device determines the target matting loss based on the matting result image in the target training data and its corresponding corrected matting image.

[0113] Optionally, the target matting loss includes a minimum absolute error loss (L1 loss) of an alpha channel after matting and an L1 loss of an RGB channel after matting.

[0114] Among them, the Alpha channel is an 8-bit grayscale channel, which uses 256 levels of gray to record the transparency information in the image and define transparent, opaque and semi-transparent areas.

[0115] The sub-network training device trains the target neural network based on the target cutout loss until the target cutout loss approaches an expected value.

[0116] Optionally, the device focuses on the loss of the alpha channel in the early stage of training the target neural network, and increases the weight of the loss of the RGB channels in the later stage of training.

[0117] According to one embodiment, when the network training device 103 trains the target neural network, the weight of the target cutout loss of the cutout foreground edge area is increased, and the cutout result obtained by the target neural network is subjected to edge refinement processing, so that the trained target neural network pays extra attention to the effect of the cutout edge part.

[0118] According to one embodiment, the method uses a target neural network based on a Unet network structure. UNet has an encoder-decoder structure and integrates feature information of different levels in a skip connection. In addition, in order to achieve a faster speed, the target neural network used in this embodiment subtracts two encoder-decoder groups from the Unet network structure.

[0119] Specifically, for the image to be processed that is input into the target neural network with a Unet structure, the method performs multi-level feature extraction processing on the image to be processed, and fuses and refines the edges of the extracted features in subsequent layers, so that the edges of the matting results are finer, avoiding the jagged edges and matting errors of traditional algorithms.

[0120] According to the device of the embodiment of the present application, a target neural network is trained for green screen matting processing, and when training the target neural network, the training data is corrected and processed in combination with manual fine adjustment of parameters, spill suppression algorithm and / or data enhancement to improve the defects of the traditional green screen matting algorithm, so that the trained target neural network can effectively suppress the degradation of the matting effect caused by the spill phenomenon, changes in lighting conditions and the like during the matting process, thereby improving the quality and stability of the matting effect; edge refinement processing is performed in the process of training the neural network, so that the trained target neural network can better handle complex scenes and textures, avoiding the jagged edges and matting errors of the traditional algorithm.

[0121] Figure 4 A schematic structural diagram of a device for image matting processing provided in an embodiment of the present application is shown.

[0122] Figure 4 The device shown includes a device for acquiring an image to be processed (hereinafter referred to as "image acquisition device 201"), and a device for performing matting processing on the image to be processed through a target neural network to obtain a matting result image corresponding to the image to be processed (hereinafter referred to as "matting processing device 202").

[0123] Reference Figure 4 , the image acquisition device 201 acquires the image to be processed.

[0124] The image to be processed contains an object that needs to be separated from the background.

[0125] The matting processing device 202 performs matting processing on the image to be processed through a target neural network to obtain a matting result image corresponding to the image to be processed.

[0126] The target neural network is obtained by executing the operations of the data acquisition device 101, the data correction device 102 and the network training device 103.

[0127] According to the device of the embodiment of the present application, a trained target neural network is used to perform matting processing on the image to be processed, thereby effectively suppressing the degradation of the matting effect caused by the overflow phenomenon, changes in lighting conditions and the like during the matting process, thereby improving the quality and stability of the matting effect; edge refinement processing is performed during the training of the neural network, so that the trained target neural network can better handle complex scenes and textures, avoiding the jagged edges and matting errors of traditional algorithms.

[0128] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method of the neural network training method for performing matting processing or the image matting processing method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0129] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0130] Figure 5The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM 1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0131] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0132] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0133] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0134] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0135] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0137] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0143] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0145] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A neural network training method for performing image matting processing, wherein: The method comprises: Acquire initial training data, wherein the initial training data includes a matting result image obtained by matting an image sample using a predetermined matting algorithm; By performing effect correction processing on the matting result image in the initial training data, a corresponding corrected matting image is obtained, thereby obtaining target training data based on the corrected matting image; The target training data is used to train a target neural network through a back-propagation algorithm and a pre-constructed loss function, and the target neural network is used to perform image matting processing on the image.

2. The method according to claim 1, wherein: The method of using the target training data to train the target neural network through a back propagation algorithm and a pre-constructed loss function includes: Determine a target matting loss based on a matting result image in the target training data and its corresponding corrected matting image; The target neural network is trained based on the target cutout loss until the target cutout loss approaches an expected value.

3. The method according to claim 2, wherein: The method further comprises: When training the target neural network, the edge refinement of the cutout result obtained by the target neural network is performed by increasing the weight of the target cutout loss in the cutout foreground edge area.

4. The method according to claim 1, wherein: The step of performing effect correction processing on the matting result image in the initial training data to obtain a corresponding corrected matting image, thereby obtaining target training data based on the corrected matting image includes: The corrected matte image after manual adjustment is obtained, and the corrected matte image after manual adjustment is used as target training data.

5. The method according to claim 1, wherein: The step of performing effect correction processing on the matting result image in the initial training data to obtain a corresponding corrected matting image, thereby obtaining target training data based on the corrected matting image includes: Data enhancement processing is performed on the image samples, so that the corrected matte images obtained after the data enhancement processing are used as target training data.

6. The method according to claim 1, wherein: The step of performing effect correction processing on the matting result image in the initial training data to obtain a corresponding corrected matting image, thereby obtaining target training data based on the corrected matting image includes: The initial training data is subjected to overflow suppression processing, so that the corrected matte image obtained after the overflow suppression processing is used as target training data.

7. A method for processing image matting, wherein: The method comprises: Get the image to be processed; Performing matting processing on the image to be processed by a target neural network to obtain a matting result image corresponding to the image to be processed; Wherein, the target neural network is trained by the neural network training method for matting processing described in any one of claims 1 to 6.

8. A device for training a neural network for matting processing, wherein: The device comprises: A device for acquiring initial training data, wherein the initial training data includes a matting result image obtained by matting an image sample using a predetermined matting algorithm; A device for obtaining target training data based on the corrected matte image by performing effect correction processing on the matte result image in the initial training data to obtain a corresponding corrected matte image; A device for training a target neural network using the target training data through a back-propagation algorithm and a pre-constructed loss function, wherein the target neural network is used to perform image matting processing on an image.

9. An image matting processing device, wherein: The device comprises: A device for acquiring an image to be processed; A device for performing matting processing on the image to be processed through a target neural network to obtain a matting result image corresponding to the image to be processed; Wherein, the target neural network is trained by the neural network training method for matting processing described in any one of claims 1 to 6.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 6, or execute the method of claim 7.

11. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 6, or to implement the method according to claim 7.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented, or the method according to claim 7 is implemented.

Citation Information

Patent Citations

  • Image synthesis method and device for thoroughly removing green screen, equipment and storage medium

    CN114372990A

  • Picture matting method and device, electronic equipment and storage medium

    CN114926491A

  • Hierarchical supervision coarse-to-fine portrait matting method

    CN116228806A

  • Picture matting method, matting model training method and related equipment

    CN116503421A

  • Method and device for training cutout model and for cutout, equipment, and storage medium

    WO2022127454A1