Image processing method and apparatus, and electronic device and storage medium

Abstract

An image processing method and apparatus, and an electronic device and a storage medium. The method comprises: fusing a pre-segmented mask and a green-screen image, so as to obtain a composite image; determining from the composite image a target area that comprises a target part; driving the target part in the target area, and determining the target area that includes the driven target part to be a driving image; extracting the driven target part according to pixels of the driving image and pixels of a background area; and obtaining a target foreground area according to the driven target part and an area in a foreground area except the target part. In the present application, pixels of a driving image and pixels of a background area are used to extract a driven target part in the driving image, thus improving the accuracy and the segmentation effect of a segmented target foreground area.

Description

Image processing method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 31, 2023, with application number 2023109510704 and application name “Image processing method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and more specifically, to an image processing method, device, electronic device and storage medium. Background Art

[0003] In the field of image processing technology, the foreground of a green screen image is extracted using a segmentation mask determined by a deep learning model to obtain a foreground area including a target object. The target object in the foreground area is then driven to perform a required action (for example, the mouth of a person in a background replacement image is driven to make the sound "a"). A driving image is obtained, and the segmentation mask before driving is reused to segment the driving image to obtain a foreground image with the target object as the foreground after action driving.

[0004] However, after moving a certain part of the target object in the background replacement image (such as the mouth), the area where the target object is located in the driving image may not actually coincide with the area where the target object is located in the original image. Reusing the segmentation mask before driving to segment the driving image may easily lead to inaccurate target objects in the segmented foreground image, resulting in poor foreground segmentation effect from the driving image.

[0005] Summary of the Invention

[0006] In view of this, embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium.

[0007] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0008] The pre-segmentation mask corresponding to the green screen image and the green screen image are fused to obtain a composite image including a foreground area and a background area, wherein the foreground area has a target object; the target object includes a target part; a target area including the target part is determined from the composite image; the target part in the target area is driven, and the target area including the driven target part is determined as a driven image; the driven target part in the driven image is extracted based on the pixels of the driven image and the pixels of the background area; and a target foreground area corresponding to the target object is generated based on the driven target part and the area in the foreground area other than the target part.

[0009] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:

[0010] A fusion module is used to fuse the pre-segmentation mask corresponding to the green screen image and the green screen image to obtain a composite image including a foreground area and a background area, wherein the foreground area has a target object; the target object includes a target part; a determination module is used to determine a target area including the target part from the composite image; a driving module is used to drive the target part in the target area and determine the target area including the driven target part as a driving image; an extraction module is used to extract the driven target part in the driving image based on the pixels of the driving image and the pixels of the background area; an acquisition module is used to generate a target foreground area corresponding to the target object based on the driven target part and the area in the foreground area other than the target part.

[0011] Optionally, the determination module is also used to replace the pixel values ​​of the pixel points in the background area of ​​the synthetic image with the target pixel values ​​to obtain a background replacement image; determine the target area including the target part from the background replacement image; accordingly, the extraction module is also used to extract the driven target part in the driving image based on the pixel values ​​of the pixel points of the driving image and the target pixel values.

[0012] Optionally, the extraction module is further configured to determine a first segmentation mask corresponding to the driving image based on a difference between a pixel value of each pixel point in the driving image and a target pixel value; and to extract a driven target portion in the driving image using the first segmentation mask.

[0013] Optionally, the extraction module is also used to determine the mask value corresponding to each pixel point in the driving image based on the difference between the pixel value of each pixel point in the driving image and the target pixel value; and determine the first segmentation mask corresponding to the driving image based on the mask value corresponding to each pixel point in the driving image.

[0014] Optionally, the extraction module is also used to determine that the mask value of the driven pixel point is a first value if the difference between the pixel value of the driven pixel point and the target pixel value is greater than or equal to a first threshold; the driven pixel point is any pixel point in the driven image; if the difference between the pixel value of the driven pixel point and the target pixel value is less than or equal to a second threshold, the mask value of the driven pixel point is determined to be a second value; the first value is greater than the second value; if the difference between the pixel value of the driven pixel point and the target pixel value is not greater than the first threshold, and the difference between the pixel value of the driven pixel point and the target pixel value is not less than the second threshold, the mask value of the driven pixel point is determined according to the difference between the pixel value of the driven pixel point and the target pixel value, the first threshold and the second threshold.

[0015] Optionally, the extraction module is further configured to perform an edge-inward erosion process on the edge of the target portion in the first segmentation mask to obtain a second segmentation mask corresponding to the driving image; and extract the driven target portion in the driving image using the second segmentation mask.

[0016] Optionally, the extraction module is further used to convolve the edges of the target part in the first segmentation mask through a convolution kernel of a target size to obtain a third segmentation mask corresponding to the driving image; and smooth the edges of the target part in the third segmentation mask through a blur kernel to obtain a second segmentation mask corresponding to the driving image.

[0017] Optionally, the green screen image is a video frame included in the target video; the extraction module is further used to obtain a relevant segmentation mask corresponding to a relevant area including the target part in the adjacent green screen image, and the adjacent green screen image refers to a video frame in the target video that is adjacent to the green screen image and includes the target object; the relevant segmentation mask is used to indicate the area where the target part is located after the driving part in the related area is driven; the second segmentation mask is time-smoothed according to the relevant segmentation mask to obtain a target segmentation mask corresponding to the driving image; and the target segmentation mask is used to extract the target part after driving in the driving image.

[0018] Optionally, the extraction module is further used to fuse the pre-segmentation masks corresponding to adjacent green screen images and the adjacent green screen images to obtain a related composite image including a related foreground area and a related background area, wherein the related foreground area has a target object; replace the pixel values ​​of the pixel points in the related background area in the related composite image with the target pixel values ​​to obtain a related background replacement image; determine the related target area including the target part from the related background replacement image; drive the target part in the related target area to obtain a related drive image; and determine the related segmentation mask corresponding to the related target area based on the pixels and target pixel values ​​of the related drive image.

[0019] Optionally, the adjacent green screen image includes a first adjacent green screen image located before the green screen image in the target video, and a second adjacent green screen image located after the green screen image; the relevant segmentation mask includes a first relevant segmentation mask corresponding to the first adjacent green screen image and a second relevant segmentation mask corresponding to the second adjacent green screen image; the extraction module is further used to perform weighted summation on the first relevant segmentation mask, the second relevant segmentation mask and the second segmentation mask to obtain a target segmentation mask.

[0020] Optionally, the obtaining module is further configured to use a preset background image as the background of the target foreground area, fuse the target foreground area and the preset background image, and obtain a target background replacement image.

[0021] Optionally, the extraction module is further used to obtain a regional segmentation mask corresponding to the target area from the pre-segmentation mask corresponding to the green screen image; fuse the regional segmentation mask with the driving image to obtain a fused driving image; if the fused driving image does not meet the preset conditions, extract the target part after driving in the driving image based on the pixels of the driving image and the pixels of the background area.

[0022] On the one hand, an embodiment of the present application provides an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above method.

[0023] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0024] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium; a processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method in the embodiment of the present application.

[0025] An embodiment of the present application provides an image processing method, device, electronic device and storage medium. In the present application, a target part in a target area is driven, and the target area including the driven target part is determined as a driving image. The driven target part in the driving image is extracted using the pixels of the driving image and the pixels of the background area, rather than directly reusing the pre-segmentation mask corresponding to the green screen image before driving to segment the driving image. Therefore, the present application avoids the situation where the segmented driven target part includes pixels in the background area and the driven target part lacks some pixels due to reusing the pre-segmentation mask to segment the driving image, thereby improving the accuracy of the segmented target foreground area and thereby improving the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram showing an application scenario to which an embodiment of the present application is applicable;

[0027] FIG2 shows a flow chart of an image processing method proposed in one embodiment of the present application;

[0028] FIG3 shows a schematic diagram of a green screen image fusion process in an embodiment of the present application;

[0029] FIG4 shows a schematic diagram of another green screen image fusion process in an embodiment of the present application;

[0030] FIG5 shows a flowchart of an image processing method proposed in yet another embodiment of the present application;

[0031] FIG6 shows a flowchart of an image processing method proposed in yet another embodiment of the present application;

[0032] FIG7 is a schematic diagram showing a background replacement process of a green screen image in an embodiment of the present application;

[0033] FIG8 shows a block diagram of an image processing device proposed in one embodiment of the present application;

[0034] FIG9 shows a structural block diagram of an electronic device for executing an image processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the following description, the terms "first\second" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0037] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0038] The present application discloses an image processing method, device, electronic device and storage medium, which relate to artificial intelligence technology.

[0039] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0040] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0042] The development of artificial intelligence (AI) has given rise to a new type of virtual being: the digital human. This AI-powered virtual being can communicate with users and perform tasks like a real person. Integrating AI capabilities such as voice interaction, natural language understanding, and image recognition, the digital human has a more vivid appearance and more natural conversations with people, transforming human-computer interaction from a simple conversation tool into true communication. Compared to digital humans, the digital human is more intelligent and humane.

[0043] Green screen segmentation technology is a technology that is widely used in special effects generation in movies, TV series, and games. It separates the subject and background during shooting, and then replaces the background with another image or video through image processing technology to achieve the effect of mixing virtual background and real foreground.

[0044] As shown in Figure 1, the application scenario applicable to the embodiment of the present application includes a terminal 20 and a server 10, which are connected to each other via a wired network or a wireless network. The terminal 20 can be a smart phone, tablet computer, laptop computer, desktop computer, smart home appliance, vehicle terminal, aircraft, wearable device terminal, virtual reality device, or other terminal device that can display pages, or run other applications that can call page display applications (such as instant messaging applications, shopping applications, search applications, game applications, forum applications, map and transportation applications, etc.).

[0045] The server 10 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 10 can be used to provide services for applications running on the terminal 20.

[0046] Among them, the terminal 20 can send a green screen image to the server 10, and the server 10 can fuse the pre-segmentation mask corresponding to the green screen image and the green screen image to obtain a synthetic image, and determine the target area including the target part from the synthetic image; the target part is the part of the target object that includes the driving part; the target part in the target area is driven, and the target area including the driven target part is determined as the driving image; according to the pixels of the driving image and the pixels of the background area, the driven target part in the driving image is extracted; according to the driven target part and the area in the foreground area other than the target part, the target foreground area corresponding to the target object is generated; finally, the server 10 determines the target background replacement image after background replacement based on the target foreground area, and then returns the target background replacement image to the terminal 20.

[0047] A green screen image refers to an image that includes a target object and a green screen background. The target object can be a person, an animal, or a piece of machinery. A target part refers to a part of the target object, and a driving part refers to a portion of the target part. For example, if the target object is a person, the target part can be the head, and the driving part can be the face (including the mouth) of the head. For another example, if the target object is a dog, the target part can be the dog's rear end, and the driving part can be the tail.

[0048] The server 10 may determine a pre-segmentation mask of the green screen image using a deep learning-based segmentation model. The server 10 may train an initial segmentation model using sample images of the target object and mask images corresponding to the sample images to obtain a segmentation model.

[0049] In another embodiment, the terminal 20 may be used to execute the method of the present application. After obtaining a target foreground area including a target object, the terminal 20 determines a target background replacement image after background replacement according to the target foreground area.

[0050] It is understood that the terminal 20 may also determine the pre-segmentation mask for the green screen image using a deep learning-based segmentation model. After the server 10 obtains the segmentation model, it may store the segmentation model in a distributed cloud storage system. The terminal 20 then retrieves the segmentation model from the distributed cloud storage system and determines the pre-segmentation mask based on the segmentation model.

[0051] For the convenience of description, in the following embodiments, image processing is performed by an electronic device as an example.

[0052] Please refer to FIG. 2 , which shows a flow chart of an image processing method proposed in one embodiment of the present application. The method can be applied to an electronic device, which can be at least one of the terminal 20 or the server 10 in FIG. 1 . The method includes:

[0053] S110 , fusing the pre-segmented mask corresponding to the green screen image and the green screen image to obtain a composite image including a foreground area and a background area, wherein the foreground area contains a target object; the target object includes a target part.

[0054] The target object with a green screen as the background can be photographed to obtain a green screen image; the target object with a green screen as the background can also be photographed to obtain a video, and then any video frame or specific video frame (the specific video frame can be, for example, the first frame in every ten frames, etc.) is obtained from the video as a green screen image.

[0055] The green screen image uses the target object as the foreground, that is, the area within the green screen image where the target object is located is used as the foreground, and the green screen area excluding the target object is used as the background. In some embodiments, a pre-segmentation mask for the green screen image may be determined using a segmentation model, and then the green screen image and the corresponding pre-segmentation mask are fused to generate a composite image; the area within the composite image where the target object is located is used as the foreground, and the green screen area excluding the target object is used as the background.

[0056] The pre-segmentation mask (also called alphamask) corresponding to the green screen image includes the mask value corresponding to each pixel in the green screen image. The pixel value of each pixel in the green screen image can be multiplied by the corresponding mask value to obtain a composite image to achieve the fusion of the green screen image and the pre-segmentation mask corresponding to the green screen image.

[0057] For example, as shown in Figure 3, a in Figure 3 is a green screen image, b in Figure 3 is the pre-segmentation mask corresponding to the green screen image shown in a in Figure 3, and c in Figure 3 is the composite image obtained by fusing a in Figure 3 and b in Figure 3. For another example, as shown in Figure 4, a in Figure 4 is another green screen image, b in Figure 4 is the pre-segmentation mask corresponding to the green screen image shown in a in Figure 4, and c in Figure 4 is the composite image obtained by fusing a in Figure 4 and b in Figure 4.

[0058] S120 , determining a target area including a target part from the synthesized image; the target part is a part of the target object including the driving part.

[0059] In this embodiment, the target part is a portion of the target object, and the driven part is a portion of the target part. The region in the composite image where the target part is located is obtained as the target region. For example, if the target object in the composite image is a person and the target part is the head, the driven part is the face (including the mouth), and the portion of the composite image including the head is obtained as the target region.

[0060] S130 , driving the target portion in the target area, and determining the target area including the driven target portion as a driving image.

[0061] The target portion includes a driven part. In the embodiment of the present application, the driven part of the target part is referred to as a driving part. The driving part in the target area can be driven by a preset target action. The posture of the driving part of the target object in the target area changes to the posture corresponding to the execution of the target action. When the posture of the driving part changes to the posture corresponding to the execution of the target action, the image of the target part serves as the driving image. That is, the driving image refers to the target area when the driving part performs the target action. The target action refers to the action that the driving part of the target object needs to perform. For example, when the driving part is a person's face (including a mouth), the target action can refer to an action for outputting a driving text, such as the action of saying the driving text of the word "you", such as the action for pronouncing the sound "a".

[0062] For example, the target object is a person, the target part is the head, the driving part is the face (including the mouth), the facial posture of the person in the target area is the image of the head when saying "you", the target action is to say the word "now", and the face of the person in the target area is driven according to the target action to obtain the image of the head when the person says "now" as the driving image.

[0063] S140 : Extracting a driven target portion in the driving image based on pixels of the driving image and pixels of the background area.

[0064] A mask value for each pixel in the driving image can be determined based on the difference between the pixel value of each pixel in the driving image and the pixel value of the pixel in the background area. The mask values ​​for each pixel in the driving image are aggregated to obtain a segmentation mask for segmenting the target portion after driving. The region where the target portion is located is then extracted from the driving image based on the segmentation mask for segmenting the target portion after driving, as the target portion after driving. The mask value is a numerical value between 0 and 1. Extracting the region where the target portion is located can involve multiplying each pixel in the driving image by its respective mask value.

[0065] The difference between the pixel value of each pixel in the driving image and the pixel value of the pixel in the background area can refer to the Euclidean distance, cosine similarity, square of the Euclidean distance, etc. between the pixel value of each pixel in the driving image corresponding to the target object and the pixel value of the pixel in the background area.

[0066] As an embodiment, before S120, the method may include: replacing the pixel values ​​of the pixel points in the background area of ​​the composite image with the target pixel values ​​to obtain a background replacement image. Accordingly, S120 may include: determining the target area including the target part from the background replacement image; S140 may include: extracting the driven target part in the driving image based on the pixel values ​​of the pixel points of the driving image and the target pixel values.

[0067] The target pixel value may be the pixel value RGB (0, 124, 0) corresponding to the green screen color. The pixel values ​​of the pixels in the background area of ​​the synthesized image may be replaced with the target pixel values ​​to obtain a background replacement image corresponding to the synthesized image. The pixel values ​​of the background pixels in the background replacement image are all green screen colors. Compared to the synthesized image, the pixel values ​​of the background pixels in the background replacement image are more uniform, thereby increasing the accuracy of the target part after driving extracted based on the pixel values ​​of the pixels in the driving image and the target pixel values.

[0068] After obtaining the background replacement image, the pixel values ​​of the pixel points in the background area of ​​the background replacement image are all target pixel values. At this time, the area including the target part can be determined from the background replacement image as the target area, and the target part in the target area is driven. The target area including the driven target part is determined as the driving image. After that, the driven target part in the driving image can be extracted based on the pixels of the driving image and the target pixel values.

[0069] The mask value of each pixel in the driving image can be determined based on the difference between the pixel value of each pixel in the driving image and the target pixel value. The mask values ​​of each pixel in the driving image are summarized to obtain a segmentation mask for segmenting the target part after driving. According to the segmentation mask of the target part after segmentation, the area where the target part is located is extracted from the driving image as the target part after driving.

[0070] The target pixel value is the pixel value of the pixel representing the background in the background replacement image. Furthermore, in step S130, the driven portion of the target object in the target area determined in the background replacement image is driven based on the background replacement image. Therefore, the pixel value of the pixel representing the background in the drive image is also the target pixel value. Thus, a segmentation mask for the target portion after segmentation and drive is determined based on the difference between the pixel value of each pixel in the drive image corresponding to the target object and the target pixel value. This segmentation mask for the target portion after segmentation and drive can represent the area in the drive image where the target portion is located, as well as the area where the background is located. This is equivalent to achieving foreground segmentation based on the difference between the foreground pixels and the background pixels in the drive image.

[0071] As an embodiment, before S140, the method may include: obtaining a regional segmentation mask corresponding to the target area from a pre-segmentation mask corresponding to the green screen image; fusing the regional segmentation mask with the driving image to obtain a fused driving image; and S140 includes: if the fused driving image does not meet a preset condition, extracting the target portion after driving in the driving image based on pixels of the driving image and pixels of the background area. The preset condition indicates that the fused driving image does not include pixels in the background of the driving image and that the fused driving image does not lack pixels of the target portion.

[0072] After obtaining the driving image, a partial mask for segmenting the target area in the target region can be obtained from the pre-segmentation mask corresponding to the green screen image as the region segmentation mask. If the target region is determined from the composite image, the region segmentation mask is the partial mask for segmenting the target region in the composite image. Similarly, if the target region is determined from the background replacement image, the region segmentation mask is the partial mask for segmenting the target region in the background replacement image.

[0073] The regional segmentation mask corresponding to the target area in the pre-segmentation mask corresponding to the green screen image is directly reused, and the regional segmentation mask is fused with the driving image to obtain a fused driving image. Since the driving image is obtained based on the target area, the regional segmentation mask used to segment the target area has the same size as the driving image. Since the regional segmentation mask includes a mask value for each pixel in the target area, the regional segmentation mask can also include a mask value for each pixel in the driving image. Fusion of the regional segmentation mask with the driving image can mean multiplying each pixel in the target area by its respective mask value.

[0074] If the fused drive image does not meet the preset conditions, indicating that the fused drive image includes pixels from the background of the drive image or that the fused drive image is missing pixels from the target area, processing continues according to the method of S140 of this application. The inclusion of pixels from the background of the drive image in the fused drive image may be due to the target area being smaller after driving, and the missing pixels from the target area in the fused drive image may be due to the target area being larger after driving.

[0075] If the fused driving image meets the preset conditions, it indicates that the fused driving image does not include pixels in the background of the driving image and the fused driving image does not miss pixels of the target part. At this time, the fused driving image can be used as the target part after driving, and the subsequent step S150 is continued.

[0076] S150 , generating a target foreground area corresponding to the target object according to the driven target part and the area in the foreground area excluding the target part.

[0077] After obtaining the actuated target part, the foreground region in the composite image can be obtained, as well as the region excluding the target part within the foreground region. The actuated target part and the region excluding the target part within the foreground region are then spliced ​​together to obtain the spliced ​​result as the target foreground region. The pose of the actuated part of the target object in the target foreground region is the pose after the target action is output.

[0078] It can be understood that since the part of the target object other than the target part does not change its posture, the area other than the target part in the foreground area of ​​the composite image can be directly obtained, and the area other than the target part in the foreground area can be directly spliced ​​with the target part after driving, so as to obtain the target object whose posture has changed as the driving part.

[0079] As an implementation manner, after S150 , the method may include: using a preset background image as the background of the target foreground area, and fusing the target foreground area and the preset background image to obtain a target background replacement image.

[0080] In this embodiment, the preset background image can be any image, which can be a landscape image, a building image or an animal image. The preset background image may include the target object or not, and the size of the preset background image is the same as the size of the driving image corresponding to the target object.

[0081] Any image can be acquired as a background image, and the background image is adjusted to a preset background image having the same size as the driving image corresponding to the target object.

[0082] In this embodiment, the preset background image can be used as the background of the target foreground area, and the target foreground area can be superimposed on the preset background image. The pixel values ​​of the pixel points in the target foreground area are retained in the overlapping part, and the pixel values ​​of the pixel points in the preset background image are retained in the non-overlapping part to obtain the target background replacement image.

[0083] In this embodiment, the target part in the target area is driven, and the target area including the driven target part is determined as the driving image. The driven target part in the driving image is extracted using the pixels of the driving image and the pixels of the background area. Since the posture of the driving part changes, it may cause other parts around the driving part to change. Therefore, in this application, a target part with a range larger than the driving part can be determined. Therefore, when processing the partial image corresponding to the driving part, the images corresponding to other parts other than the driving part are also processed, thereby improving the accuracy of the segmentation of the driven target part. This application avoids reusing the pre-segmentation mask to segment the driving image, so it can avoid the situation where the segmented driven target part includes pixels in the background area, and can avoid the situation where the driven target part lacks some pixels, thereby improving the accuracy of the segmented target foreground area and thus improving the segmentation effect.

[0084] At the same time, the area except the target part in the foreground area is directly spliced ​​with the target part after driving. It is not necessary to process the entire synthetic image of the target object. It is only necessary to process the driving image corresponding to the target area where the target part is located, which greatly reduces the amount of data processing. At the same time, the area except the target part in the foreground area determined by the pre-segmentation mask is reused, which further improves the efficiency of segmenting the target foreground area.

[0085] In addition, the regional segmentation mask corresponding to the target area can be obtained from the pre-segmentation mask corresponding to the green screen image; the regional segmentation mask is fused with the driving image to obtain a fused driving image. If the fused driving image meets the preset conditions, the fused driving image is directly obtained as the target part after driving, and the pixels of the driving image and the pixels of the background area are no longer used to re-extract the target part after driving, thereby improving the extraction efficiency of the target part and thereby improving the segmentation efficiency of the target foreground area.

[0086] Please refer to FIG5 , which shows a flowchart of an image processing method proposed in another embodiment of the present application. The method can be applied to an electronic device, which can be the terminal 20 or the server 10 in FIG1 . The method includes:

[0087] S210, fusing the pre-segmented mask corresponding to the green screen image and the green screen image to obtain a composite image including a foreground area and a background area; replacing the pixel values ​​of the pixels in the background area of ​​the composite image with the target pixel values ​​to obtain a background replacement image; and determining a target area including the target part from the background replacement image.

[0088] The description of S210 refers to the description of S110 to S130 above and will not be repeated here.

[0089] S220 : Determine a first segmentation mask corresponding to the driving image according to a difference between a pixel value of each pixel point in the driving image and a target pixel value.

[0090] The mask value corresponding to each pixel in the driving image can be determined based on the difference between the pixel value of each pixel in the driving image and the target pixel value; and the first segmentation mask corresponding to the driving image can be determined based on the mask value corresponding to each pixel in the driving image.

[0091] For example, a mask value for each pixel in the driving image may be determined based on a comparison result between a difference between a pixel value of each pixel in the driving image and a target pixel value and a preset difference. The preset difference may be a value set based on demand. For example, the difference between the pixel value of the driving pixel and the target pixel value may be the square of the Euclidean distance between the pixel value of the driving pixel and the target pixel value, and the preset difference may be a threshold value indicating the square of the Euclidean distance.

[0092] If the difference between the pixel value of the driven pixel point and the target pixel value refers to the square of the Euclidean distance between the pixel value of the driven pixel point and the target pixel value, then the calculation process of the difference between the pixel value of the driven pixel point and the target pixel value refers to Formula 1, which is as follows:

[0093] D = (x-p1)^2 + (y-p2)^2 + (z-p3)^2 (-)

[0094] Where D is the square of the Euclidean distance between the pixel value of the driving pixel and the target pixel value, (x, y, z) is the RGB pixel value of the driving pixel, and (p1, p2, p3) is the RGB pixel value of the target pixel.

[0095] Exemplarily, the preset differences include a first threshold and a second threshold, and the first threshold is greater than the second threshold; for each pixel point in the driving image corresponding to the target object, if the difference between the pixel value of the pixel point and the target pixel value is greater than or equal to the first threshold, the mask value of the pixel point is determined to be the first value, if the difference between the pixel value of the pixel point and the target pixel value is less than or equal to the second threshold, the mask value of the pixel point is determined to be the second value, if the difference between the pixel value of the pixel point and the target pixel value is not greater than the first threshold and not less than the second threshold, the mask value of the pixel point can be calculated according to the first value, the second value, and the difference between the pixel value of the pixel point and the target pixel value. Among them, the first value is greater than the second value. For example, the first value can be 1, the second value can be 0, the first threshold can be 40, and the second threshold can be 20.

[0096] If the difference between the pixel value of the pixel point and the target pixel value is not greater than the first threshold and not less than the second threshold, calculating the mask value of the pixel point according to the first value, the second value, and the difference between the pixel value of the pixel point and the target pixel value can include: taking the difference between the difference between the pixel value of the pixel point and the target pixel value and the second threshold as the first result, taking the difference between the first threshold and the second threshold as the second result, and taking the ratio of the first result to the second result as the mask value of the pixel point.

[0097] As above, the calculation process of the mask value of the driving pixel point can be expressed as Formula Two, and Formula Two is as follows: Alpha = c1, D >= Dmax; Alpha = c2, D <= Dmin; Alpha = (D - Dmin) / (Dmax - Dmin), Dmin < D < Dmax; (Two)

[0098] Among them, Alpha is the mask value of the driving pixel point, D is the difference between the pixel value of the driving pixel point and the target pixel value (that is, the square of the Euclidean distance), Dmin is the second threshold, Dmax is the first threshold, c1 is the first value, and c2 is the second value.

[0099] S230. Use the first segmentation mask to extract the driven target part in the driving image.

[0100] After obtaining the first segmentation mask, the first segmentation mask can be fused with the driving image to segment the driving image, obtain the area corresponding to the target part in the target object, and this area is the driven target part, and the posture of the driven target part is the posture of performing the target action.

[0101] The first segmentation mask can include the mask value of each pixel point in the driving image. At this time, fusing the first segmentation mask with the driving image can refer to multiplying the pixel value of each pixel point in the driving image by its corresponding mask value.

[0102] As an embodiment, before S230, the method further includes: performing edge-inward corrosion processing on the edge of the target part in the first segmentation mask to obtain a second segmentation mask corresponding to the driving image; accordingly, S230 includes: using the second segmentation mask to extract the driven target part in the driving image.

[0103] The edge of the target portion in the first segmentation mask may refer to a contour line of the target portion in the first segmentation mask. The edge erosion process may refer to smoothing the edge of the target portion in the first segmentation mask so that changes in pixel values ​​on both sides of the edge of the target portion in the first segmentation mask are smoother and more continuous.

[0104] In some embodiments, the erosion process may be performed on all or part of the edges of the target portion in the first segmentation mask. For example, if the target portion is a head, the area of ​​the face that users typically focus on is the face. In this case, the erosion process may be performed on the edges of the face in the target portion, without performing the erosion process on other edges of the target portion except the face. This can save processing resources and reduce the time required for the erosion process.

[0105] The step of performing edge-in etching on the edges of the target portion in the first segmentation mask to obtain a second segmentation mask corresponding to the driving image includes: performing convolution processing on the edges of the target portion in the first segmentation mask using a convolution kernel of a target size to obtain a third segmentation mask corresponding to the driving image; and smoothing the edges of the target portion in the third segmentation mask using a blur kernel to obtain the second segmentation mask corresponding to the driving image. The target size may be 3x3, and the blur kernel may be 5x5.

[0106] The second segmentation mask may include a respective mask value for each pixel in the driving image. Extracting the target portion after driving in the driving image using the second segmentation mask may mean multiplying the pixel value of each pixel in the driving image by the respective corresponding mask value to obtain the target portion after driving.

[0107] S240 : Generate a target foreground area corresponding to the target object according to the driven target part and the area in the foreground area excluding the target part.

[0108] The description of S240 refers to the description of S150 above and will not be repeated here.

[0109] In this embodiment, a first segmentation mask is determined based on the difference between the pixel value of each pixel point in the driving image and the target pixel value, and then the edge of the target part in the first segmentation mask is processed by the convolution kernel and the blur kernel to achieve smoothing of the edge of the target part in the first segmentation mask. This can ensure that the edge of the target part in the target foreground area obtained by subsequent segmentation based on the first segmentation mask is smooth, and the effect of the target foreground area obtained subsequently can be guaranteed, thereby achieving the purpose of improving the image segmentation effect.

[0110] In addition, in the present application, considering that driving the driving part of the target object may cause other related parts near the driving part to be linked, in the above embodiment, the target area where the target part including the driving part is located is obtained from the background replacement image, and then action driving is performed based on this, instead of obtaining the area where the driving part is located from the background replacement image for action driving, thereby ensuring that the second segmentation mask determined subsequently can accurately express the area where the driving part is located after driving, and the area where the part that is linked with the action of the driving part after driving is located, thereby ensuring the accuracy of subsequent segmentation based on the second segmentation mask.

[0111] In addition, by processing the edge of the target part in the first segmentation mask through the convolution kernel and the blur kernel, the edge of the target part in the first segmentation mask is smoothed, which can ensure that the edge of the target part in the target foreground area obtained by the subsequent segmentation based on the second segmentation mask is smooth, and the effect of the target foreground area obtained subsequently can be guaranteed, thereby achieving the purpose of improving the image segmentation effect.

[0112] Please refer to FIG6 , which shows a flowchart of an image processing method proposed in yet another embodiment of the present application. The method can be applied to an electronic device, which can be the terminal 20 or the server 10 in FIG1 . The method includes:

[0113] S310: Determine a second segmentation mask corresponding to the green screen image.

[0114] The description of S310 refers to the description of S210-S230 above and will not be repeated here.

[0115] S320: Obtain relevant segmentation masks corresponding to relevant areas including the target part in the adjacent green screen images.

[0116] The adjacent green screen image refers to a video frame in the target video that is adjacent to the green screen image and includes the target object; the relevant segmentation mask is used to indicate the area where the target part is located after the driving part in the relevant area is driven.

[0117] A video frame in the target video that needs to be adjusted can be identified as a green screen image. A video frame in the target video that is adjacent to and in front of the green screen image and includes the target object, or a video frame in the target video that is adjacent to and behind the green screen image and includes the target object can be identified as an adjacent green screen image. The video frame that needs to be adjusted is a video frame in which a target portion of the target object is to be driven.

[0118] The relevant region may refer to a region in the adjacent green screen image that includes the target part. For example, if the adjacent green screen image is a video frame that includes a person, and the target part is the person's head, the relevant region may refer to the region in the adjacent green screen image where the person's head is located.

[0119] The relevant segmentation mask may refer to a segmentation mask for segmenting the target portion in the relevant region after driving the driving portion in the relevant region. The relevant segmentation mask may be a mask image of the same size as the relevant region. The relevant segmentation mask may include a mask value corresponding to each pixel in the relevant region.

[0120] As an implementation method, S320 may include: fusing the pre-segmentation masks corresponding to adjacent green screen images and the adjacent green screen images to obtain a related composite image including a related foreground area and a related background area, wherein the related foreground area has a target object; replacing the pixel values ​​of the pixel points in the related background area in the related composite image with the target pixel values ​​to obtain a related background replacement image; determining a related target area including a target part from the related background replacement image; driving the target part in the related target area to obtain a related drive image; and determining the related segmentation mask corresponding to the related target area based on the pixels and target pixel values ​​of the related drive image.

[0121] The area including the target object in the adjacent green screen image is used as the related foreground area, and the area excluding the target object is used as the related background area. The pre-segmentation mask of the adjacent green screen image can be determined by a segmentation model, and then the adjacent green screen images and the pre-segmentation masks corresponding to the adjacent green screen images are fused to obtain a related composite image; the pixel values ​​of the pixel points in the related background area excluding the related foreground area where the target object is located in the related composite image are replaced by the target pixel value to obtain a related background replacement image.

[0122] The relevant target area may refer to an area in the relevant background replacement image that includes the target part. For example, if the relevant background replacement image includes a person and the target part is the person's head, the relevant target area refers to the area in the relevant background replacement image where the person's head is located.

[0123] In some embodiments, the target part in the relevant target area may be driven according to the relevant action, and the posture of the driving part of the target object in the relevant target area is changed to the posture corresponding to the execution of the relevant action. When the posture of the driving part is changed to the posture corresponding to the execution of the relevant action, the image of the target part serves as the relevant driving image, that is, the relevant driving image refers to the relevant target area when the driving part performs the relevant action.

[0124] The related action refers to the action of driving the driving part of the target object in the related target area. It has the same meaning as the target action and is not further described. For example, when the driving part is a person's face (including the mouth), the related action may be the action of the person saying "you".

[0125] For example, the target object in the relevant target area is a person, the driving part is the face (including the mouth), the posture of the person's face in the relevant target area is the image when saying "I", the relevant action is saying the word "we", and the face of the person in the relevant target area is driven according to the relevant action to obtain an image of the person saying "we" as the relevant driving image.

[0126] The mask value of each pixel in the relevant driving image can be determined according to the difference between the pixel value of each pixel in the relevant driving image corresponding to the target object and the target pixel value, and the mask value of each pixel in the relevant driving image can be summarized to obtain the relevant segmentation mask.

[0127] It is understood that the relevant driving image is determined based on the relevant target region, and therefore, the relevant driving image and the relevant target region are the same size. The relevant region is the region in the adjacent green screen image that includes the target portion, and the relevant target region is the region in the relevant background replacement image that includes the target portion. The difference between the relevant background replacement image and the adjacent green screen image lies in the pixel values ​​of the background pixels. Therefore, the relevant target region and the relevant region are also the same size, and the difference between the relevant target region and the relevant region lies in the pixel values ​​of the background pixels. Therefore, after driving the driving portion in the relevant region, the target portion after driving the driving portion in the relevant region can also be segmented using the relevant segmentation mask.

[0128] As an implementation, a comparison result of the difference between the pixel value of each pixel in the relevant driving image and the target pixel value and a preset difference can be determined, and based on the comparison result, a mask value for each pixel in the relevant driving image can be determined. The difference between the pixel value of each pixel in the relevant driving image corresponding to the target object and the target pixel value can refer to the Euclidean distance, cosine similarity, etc., between the pixel value of each pixel in the relevant driving image corresponding to the target object and the target pixel value.

[0129] Exemplarily, for example, the preset difference includes a first threshold and a second threshold, and the first threshold is greater than the second threshold; for each pixel point in the relevant driving image, if the difference between the pixel value of the pixel point and the target pixel value is greater than or equal to the first threshold, the mask value of the pixel point is determined to be the first value; if the difference between the pixel value of the pixel point and the target pixel value is less than or equal to the second threshold, the mask value of the pixel point is determined to be the second value; if the difference between the pixel value of the pixel point and the target pixel value is not greater than the first threshold and not less than the second threshold, the mask value of the pixel point can be calculated based on the first value, the second value and the difference between the pixel value of the pixel point and the target pixel value.

[0130] If the difference between the pixel value of the pixel point and the target pixel value is not greater than the first threshold and not less than the second threshold, then calculating the mask value of the pixel point based on the first value, the second value and the difference between the pixel value of the pixel point and the target pixel value may include: taking the difference between the pixel value of the pixel point and the target pixel value and the difference between the second threshold as the third result, taking the difference between the first threshold and the second threshold as the second result, and comparing the third result with the second result to obtain the mask value of the pixel point.

[0131] As an implementation method, a comparison result of the difference between the pixel value of each pixel point in the relevant driving image and the target pixel value and the preset difference can also be determined. Based on the comparison result, the mask value of each pixel point in the relevant driving image is determined, and the mask value of each pixel point in the relevant driving image is summarized to obtain a relevant area mask. The edge of the target part in the relevant area mask is subjected to edge inward corrosion processing to obtain a relevant segmentation mask.

[0132] The edge of the target portion in the relevant area mask may refer to a contour line of the target portion in the relevant area mask. The edge inward erosion process may refer to smoothing the edge of the target portion in the relevant area mask so that changes in pixel values ​​on both sides of the edge of the target portion in the relevant area mask are smoother and more continuous.

[0133] In some embodiments, the erosion process may be performed on all or part of the edges of the target area in the relevant area mask. For example, if the target area is the head, the area of ​​the face that users typically focus on is the face. In this case, the erosion process may be performed on the edges of the face in the target area, without performing the erosion process on other edges of the target area except the face. This can save processing resources and reduce the time required for the erosion process.

[0134] The step of performing an edge-inward erosion process on the edge of the target portion in the relevant region mask to obtain the relevant segmentation mask includes: performing a convolution process on the edge of the target portion in the relevant region mask using a convolution kernel of a target size to obtain a preprocessing mask; and smoothing the edge of the target portion in the preprocessing mask using a blur kernel to obtain the relevant segmentation mask. The target size may be 3x3, and the blur kernel may be 5x5.

[0135] S330 , performing temporal smoothing processing on the second segmentation mask according to the relevant segmentation mask to obtain a target segmentation mask corresponding to the driving image.

[0136] The second segmentation mask is temporally smoothed using the relevant segmentation mask to avoid excessive jitter of the target foreground area segmented according to the second segmentation mask between the timing of the target video, so that the segmentation results of adjacent green screen images and the masks of the target parts of the target objects in the segmentation results of the green screen images are smoother and more continuous.

[0137] The adjacent green screen images include a first adjacent green screen image located before the green screen image in the target video and a second adjacent green screen image located after the green screen image; the relevant segmentation mask includes a first relevant segmentation mask corresponding to the first adjacent green screen image and a second relevant segmentation mask corresponding to the second adjacent green screen image; S330 may include: performing a weighted summation on the first relevant segmentation mask, the second relevant segmentation mask, and the second segmentation mask to obtain a target segmentation mask. The weights of the first relevant segmentation mask, the second relevant segmentation mask, and the second segmentation mask can be set based on requirements, and the second segmentation mask has the largest weight.

[0138] For example, if the weight of the first relevant segmentation mask is 0.1, the weight of the second relevant segmentation mask is 0.1, and the weight of the second segmentation mask is 0.8, the target segmentation mask of the green screen image is determined as follows: A21 = 0.1*A1 + 0.8*A2 + A3*0.1, where A21 is the second segmentation mask of the green screen image, A1 is the first relevant segmentation mask, A3 is the second relevant segmentation mask, and A1 is the second segmentation mask.

[0139] S340 , extracting the driven target portion in the driven image using the target segmentation mask.

[0140] After obtaining the target segmentation mask, the target segmentation mask can be fused with the driving image to segment the driving image and obtain the target part after driving. The posture of the driving part in the driven target part is the posture of the output target action.

[0141] The target segmentation mask includes a mask value for each pixel in the driving image. In this case, fusing the target segmentation mask with the driving image may refer to multiplying the pixel value of each pixel in the driving image by the corresponding mask value.

[0142] S350 : Generate a target foreground area corresponding to the target object according to the driven target part and the area in the foreground area excluding the target part.

[0143] The description of S350 refers to the description of S150 above and will not be repeated here.

[0144] Referring to Figure 7 , the green screen image and its corresponding pre-segmented mask are fused to create a composite image. The pixel values ​​of the background area in the composite image are then replaced with the target pixel values ​​to create a background replacement image. Since the green screen image only includes the target area (the head), the background replacement image can be directly determined as the target area including the target area.

[0145] Then, face driving is performed on the target area to obtain a corresponding driving image. The initial segmentation mask 81 is determined based on the difference between the pixel value of each pixel in the driving image and the target pixel value of the background pixel in the background replacement image. An enlarged image 812 of the edge local area 811 of the initial segmentation mask 81 shows that the edges of the initial segmentation mask 81 are not smooth and continuous. The initial segmentation mask 81 is further subjected to edge inward and temporal smoothing to obtain a target segmentation mask 82. The enlarged image 822 of the edge local area 821 of the target segmentation mask 82 shows that the edges of the target segmentation mask 82 are smooth and continuous.

[0146] The driven image is segmented using the target segmentation mask 82 to obtain a driven target portion 83. Since the background replacement image corresponding to the synthesized image is used as the target area, the foreground area of ​​the synthesized image no longer includes areas other than the target portion (head). Therefore, the obtained driven target portion 83 can be used as the target foreground area.

[0147] In this embodiment, when processing the green screen image in the target video, the second segmentation mask of the green screen image is temporally smoothed according to the relevant segmentation masks corresponding to the adjacent green screen images, so that the obtained target segmentation mask has a higher accuracy, thereby improving the accuracy of the driven target part extracted according to the target segmentation mask, and further improving the effect of determining the target foreground area.

[0148] In order to more clearly explain the technical solution of the present application, the image processing method of the present application is explained below in combination with an exemplary scenario. In this scenario, the target video is a 2-minute video, the target video is a video of a digital person speaking, and the speech content is A. The speech content of the target video needs to be adjusted to B, and the adjusted video is broadcast live as a live video.

[0149] For any video frame P2 in the target video, determine it as a target video frame, and obtain the previous video frame P1 and the next video frame P3 of P2, where the relevant action of P1 is saying "you", the target action of P2 is saying "we", and the relevant action of P3 is saying "good", the driving part is the face (including the mouth), and the target part is the head; the target object can be a digital human.

[0150] The process of obtaining the relevant segmentation mask of P1:

[0151] P1 is processed by a deep learning-based segmentation mask to obtain a pre-segmentation mask P12 corresponding to P1. P1 and P12 are fused to obtain a related composite image P13. The pixel values ​​of the background area other than the person in P13 are adjusted to the target pixel value RGB (0, 124, 0) to obtain a related background replacement image P14. The related target area P15 corresponding to the head area is determined in P14. The face in P15 is driven according to the action of saying "you" to obtain a related driven image P16 corresponding to the head. According to the difference between the pixel value of each pixel point in P16 and the target pixel value, the mask value of each pixel point in P16 is determined according to Formula 1 and Formula 2. The mask values ​​of each pixel point in P16 are then summarized to obtain the related area mask P17 corresponding to P15.

[0152] Afterwards, the edge of the head in P17 can be convolved with a convolution kernel of the target size to obtain a preprocessing mask P18 corresponding to P17; the edge of the head in P18 can be smoothed with a blur kernel to obtain a related segmentation mask of P1.

[0153] The process of obtaining the second segmentation mask of P2:

[0154] P2 is processed by a deep learning-based segmentation mask to obtain a pre-segmentation mask P22 corresponding to P2. P2 and P22 are fused to obtain a composite image P23. The pixel values ​​of the background area other than the person in P23 are adjusted to the target pixel value RGB (0, 124, 0) to obtain a background replacement image P24. The target area P25 corresponding to the head is determined in P24. The face in P25 is driven according to the action of saying "we" to realize human face driving to obtain a drive image P26 corresponding to the head. According to the difference between the pixel value of each pixel point in P26 and the target pixel value, the mask value of each pixel point in P26 is determined according to Formula 1 and Formula 2. The mask values ​​of each pixel point in P26 are then summarized to obtain the first segmentation mask P27.

[0155] Afterwards, the edge of the head in P27 can be convolved with a convolution kernel of the target size to obtain a third segmentation mask P28; the edge of the head in P28 can be smoothed with a blur kernel to obtain a second segmentation mask of P2.

[0156] The acquisition process of the relevant segmentation mask of P3:

[0157] P3 is processed by the deep learning-based segmentation mask to obtain the pre-segmentation mask P32 corresponding to P3. P3 and P32 are fused to obtain the related synthetic image P33. The pixel values ​​of the background area other than the person in P33 are adjusted to the target pixel value RGB (0, 124, 0) to obtain the related background replacement image P34. The related target area P35 corresponding to the head area is determined in P34. The face in P35 is driven according to the action of saying "OK" to obtain the related drive image P36 corresponding to the head. According to the difference between the pixel value of each pixel point in P36 and the target pixel value, the mask value of each pixel point in P36 is determined according to Formula 1 and Formula 2. The mask values ​​of each pixel point in P36 are then summarized to obtain the related area mask P37 corresponding to the related target area P35.

[0158] Afterwards, the edge of the head in P37 can be convolved with a convolution kernel of the target size to obtain a preprocessing mask P38 corresponding to P37; the edge of the head in P38 can be smoothed with a blur kernel to obtain a related segmentation mask of P3.

[0159] At this point, the relevant segmentation mask of P1, the second segmentation mask of P2, and the relevant segmentation mask of P3 are determined. According to the weight of 0.1 of the relevant segmentation mask of P1, the weight of 0.8 of the second segmentation mask of P2, and the weight of 0.1 of the relevant segmentation mask of P3, the relevant segmentation mask of P1, the second segmentation mask of P2, and the relevant segmentation mask of P3 are weighted summed to obtain the summation result, which is the target segmentation mask P0.

[0160] The driven image P26 is segmented using the target segmentation mask P0 to obtain the driven head P29. The area P210 excluding the head is determined from the foreground area of ​​the composite image. P29 and P210 are spliced ​​into the target object to obtain the target foreground area.

[0161] Afterwards, a preset background image is obtained and used as the background of the target foreground area. The target foreground area is superimposed on the preset background image to obtain the target background replacement image corresponding to P2. After obtaining the target background replacement image corresponding to P2, the target background replacement image can be played in a live broadcast mode to achieve live broadcast of Homo sapiens.

[0162] In this scenario, a fast-driven post-segmentation solution suitable for digital human live streaming is proposed, contributing to efficient digital human live streaming. No manual parameter adjustment is required. Furthermore, the patent rationally utilizes the pre-segmentation results, changing only the segmentation mask (alpha) of the head area to achieve a refined cutout effect. Each image takes only 3ms, meeting the requirements of live streaming.

[0163] This also overcomes the inaccuracy of previously reused segmentation masks caused by changes in cheek size due to driving mouth shape, improving the live broadcast quality of digital humans. Post-driving segmentation is performed on the driver's head, optimizing the CPU (Central Processing Unit) segmentation time to 3 milliseconds per image, leaving ample time for action driving.

[0164] The post-driving segmentation algorithm performs edge corrosion and spatial temporal smoothing based on the color gamut information, which can achieve a refined and temporally stable cutout effect, thereby correcting the problem of exposed facial edges after driving caused by reusing the original segmentation map.

[0165] Please refer to FIG8 , which shows a block diagram of an image processing device according to an embodiment of the present application. The device 900 includes:

[0166] A fusion module 910 is configured to fuse the pre-segmented mask corresponding to the green screen image and the green screen image to obtain a composite image including a foreground region and a background region, wherein the foreground region contains a target object; the target object includes a target part;

[0167] a determination module 920 for determining a target region including a target part from the synthesized image;

[0168] A driving module 930 is configured to drive a target portion in the target area, and determine the target area including the driven target portion as a driving image;

[0169] An extraction module 940 is configured to extract a driven target portion in the driving image based on pixels of the driving image and pixels of the background area;

[0170] The obtaining module 950 is configured to generate a target foreground area corresponding to the target object according to the driven target part and the area in the foreground area excluding the target part.

[0171] Optionally, the determination module 920 is also used to replace the pixel values ​​of the pixel points in the background area of ​​the synthetic image with the target pixel values ​​to obtain a background replacement image; determine the target area including the target part from the background replacement image; accordingly, the extraction module 940 is also used to extract the driven target part in the driving image based on the pixel values ​​of the pixel points of the driving image and the target pixel values.

[0172] Optionally, the extraction module 940 is further configured to determine a first segmentation mask corresponding to the driving image based on the difference between the pixel value of each pixel point in the driving image and the target pixel value; and to extract the driven target portion in the driving image using the first segmentation mask.

[0173] Optionally, the extraction module is also used to determine the mask value corresponding to each pixel point in the driving image based on the difference between the pixel value of each pixel point in the driving image and the target pixel value; and determine the first segmentation mask corresponding to the driving image based on the mask value corresponding to each pixel point in the driving image.

[0174] Optionally, the extraction module 940 is also used to determine that the mask value of the driven pixel point is a first value if the difference between the pixel value of the driven pixel point and the target pixel value is greater than or equal to a first threshold; the driven pixel point is any pixel point in the driven image; if the difference between the pixel value of the driven pixel point and the target pixel value is less than or equal to a second threshold, the mask value of the driven pixel point is determined to be a second value; the first value is greater than the second value; if the difference between the pixel value of the driven pixel point and the target pixel value is not greater than the first threshold, and the difference between the pixel value of the driven pixel point and the target pixel value is not less than the second threshold, the mask value of the driven pixel point is determined according to the difference between the pixel value of the driven pixel point and the target pixel value, the first threshold and the second threshold.

[0175] Optionally, the extraction module 940 is further configured to perform an edge-inward erosion process on the edge of the target portion in the first segmentation mask to obtain a second segmentation mask corresponding to the driving image; and extract the driven target portion in the driving image using the second segmentation mask.

[0176] Optionally, the extraction module 940 is also used to convolve the edge of the target part in the first segmentation mask through a convolution kernel of a target size to obtain a third segmentation mask corresponding to the driving image; and smooth the edge of the target part in the third segmentation mask through a blur kernel to obtain a second segmentation mask corresponding to the driving image.

[0177] Optionally, the green screen image is a video frame included in the target video; the extraction module 940 is also used to obtain a relevant segmentation mask corresponding to a relevant area including the target part in the adjacent green screen image, and the adjacent green screen image refers to a video frame in the target video that is adjacent to the green screen image and includes the target object; the relevant segmentation mask is used to indicate the area where the target part is located after the driving part in the related area is driven; the second segmentation mask is time-smoothed according to the relevant segmentation mask to obtain a target segmentation mask corresponding to the driving image; and the target segmentation mask is used to extract the target part after driving in the driving image.

[0178] Optionally, the extraction module 940 is further used to fuse the pre-segmentation masks corresponding to adjacent green screen images and the adjacent green screen images to obtain a related composite image including a related foreground area and a related background area, wherein the related foreground area has a target object; replace the pixel values ​​of the pixel points in the related background area in the related composite image with the target pixel values ​​to obtain a related background replacement image; determine the related target area including the target part from the related background replacement image; drive the target part in the related target area to obtain a related drive image; and determine the related segmentation mask corresponding to the related target area based on the pixels and target pixel values ​​of the related drive image.

[0179] Optionally, the adjacent green screen image includes a first adjacent green screen image located before the green screen image in the target video, and a second adjacent green screen image located after the green screen image; the relevant segmentation mask includes a first relevant segmentation mask corresponding to the first adjacent green screen image and a second relevant segmentation mask corresponding to the second adjacent green screen image; the extraction module 940 is further used to perform weighted summation on the first relevant segmentation mask, the second relevant segmentation mask and the second segmentation mask to obtain a target segmentation mask.

[0180] Optionally, the obtaining module 950 is further configured to use a preset background image as the background of the target foreground area, fuse the target foreground area and the preset background image, and obtain a target background replacement image.

[0181] Optionally, the extraction module 940 is further used to obtain a regional segmentation mask corresponding to the target area from the pre-segmentation mask corresponding to the green screen image; fuse the regional segmentation mask with the driving image to obtain a fused driving image; if the fused driving image does not meet the preset conditions, extract the target part after driving in the driving image based on the pixels of the driving image and the pixels of the background area.

[0182] It should be noted that the device embodiment in this application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0183] FIG9 shows a block diagram of an electronic device for executing an image processing method according to an embodiment of the present application. The electronic device may be the terminal 20 or the server 10 in FIG1 . It should be noted that the computer system 1200 of the electronic device shown in FIG9 is merely an example and should not limit the functionality or scope of use of the embodiments of the present application.

[0184] As shown in Figure 9, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, ROM 1202 and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0185] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1210 as needed, so that computer programs read therefrom can be installed into the storage section 1208 as needed.

[0186] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, the various functions defined in the system of the present application are executed.

[0187] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0189] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0190] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable storage medium carries computer-readable instructions, and when the computer-readable instructions are executed by a processor, the method of any of the above embodiments is implemented.

[0191] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the method of any of the above embodiments.

[0192] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0193] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable an electronic device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0194] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in this application. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of this application is limited only by the appended claims.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method is performed by an electronic device, and includes: The pre-segmented mask corresponding to the green screen image and the green screen image are fused to obtain a composite image including a foreground area and a background area, wherein the foreground area has a target object; the target object includes a target part; determining a target region including the target site from the composite image; driving a target part in the target region, and determining the target region including the driven target part as a driving image; Extracting a driven target part in the driving image according to pixels of the driving image and pixels of the background area; A target foreground area corresponding to the target object is generated according to the driven target part and an area in the foreground area except the target part.

2. The method according to claim 1, characterized in that Before determining the target area including the target part from the synthetic image, the method further includes: Replacing the pixel values ​​of the pixels in the background area of ​​the synthetic image with target pixel values ​​to obtain a background replacement image; The step of determining a target area including the target part from the composite image comprises: Determining a target area including the target part from the background replacement image; The step of extracting the driven target part in the driving image according to the pixels of the driving image and the pixels of the background area comprises: The driven target part in the driving image is extracted according to the pixel value of the pixel point of the driving image and the target pixel value.

3. The method according to any one of claims 1 to 2, characterized in that: The step of extracting the driven target part in the drive image according to the pixel value of the pixel point of the drive image and the target pixel value comprises: Determining a first segmentation mask corresponding to the driving image according to a difference between a pixel value of each pixel point in the driving image and the target pixel value; The driven target part in the driven image is extracted using the first segmentation mask.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining a first segmentation mask corresponding to the driving image according to a difference between a pixel value of each pixel point in the driving image and the target pixel value comprises: Determining a mask value corresponding to each pixel point in the driving image according to a difference between a pixel value of each pixel point in the driving image and the target pixel value; A first segmentation mask corresponding to the driving image is determined according to a mask value corresponding to each pixel in the driving image.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining the mask value corresponding to each pixel point in the driving image according to the difference between the pixel value of each pixel point in the driving image and the target pixel value comprises: If the difference between the pixel value of the driving pixel point and the target pixel value is greater than or equal to a first threshold, determining the mask value of the driving pixel point to be a first value; the driving pixel point is any pixel point in the driving image; If the difference between the pixel value of the driving pixel point and the target pixel value is less than or equal to a second threshold, determining the mask value of the driving pixel point to be a second value; and the first value is greater than the second value; If the difference between the pixel value of the driven pixel point and the target pixel value is not greater than a first threshold, and the difference between the pixel value of the driven pixel point and the target pixel value is not less than a second threshold, the mask value of the driven pixel point is determined according to the difference between the pixel value of the driven pixel point and the target pixel value, the first threshold and the second threshold.

6. The method according to any one of claims 1 to 3, characterized in that: Before extracting the driven target part in the driven image using the first segmentation mask, the method further includes: Performing an edge-inward erosion process on the edge of the target portion in the first segmentation mask to obtain a second segmentation mask corresponding to the drive image; The step of extracting the driven target part in the driven image by using the first segmentation mask includes: The driven target part in the driven image is extracted using the second segmentation mask.

7. The method according to any one of claims 1 to 6, characterized in that: The step of performing an edge-inward erosion process on the edge of the target portion in the first segmentation mask to obtain a second segmentation mask corresponding to the drive image includes: Performing convolution processing on the edge of the target part in the first segmentation mask by using a convolution kernel of a target size to obtain a third segmentation mask corresponding to the driving image; The edge of the target part in the third segmentation mask is smoothed by using a fuzzy kernel to obtain a second segmentation mask corresponding to the driving image.

8. The method according to any one of claims 1 to 6, characterized in that: The green screen image is a video frame included in the target video; before using the second segmentation mask to extract the driven target part in the driven image, the method further includes: Acquire a relevant segmentation mask corresponding to a relevant area including a target part in an adjacent green screen image, wherein the adjacent green screen image refers to a video frame in the target video that is adjacent to the green screen image and includes the target object; the relevant segmentation mask is used to indicate an area where the target part is located after the driving part in the relevant area is driven; Performing temporal smoothing processing on the second segmentation mask according to the related segmentation mask to obtain a target segmentation mask corresponding to the driving image; The step of extracting the driven target part in the driven image by using the second segmentation mask comprises: The driven target part in the driven image is extracted using the target segmentation mask.

9. The method according to any one of claims 1 to 8, characterized in that: The step of obtaining a relevant segmentation mask corresponding to a relevant target area including the target part in the adjacent green screen image includes: The pre-segmented masks corresponding to the adjacent green screen images and the adjacent green screen images are fused to obtain a related synthetic image including a related foreground area and a related background area, wherein the related foreground area has the target object; Replacing the pixel values ​​of the pixels in the relevant background area in the relevant synthetic image with the target pixel values ​​to obtain a relevant background replacement image; Determining a relevant target area including a target part from the relevant background replacement image; Driving the target part in the relevant target area to obtain a relevant driving image; A relevant segmentation mask corresponding to the relevant target area is determined according to the pixels of the relevant drive image and the target pixel value.

10. The method according to any one of claims 1 to 8, characterized in that: The adjacent green screen image includes a first adjacent green screen image located before the green screen image in the target video, and a second adjacent green screen image located after the green screen image; the relevant segmentation mask includes a first relevant segmentation mask corresponding to the first adjacent green screen image and a second relevant segmentation mask corresponding to the second adjacent green screen image; The step of performing temporal smoothing processing on the second segmentation mask according to the related segmentation mask to obtain a target segmentation mask corresponding to the driving image includes: A weighted sum is performed on the first related segmentation mask, the second related segmentation mask, and the second segmentation mask to obtain the target segmentation mask.

11. The method according to any one of claims 1 to 10, characterized in that: After generating the target foreground area corresponding to the target object according to the driven target part and the area in the foreground area other than the target part, the method further includes: The preset background image is used as the background of the target foreground area, and the target foreground area and the preset background image are fused to obtain a target background replacement image.

12. The method according to any one of claims 1 to 10, characterized in that: Before extracting the driven target part in the driving image according to the pixels of the driving image and the pixels of the background area, the method further includes: Acquire a regional segmentation mask corresponding to the target area from the pre-segmentation mask corresponding to the green screen image; Fusing the region segmentation mask with the driving image to obtain a fused driving image; The step of extracting the driven target part in the driving image according to the pixels of the driving image and the pixels of the background area comprises: If the fused driving image does not satisfy a preset condition, the driven target part in the driving image is extracted according to pixels of the driving image and pixels of the background area.

13. An image processing device, characterized in that: The device is run on an electronic device, and the device includes: A fusion module, used for fusing the pre-segmentation mask corresponding to the green screen image and the green screen image to obtain a composite image including a foreground area and a background area, wherein the foreground area has a target object; the target object includes a target part; A determination module, configured to determine a target area including the target part from the composite image A driving module, used for driving a target part in the target area, and determining the target area including the driven target part as a driving image; An extraction module, used for extracting a driven target part in the driving image according to pixels of the driving image and pixels of the background area; The acquisition module is used to generate a target foreground area corresponding to the target object according to the driven target part and the area in the foreground area other than the target part.

14. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to execute the method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises a computer program, which is stored in a computer-readable storage medium. The computer program is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 12.

Images