Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a to-be-matted target image and description information of a target background; the target background is different from the background in the target image; fusing the image features of the target image and the description features of the description information to obtain fused features; and decoding based on the fusion feature to obtain a target decoding result, the target decoding result comprising a foreground segmentation result of the target image and a fusion image of the foreground and the target background in the target image. In this way, rich image information can be provided for a user.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] In the field of image processing, image cutout and image synthesis are two important technical directions. Image cutout technology aims to accurately extract the foreground part from the original image, while image synthesis technology focuses on fusing the extracted foreground part with the new background to generate a realistic image.

[0003] However, simply providing foreground segmentation results, or simply providing a fusion image of the foreground and new background, is not sufficient to meet all application requirements. It is of great significance to provide users with richer information for subsequent image applications. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an image processing method, apparatus, device and storage medium, which can provide users with rich image information.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide an image processing method, the method comprising: obtaining a target image to be cutout, and description information of a target background; the target background is different from the background in the target image; fusing the image features of the target image and the description features of the description information to obtain fused features; decoding based on the fused features to obtain a target decoding result, the target decoding result comprising a foreground segmentation result of the target image, and a fused image of the foreground and the target background in the target image.

[0006] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide an image processing device, including: an acquisition module, a fusion module and a decoding module. The acquisition module is used to obtain the image association information of the target image to be cut out and the target background; the target background is different from the background in the target image; the fusion module is used to fuse the image features of the target image and the image association features of the image association information to obtain the image fusion features; the decoding module is used to decode based on the image fusion features to obtain the corresponding target decoded image, and the target decoded image includes the target cutout image of the target image and the fusion image of the foreground and the target background in the target image.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0008] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0009] The above scheme decodes based on the fusion feature to obtain the target decoding result, which includes the foreground segmentation result of the target image and the fusion image of the foreground and the target background in the target image. It can be seen that the present application decodes synchronously to obtain the foreground segmentation result and the fusion image. Among them, the method of synchronously decoding two images in the present application can provide users with rich image information, thereby meeting the needs of different users or different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of an image processing method provided by the present application;

[0011] Figure 2 It is a flowchart of an embodiment of determining a foreground segmentation result of a target image provided by the present application;

[0012] Figure 3 yes Figure 2 The flowchart of step S24 is shown as an embodiment;

[0013] Figure 4 is a flowchart of another embodiment of the image processing method provided by the present application;

[0014] Figure 5 It is a schematic diagram of a framework of an embodiment of an image processing device provided by the present application;

[0015] Figure 6 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0016] Figure 7 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0018] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0019] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of an embodiment of an image processing method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily refer to FIG. Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0020] S11: Obtain the target image to be cut out and description information of the target background; the target background is different from the background in the target image.

[0021] This embodiment is used to fuse the image features of the target image and the descriptive features of the descriptive information to obtain the fused features, decode based on the fused features, and synchronously output the foreground segmentation result of the target image and the fused image of the foreground and target background in the target image, so as to provide users with richer information and thus meet the needs of different users or different application scenarios.

[0022] Among them, the fused image is the result of extracting the foreground object from the target image and synthesizing it with the new background (target background). The fused image provides the user with the possibility of placing the foreground object in different environments, increasing the diversity and interest of the image. The foreground segmentation result is generally a binary or multi-valued image in which the foreground and the background are clearly distinguished. In this embodiment, the foreground segmentation result can be used for image editing or target detection, or when the background of the decoded fused image does not meet the user's requirements, the foreground segmentation result can be used to fuse the foreground with a new background image that the user likes to generate a new image that meets the user's requirements.

[0023] Generally speaking, the target image consists of three parts, namely the foreground, the background and the uncertain area. The uncertain area is generally the edge area of ​​the foreground, where the pixels in the area are uncertain whether they belong to the foreground or the background.

[0024] In one embodiment, the foreground segmentation result is a target cutout image of the target image, which is a binary image, through which the foreground and background in the target image can be clearly distinguished. Specifically, the pixel values ​​of the foreground and background in the target cutout image are different. For example, the pixel value of the foreground is set to 255 (the color seen by the user is white), and the pixel value of the background is set to 0 (the color seen by the user is black).

[0025] The target image in this embodiment is an image to be cut out and the old background to be replaced by the new background, and the foreground in the target image can be any target object to be extracted and merged with the new background to replace the old background. For example, the target object is a person, or any target object that is a foreground in the target image (such as a car or a house, etc.).

[0026] In this embodiment, the target background is different from the background in the target image, and is a new background to be merged with the foreground in the target image. The description information of the target background is the description information about the new background. The description information may be, but is not limited to, an image, or may be a description text about the target background, or may be a voice description of the target background. The specific description may be determined according to actual needs.

[0027] S12: Fusing the image features of the target image and the descriptive features of the descriptive information to obtain fused features.

[0028] In this embodiment, both the image features of the target image and the description features of the description information can be extracted using a related feature extraction network.

[0029] In one embodiment, the description information of the target background is a description text about the target background, and the description feature is a text vector obtained by encoding the description text. The description text of the target background can be a description text obtained by performing speech recognition on a speech input by a user, or can be a description text directly input by a user through an input interface.

[0030] In some embodiments, the fused features are obtained by performing attention processing on the image features of the target image and the descriptive features of the descriptive information using a relevant attention model, or by weighted summing the image features and the text features. The obtained fused features contain both the information of the target image itself and the descriptive information of the new background.

[0031] In a specific embodiment, the fusion feature is an attention matrix obtained by using a relevant attention model to perform attention processing on the image features of the target image and the descriptive features of the descriptive information. The attention matrix is ​​used to guide the decoder to pay attention to the area in the target image (background area) that needs to be paid attention to during the subsequent decoding process, and to generate a target background described by the descriptive features in the area of ​​interest.

[0032] It should be noted that the above-mentioned feature extraction network, attention model, decoder, etc. are all obtained through training.

[0033] In one implementation scenario, the target image to be cut out is a plurality of image frames in the video to be processed, wherein the plurality of image frames are images collected at different time sequences. In order to fuse the foreground and target background in the plurality of image frames, the target background in the obtained fused image is a dynamically changing background. Before the image features of the target image and the descriptive features of the descriptive information are fused to obtain the fused features, the descriptive information of the target background is dynamically encoded using a dynamic encoder to obtain the descriptive features corresponding to each image frame. The feature representations of the descriptive features corresponding to each image frame are different, and the descriptive features corresponding to each image frame are used to fuse with the image features of the corresponding image frame to obtain the corresponding fused features.

[0034] It can be understood that although the feature representations of the descriptive features corresponding to each image frame are different, each feature representation is a feature representation of the descriptive text. Therefore, the objects contained in the background of the fused image finally decoded based on the fusion features of each image frame and the corresponding feature representation are roughly the same. It is just that due to the different specific feature representations, the objects in the background will be slightly different for different fused images, for example, the position of the object or the color of the object will be slightly different, making the new background of the processed video appear to be dynamically changing visually.

[0035] For example, the description information of the target background is "under a clear sky, there is a cow on the green grass", and the corresponding new background is a background image containing blue sky, white clouds, green grass and cows. The description information of the target background "under a clear sky, there is a cow on the green grass" is dynamically encoded using a dynamic encoder to obtain the description features corresponding to each image frame. The different description features can be fused with the image features of the image frames at different time sequences to obtain the fused images corresponding to each image frame. Among them, the backgrounds in the fused images at different time sequences are all background images including blue sky, white clouds, green grass and cows, but the backgrounds in the fused images at different time sequences are dynamically changing. For example, the positions of clouds, cows or the colors of grass in the fused images at different time sequences are different.

[0036] In this embodiment, the description text is integrated into the video processing flow, so that the new background in the generated fused image can be intelligently matched with the description text, enhancing the intelligence and overall effect of background generation. Furthermore, the application of the dynamic text encoder can make the background in the fused image change in real time, achieving a more natural and coherent video effect.

[0037] S13: Decoding is performed based on the fusion features to obtain a target decoding result, where the target decoding result includes a foreground segmentation result of the target image and a fusion image of the foreground and the target background in the target image.

[0038] In one embodiment, the decoder includes a first decoder and a second decoder, and the foreground segmentation result (image) and the fused image are decoded by different decoding modules. Specifically: the first decoder is used to perform a first decoding based on the fusion feature to obtain a fused image of the foreground and the target background in the target image; and the second decoder is used to perform a second decoding based on the fusion feature to obtain the foreground segmentation result of the target image. That is, in this embodiment, the fused image is obtained by decoding the first decoder based on the fusion feature, and the foreground segmentation result is obtained by decoding the second decoder based on the fusion feature.

[0039] In a specific embodiment, the second decoder includes an encoder, a segmentation decoder and a cutout decoder. The encoder is used to further encode the fused features to obtain the encoded features, the segmentation decoder is used to identify the foreground object and the background area in the image based on the information in the encoded features, and generate a corresponding segmentation map (ternary segmentation map); the cutout coder is used to perform fine cutout based on the ternary segmentation map and the encoded features to obtain an initial cutout image, and to perform preset calculations on the pixel values ​​of the corresponding pixels in the ternary segmentation map and the initial cutout image to obtain a more accurate foreground segmentation result (i.e., the target cutout image of the target image).

[0040] Specifically, see Figure 2 , Figure 2 1 is a flow chart of an embodiment of determining the foreground segmentation result of a target image provided by the present application. In this embodiment, a second decoder is used to perform a second decoding based on the fusion feature to obtain the foreground segmentation result of the target image, including:

[0041] S21: Encode the fused features using an encoder to obtain encoded features.

[0042] In this embodiment, the encoder includes multiple encoding layers with different resolutions, and the multiple encoding layers are arranged in descending order of resolution, and each encoding layer is used to perform downsampling processing on the input first input data by a corresponding preset multiple. Among them, each resolution described in this embodiment can be preset according to actual needs.

[0043] In this embodiment, the fusion feature is encoded by using an encoder to obtain the encoded feature, including: using each encoding layer to downsample the input first input data to obtain the encoded feature output by each encoding layer.

[0044] In one embodiment, the first input data of the first coding layer is the fusion feature, and the first input data of the non-first coding layer is the coding feature output by the previous coding layer. The coding feature output by each layer is a coding feature map, and as the data is transmitted layer by layer, the resolution / size of the coding feature map is gradually reduced.

[0045] In another embodiment, in order to extract richer information from the fused features, the first input data of the first coding layer can be set as the fused features, and the first input data of the non-first coding layer includes the fused features and the coding features output by the previous coding layer.

[0046] S22: Use the segmentation decoder to perform a third decoding on the encoded features to obtain a ternary segmentation map.

[0047] In this embodiment, the segmented decoder includes a plurality of first decoding layers of different resolutions, and the plurality of first decoding layers are arranged in order of resolution from small to large. Each segmented decoder is used to perform upsampling processing corresponding to a preset multiple on the input second input data, and in addition, the number of the first decoding layers and the encoding layers is the same, and each segmented decoder has encoding layers of the same resolution.

[0048] In this embodiment, a segmentation decoder is used to perform a third decoding on the encoded features to obtain a ternary segmentation map, including: using each first decoding layer to upsample the input second input data to obtain a ternary segmentation map output by each first decoding layer.

[0049] Among them, multiple first decoding layers are arranged in order from small to large resolution, the second input data of the first decoding layer in the first position is the coding feature output by the last coding layer, and the second input data of the first decoding layer in the non-first position is the ternary segmentation map output by the previous first decoding layer. Among them, the ternary segmentation map is an image mask used to mark whether each pixel in the image belongs to the foreground, background or uncertain (or boundary) area. It is usually composed of three different values ​​or colors. The ternary segmentation map can be understood as a rough division of the image, which divides the image into foreground, background and unknown area to be determined. In general, the pixel values ​​corresponding to pixels in different areas are different. Exemplarily, the pixel value of the pixel in the area belonging to the foreground is determined as the first pixel value, the pixel value of the pixel in the background area is determined as the second pixel value, and the pixel value of the pixel in the uncertain area is determined as the third pixel value.

[0050] S23: Using a cutout decoder to perform a fourth decoding based on the encoded features and the ternary segmentation map to obtain an initial cutout image.

[0051] The cutout decoder includes a plurality of second decoding layers of resolutions, and the plurality of second decoding layers are arranged in order of resolution from small to large, wherein each second decoding layer has a coding layer of the same resolution and a first decoding layer of the same resolution.

[0052] In this embodiment, a cutout decoder is used to perform fourth decoding based on the encoding features and the ternary segmentation map to obtain an initial cutout image, including the following steps:

[0053] First, each second decoding layer is used to perform up-sampling processing on the input third input data to obtain a cutout image output by each second decoding layer.

[0054] Second, the cutout image output by the last second decoding layer is used as the initial cutout image.

[0055] In one embodiment, among the multiple second decoding layers, the third input data of the second decoding layer located at the first position is the coding feature output by the last coding layer. For each second decoding layer located at a non-first position, each second decoding layer located at a non-first position is sequentially used as a target decoding layer, wherein the third input data of the target decoding layer includes: the cutout image output by the previous second decoding layer, the coding feature output by the coding layer with the same resolution as the target decoding layer, and the second input data of the first decoding layer with the same resolution as the target decoding layer.

[0056] That is, each second coding layer that is not in the first position in the cutout decoder will fuse the cutout image output by the previous second decoding layer, the coding features output by the coding layer with the same resolution as the target decoding layer, and the second input data of the first decoding layer with the same resolution as the target decoding layer (i.e., the ternary segmentation map output by the previous first decoding layer) for upsampling to obtain the cutout image output by each second decoding layer. The cutout result is optimized by using the ternary segmentation image to gradually separate the foreground and background.

[0057] The above-mentioned initial cutout image refers to the cutout image output by the last second decoding layer.

[0058] In one implementation, the initial cutout image may be used as the target cutout image.

[0059] In another embodiment, the target cutout image may be obtained by further adopting the method of step S24.

[0060] S24: performing a preset calculation on the pixel values ​​of corresponding pixels in the ternary segmentation image and the initial cutout image to obtain a target cutout image of the target image.

[0061] In this embodiment, considering that the initial cutout image obtained by the above decoding may not be accurate enough, in order to improve the accuracy of the final target cutout image, the pixel values ​​of the corresponding pixel points in the ternary segmentation map output by the last layer of the first decoding layer and the initial cutout image can be preset calculated to obtain a more accurate target cutout image.

[0062] Specifically, see Figure 3 , Figure 3 yes Figure 2 The flowchart of step S24 is shown as an embodiment. This embodiment includes:

[0063] S31: taking the ternary segmentation map output by the last first decoding layer in the segmentation decoder as the target ternary map.

[0064] In this embodiment, each pixel point in the target ternary image is marked as a pixel value corresponding to the foreground, the background or the uncertain area.

[0065] S32: Perform expansion and erosion operations on the target ternary image respectively to obtain a corresponding expansion result image and an erosion result image.

[0066] Step S32 is to adjust the target ternary image through morphological operations (dilation and erosion).

[0067] Among them, the foreground area in the target ternary image is expanded to expand the edge of the original foreground outward to reduce the jagged or discontinuous phenomenon of the edge; the foreground area in the target ternary image is eroded to remove small foreground isolated points or refine the foreground edge, so that the transition between the foreground and the background in the eroded result image is smoother.

[0068] S33: Obtain the product of the pixel values ​​of the corresponding pixel points in the initial cutout image and the dilation result image to serve as the pixel value of the corresponding pixel point in the intermediate cutout image.

[0069] Specifically, the product of the pixel values ​​of the corresponding pixels in the initial cutout image and the dilation result image is calculated so that the obtained intermediate cutout image retains the details in the initial cutout image, and the dilation result is used to enhance the saliency of the foreground area.

[0070] S34: Obtain the sum of the pixel values ​​corresponding to the corresponding pixel points in the intermediate cutout image and the erosion result image as the target pixel value of the corresponding pixel point in the target cutout image.

[0071] The target pixel value of a first pixel point belonging to the foreground in the target cutout image is a first pixel value, and the target pixel value of a second pixel point belonging to the background is a second pixel value.

[0072] Specifically, for each pixel point in the intermediate cutout image and the erosion result image, the sum of the pixel values ​​of the corresponding pixel points in the intermediate cutout image and the erosion result image is calculated, and the pixel point is classified as foreground or background according to the calculated sum.

[0073] Among them, the target pixel value of the first pixel point belonging to the foreground in the target cutout image is the first pixel value, and the target pixel value of the second pixel point belonging to the background is the second pixel value, so as to clearly distinguish the foreground and background in the target cutout image.

[0074] The above scheme decodes based on the fusion feature to obtain the target decoding result, which includes the foreground segmentation result of the target image and the fusion image of the foreground and the target background in the target image. It can be seen that the present application decodes synchronously to obtain the foreground segmentation result and the fusion image. Among them, the method of synchronously decoding two images in the present application can provide users with rich image information, thereby meeting the needs of different users or different application scenarios.

[0075] In one specific embodiment, see Figure 4 , Figure 4 It is a flowchart of another embodiment of the image processing method provided by the present application.

[0076] In this embodiment, a plurality of image frames in a target image to be cut out in a video and a description text about a newly added target background are first obtained.

[0077] Among them, for each image frame, the image features of each image frame are extracted; for the description text of the target image, the description text is firstly encoded to obtain a first text vector, and then a dynamic encoder is used to dynamically encode based on the first text vector to obtain a plurality of second text vectors, wherein different second text vectors are used to fuse with the image features of the corresponding image frame to obtain corresponding fusion features.

[0078] Further, the pre-trained decoder is used to perform decoding based on the fusion features to obtain the foreground segmentation result of the target image and the fusion image of the foreground and the target background in the target image. The specific process of using the decoder to perform decoding based on the fusion features to obtain the foreground segmentation result of the target image and the fusion image of the foreground and the target background in the target image can be referred to the above description, and no further details are given here.

[0079] Before using the pre-trained decoder to perform decoding based on the fusion features, the fusion features may be denoised to obtain processed fusion features.

[0080] Optionally, the correlation model may be used to obtain an initial fused image of each image frame and the corresponding target background based on the image features of each image frame and the second text vector of the description text of the target background, and the initial fused image may be used as a fused feature for subsequent processing.

[0081] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of an image processing device provided by the present application. In this embodiment, the image processing device 50 includes an acquisition module 51, a fusion module 52 and a decoding module 53. The acquisition module 51 is used to obtain image association information of a target image to be cut out and a target background; the target background is different from the background in the target image; the fusion module 52 is used to fuse the image features of the target image and the image association features of the image association information to obtain an image fusion feature; the decoding module 53 is used to decode based on the image fusion feature to obtain a corresponding target decoded image, and the target decoded image includes a target cutout image of the target image, and a fusion image of the foreground in the target image and the target background.

[0082] In some embodiments, the description information acquired by the acquisition module 51 is a description text about the target background, and the description feature is a text vector obtained by encoding the description text; and / or, the foreground segmentation result decoded by the decoding module 53 is a target cutout image of the target image, and the pixel values ​​corresponding to the foreground and background of the target image in the target cutout image are different.

[0083] In some embodiments, the fusion module 52 fuses the image features of the target image and the descriptive features of the descriptive information to obtain the fused features, including: performing attention processing on the image features and the descriptive features to generate the fused features.

[0084] In some embodiments, the target image to be cutout acquired by the acquisition module 51 is a plurality of image frames in the video to be processed; before the fusion module 52 fuses the image features of the target image and the descriptive features of the descriptive information to obtain the fused features, it also includes: dynamically encoding the description information using a dynamic encoder to obtain the descriptive features corresponding to each of the image frames; wherein the feature representations of the descriptive features corresponding to each of the image frames are different, and the descriptive features corresponding to the image frames are used to fuse with the image features of the image frames to obtain the corresponding fused features.

[0085] In some embodiments, the decoding module 53 performs decoding based on the fusion feature to obtain a target decoding result, including: using a first decoder to perform a first decoding based on the fusion feature to obtain a fused image of the foreground in the target image and the target background; and using a second decoder to perform a second decoding based on the fusion feature to obtain a foreground segmentation result of the target image.

[0086] In some embodiments, the foreground segmentation result decoded by the decoding module 53 is a target cutout image of the target image, and the pixel values ​​corresponding to the foreground and background of the target image in the target cutout image are different. The second decoder includes an encoder, a segmentation decoder and a cutout decoder; the second decoding based on the fusion feature using the second decoder to obtain the foreground segmentation result of the target image includes: encoding the fusion feature using the encoder to obtain the encoded feature; performing a third decoding on the encoded feature using the segmentation decoder to obtain a ternary segmentation map; performing a fourth decoding based on the encoded feature and the ternary segmentation map using the cutout decoder to obtain an initial cutout image; performing a preset calculation on the pixel values ​​of corresponding pixel points in the ternary segmentation map and the initial cutout image to obtain the target cutout image of the target image.

[0087] In some embodiments, the encoder includes multiple encoding layers with different resolutions; the encoding of the fused features by the encoder to obtain the encoding features includes: downsampling the first input data by each encoding layer to obtain the encoding features output by each encoding layer; wherein the multiple encoding layers are arranged in descending order of resolution, the first input data of the encoding layer at the first position is the fused features, and the first input data of the encoding layer not at the first position includes the fused features and the encoding features output by the previous encoding layer.

[0088] In some embodiments, the segmentation decoder includes multiple first decoding layers of different resolutions, and the cutout decoder includes multiple second decoding layers of different resolutions; the use of the segmentation decoder to perform a third decoding on the encoded features to obtain a corresponding ternary segmentation map includes: using each of the first decoding layers to up-sample the second input data input to obtain a ternary segmentation map output by each of the first decoding layers; wherein the multiple first decoding layers are arranged in order of resolution from small to large, the second input data of the first decoding layer at the first position is the encoded features output by the last encoding layer, and the second input data of the first decoding layer not at the first position is the ternary segmentation map output by the previous first decoding layer; the use of the cutout decoder to perform a third decoding on the encoded features based on the encoded features and the ternary segmentation map Perform the fourth decoding to obtain an initial cutout image, including: respectively using each of the second decoding layers to upsample the third input data to obtain the cutout image output by each of the second decoding layers; wherein the multiple second decoding layers are arranged in order of resolution from small to large, the third input data of the second decoding layer at the first position is the coding feature output by the last coding layer, and each second decoding layer not at the first position is sequentially used as the target decoding layer, and the third input data of the target decoding layer includes: the cutout image output by the previous second decoding layer, the coding feature output by the coding layer with the same resolution as the target decoding layer, and the second input data of the first decoding layer with the same resolution as the target decoding layer; the cutout image output by the last second decoding layer is used as the initial cutout image.

[0089] In some embodiments, the pixel values ​​of corresponding pixels in the ternary segmentation map and the initial cutout image are preset for calculation to obtain the target cutout image of the target image, including: taking the ternary segmentation map output by the last first decoding layer in the segmentation decoder as the target ternary map; performing dilation and erosion operations on the target ternary map respectively to obtain a corresponding dilated result image and an eroded result image; obtaining the product of the pixel values ​​of corresponding pixels in the initial cutout image and the dilated result image as the pixel value of the corresponding pixel in the intermediate cutout image; obtaining the sum of the pixel values ​​corresponding to the corresponding pixel in the intermediate cutout image and the eroded result image as the target pixel value of the corresponding pixel in the target cutout image; wherein the target pixel value of the first pixel belonging to the foreground in the target cutout image is the first pixel value, and the target pixel value of the second pixel belonging to the background is the second pixel value.

[0090] See also Figure 6 , Figure 6 Schematic diagram of the framework of an embodiment of an electronic device provided by the present application. In this embodiment, the electronic device 60 includes a memory 61 and a processor 62 coupled to each other.

[0091] The memory 61 stores program instructions, and the processor 62 is used to execute the program instructions stored in the memory 61 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0092] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any of the above-mentioned embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0093] See also Figure 7 , Figure 7 It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 70 of the embodiment of the present application stores a program instruction 71, and when the program instruction 71 is executed, the method provided by any embodiment of the above method and any non-conflicting combination is implemented. Among them, the program instruction 71 can form a program file and be stored in the above-mentioned computer-readable storage medium 70 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 70 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0094] The above scheme decodes based on the fusion feature to obtain the target decoding result, which includes the foreground segmentation result of the target image and the fusion image of the foreground and the target background in the target image. It can be seen that the present application decodes synchronously to obtain the foreground segmentation result and the fusion image. Among them, the method of synchronously decoding two images in the present application can provide users with rich image information, thereby meeting the needs of different users or different application scenarios.

[0095] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0096] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0101] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0102] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a target image to be cut out and description information of a target background; the target background is different from the background in the target image; fusing the image features of the target image and the description features of the description information to obtain a fused feature; Decoding is performed based on the fusion feature to obtain a target decoding result, wherein the target decoding result includes a foreground segmentation result of the target image and a fusion image of the foreground in the target image and the target background.

2. The method according to claim 1, characterized in that: The description information is a description text about the target background, and the description feature is a text vector obtained by encoding the description text; And / or, the foreground segmentation result is a target cutout image of the target image, and the pixel values ​​corresponding to the foreground and background of the target image in the target cutout image are different.

3. The method according to claim 1, characterized in that The fusing the image feature of the target image and the description feature of the description information to obtain the fused feature includes: Attention processing is performed on the image features and the description features to generate the fused features.

4. The method according to claim 1, characterized in that The target image to be cut out is a plurality of image frames in the video to be processed; Before fusing the image feature of the target image and the description feature of the description information to obtain the fused feature, the method further includes: The description information is dynamically encoded using a dynamic encoder to obtain description features corresponding to each of the image frames; wherein the feature representations of the description features corresponding to each of the image frames are different, and the description features corresponding to the image frames are used to fuse with the image features of the image frames to obtain corresponding fused features.

5. The method according to claim 1, characterized in that The decoding based on the fusion feature to obtain a target decoding result includes: Using a first decoder to perform a first decoding based on the fusion feature to obtain a fusion image of the foreground in the target image and the target background; and A second decoder is used to perform a second decoding based on the fusion feature to obtain a foreground segmentation result of the target image.

6. The method according to claim 5, characterized in that The foreground segmentation result is a target cutout image of the target image, the pixel values ​​corresponding to the foreground and the background of the target image in the target cutout image are different, and the second decoder includes an encoder, a segmentation decoder and a cutout decoder; The step of performing a second decoding based on the fusion feature using a second decoder to obtain a foreground segmentation result of the target image includes: Encoding the fused features using the encoder to obtain encoded features; Performing a third decoding on the encoded features using the segmentation decoder to obtain a ternary segmentation map; Using the cutout decoder to perform a fourth decoding based on the encoding feature and the ternary segmentation map to obtain an initial cutout image; A preset calculation is performed on the pixel values ​​of corresponding pixels in the ternary segmentation image and the initial cutout image to obtain the target cutout image of the target image.

7. The method according to claim 6, characterized in that The encoder includes a plurality of encoding layers of different resolutions; The using the encoder to encode the fused features to obtain the encoded features includes: The first input data is downsampled by each coding layer to obtain the coding features output by each coding layer; wherein the multiple coding layers are arranged in descending order of resolution, the first input data of the coding layer at the first position is the fusion feature, and the first input data of the coding layer not at the first position includes the fusion feature and the coding features output by the previous coding layer.

8. The method according to claim 7, characterized in that The segmentation decoder comprises a plurality of first decoding layers of different resolutions, and the cutout decoder comprises a plurality of second decoding layers of different resolutions; The using the segmentation decoder to perform a third decoding on the encoded feature to obtain a corresponding ternary segmentation map includes: Using each of the first decoding layers to perform upsampling processing on the second input data, respectively, to obtain a ternary segmentation map output by each of the first decoding layers; wherein the multiple first decoding layers are arranged in ascending order of resolution, the second input data of the first decoding layer at the first position is the coding feature output by the last coding layer, and the second input data of the first decoding layer at a non-first position is the ternary segmentation map output by the previous first decoding layer; The step of performing a fourth decoding based on the encoding feature and the ternary segmentation map using the cutout decoder to obtain an initial cutout image includes: Using each of the second decoding layers to perform upsampling processing on the input third input data, respectively, to obtain a cutout image output by each of the second decoding layers; wherein the multiple second decoding layers are arranged in order of resolution from small to large, the third input data of the second decoding layer at the first position is the coding feature output by the last coding layer, and each of the second decoding layers not at the first position is sequentially used as a target decoding layer, and the third input data of the target decoding layer includes: the cutout image output by the previous second decoding layer, the coding feature output by the coding layer with the same resolution as the target decoding layer, and the second input data of the first decoding layer with the same resolution as the target decoding layer; The cutout image output by the last second decoding layer is used as the initial cutout image.

9. The method according to any one of claims 6 to 8, characterized in that: The step of performing a preset calculation on the pixel values ​​of corresponding pixels in the ternary segmentation map and the initial cutout image to obtain the target cutout image of the target image comprises: Using the ternary segmentation map output by the last first decoding layer in the segmentation decoder as the target ternary map; Performing expansion and erosion operations on the target ternary image respectively to obtain a corresponding expansion result image and an erosion result image; Obtaining the product of the pixel values ​​of the corresponding pixel points in the initial cutout image and the dilation result image as the pixel value of the corresponding pixel point in the intermediate cutout image; The sum of the pixel values ​​corresponding to the corresponding pixel points in the intermediate cutout image and the erosion result image is obtained as the target pixel value of the corresponding pixel point in the target cutout image; wherein the target pixel value of the first pixel point belonging to the foreground in the target cutout image is the first pixel value, and the target pixel value of the second pixel point belonging to the background is the second pixel value.

10. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire image association information of a target image to be cut out and a target background; the target background is different from the background in the target image; A fusion module, used for fusing the image features of the target image and the image association features of the image association information to obtain an image fusion feature; A decoding module is used to perform decoding based on the image fusion feature to obtain a corresponding target decoded image, wherein the target decoded image includes a target cutout image of the target image and a fused image of a foreground in the target image and the target background.

11. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 9.

Images