Image processing method and device

Abstract

The present disclosure relates to an image processing method and device. The method comprises: obtaining an image to be processed, a first parsing mask and a second parsing mask, wherein the image to be processed contains a target object, a part of the target object is blocked by an occluder, the first parsing mask is a parsing mask of the target object's area not blocked by the occluder, and the second parsing mask is a parsing mask of the target object's complete area; the image to be processed, the first parsing mask and the second parsing mask are input into a neural network model to complete the content of the area of ​​the target object blocked by the occluder. Thus, the content of the area to be completed is accurately inferred, which is conducive to achieving tasks such as target tracking, target detection, and image segmentation.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] In the process of image acquisition, affected by factors such as lighting, equipment, and algorithms, images are easily damaged or interfered by noise, resulting in the image being unable to correctly express the content. Therefore, image inpainting has become one of the important steps in computer vision task preprocessing, and is directly related to the results of tasks such as target tracking, target detection, and image segmentation.

[0003] At present, neural network models are often used to complete images in the image restoration process. Neural network models usually infer the content that needs to be completed based on the characteristics of the surrounding area. For example, if the left leg is blocked by a human body, the inference is made based on the appearance characteristics of the right leg. However, neural network models cannot accurately infer the content that needs to be completed in the image, which is not conducive to achieving tasks such as target tracking, target detection, and image segmentation. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present disclosure provides an image processing method and device.

[0005] In a first aspect, the present disclosure provides an image processing method, comprising:

[0006] Acquire an image to be processed, a first parsing mask, and a second parsing mask, wherein the image to be processed includes a target object, a part of the target object is blocked by an occluder, the first parsing mask is a parsing mask of a region of the target object that is not blocked by the occluder, and the second parsing mask is a parsing mask of a complete region of the target object;

[0007] The image to be processed, the first parsing mask and the second parsing mask are input into the neural network model, and the content of the area blocked by the occluder of the target object is completed based on the attention mechanism.

[0008] Through the method provided in the first aspect, the analytical mask of the area of ​​the target object not obscured by the occluder and the analytical mask of the complete area of ​​the target object are introduced into the attention mechanism, so that the attention mechanism can focus on the target object, so as to accurately infer the content of the area obscured by the occluder of the target object based on the characteristics of the area of ​​the target object not obscured by the occluder in the image to be processed and the characteristics of the predicted complete area of ​​the target object, thereby realizing the completion of the content of the area obscured by the occluder of the target object in the image to be processed, and being able to accurately output the image to be processed after the area obscured by the occluder of the target object is completed, which is conducive to the reliable completion of tasks such as target tracking, target detection, and image segmentation.

[0009] In a possible design, the image to be processed, the first parsing mask and the second parsing mask are input into a neural network model, and the content of the area blocked by the occluder of the target object is completed based on the attention mechanism, including: processing the image to be processed through multiple network layers in the neural network model, and completing the content of the area blocked by the occluder of the target object based on the attention mechanism; wherein the processing process of at least one network layer among the multiple network layers includes: receiving a first feature map output by a previous network layer, wherein the first feature map is obtained based on the image to be processed, the first parsing mask and the second parsing mask; obtaining a second feature map according to the first feature map, the first parsing mask and the second parsing mask, wherein the second feature map is a feature after the content of the area blocked by the occluder of the target object is enhanced, the sizes of the first feature map, the first parsing mask and the second parsing mask are the same, and the number of channels of the first parsing mask and the second parsing mask are the same; performing convolution processing on the second feature map, and transmitting the feature map after the convolution processing to the next network layer.

[0010] In one possible design, the multiple network layers include a first network layer and a second network layer, the first feature map received by the first network layer has a different size from the first feature map received by the second network layer, the first parsing mask in the processing process of the first network layer has a different size from the first parsing mask in the processing process of the second network layer, and the second parsing mask in the processing process of the first network layer has a different size from the second parsing mask in the processing process of the second network layer.

[0011] In one possible design, a second feature map is obtained according to a first feature map, a first parsing mask, and a second parsing mask, including: obtaining a first attention result according to the first feature map, the first parsing mask, and the second parsing mask, the first attention result being used to represent the association relationship between each spatial position point of the target object in the feature map corresponding to the image to be processed and all spatial position points, and / or obtaining a second attention result according to the first feature map, the first parsing mask, and the second parsing mask, the second attention result being used to represent the semantic information and depth features of a sub-component area in an area obscured by an occluded object of the target object corresponding to the difference between the first parsing mask and the second parsing mask; obtaining a second feature map according to the first attention result and / or the second attention result.

[0012] In one possible design, a first attention result is obtained according to the first feature map, the first parsing mask, and the second parsing mask, including: obtaining an associated feature map according to the first feature map, the first parsing mask, and the second parsing mask, and using the associated feature map as the first attention result, the associated feature map being used to represent the association relationship between the area of ​​the target object that is not obscured by the occluder and the complete area of ​​the target object; stacking the associated feature map and the first feature map to obtain a first stacking result; and performing dimensionality reduction processing on the first stacking result to obtain a second feature map.

[0013] In a possible design, a first attention result and a second attention result are obtained according to a first feature map, a first parsing mask and a second parsing mask, including: obtaining an associated feature map according to the first feature map, the first parsing mask and the second parsing mask, and using the associated feature map as the first attention result, the associated feature map being used to represent the association relationship between the area of ​​the target object that is not blocked by the occluder and the complete area of ​​the target object; performing channel number dimensionality reduction processing on the first feature map to obtain a reduced dimensionality feature map, the number of channels of the reduced dimensionality feature map being the same as the number of channels of the first parsing mask or the second parsing mask; performing dot multiplication processing on the reduced dimensionality feature map and the first parsing mask to obtain a first dot multiplication result, the first dot multiplication result being used to represent the target object. The semantic information and depth features of the sub-component area corresponding to the first parsing mask in the area of ​​the target object that is not blocked by the occluder are obtained; the reduced dimension feature map and the second parsing mask are subjected to dot multiplication processing to obtain a second dot product result, and the second dot product result is used to represent the semantic information and depth features of the sub-component area corresponding to the second parsing mask in the complete area of ​​the target object; the first dot product result and the second dot product result are stacked and convolved to obtain a second attention result; according to the first attention result and the second attention result, a second feature map is obtained, including: stacking the second attention result, the associated feature map and the first feature map to obtain a second stacking result; and reducing the dimension of the second stacking result to obtain a second feature map.

[0014] In one possible design, an associated feature map is obtained according to the first feature map, the first parsing mask, and the second parsing mask, including: stacking the first feature map and the first parsing mask to obtain a third stacking result; stacking the first feature map and the second parsing mask to obtain a fourth stacking result; and performing attention mechanism processing on the third stacking result and the fourth stacking result to obtain an associated feature map.

[0015] In one possible design, a second attention result is obtained according to the first feature map, the first parsing mask and the second parsing mask, including: performing channel number dimensionality reduction processing on the first feature map to obtain a reduced dimensionality feature map, the number of channels of the reduced dimensionality feature map is the same as the number of channels of the first parsing mask or the second parsing mask; performing dot multiplication processing on the reduced dimensionality feature map and the first parsing mask to obtain a first dot multiplication result, the first dot multiplication result is used to represent the semantic information and depth features of the sub-component area corresponding to the first parsing mask in the area of ​​the target object not obstructed by the occluder; performing dot multiplication processing on the reduced dimensionality feature map and the second parsing mask to obtain a second dot multiplication result, the second dot multiplication result is used to represent the semantic information and depth features of the sub-component area corresponding to the second parsing mask in the complete area of ​​the target object; stacking and convolving the first dot multiplication result and the second dot multiplication result to obtain a second attention result; obtaining a second feature map according to the second attention result, including: stacking the second attention result and the first feature map to obtain a fifth stacking result; performing dimensionality reduction processing on the fifth stacking result to obtain a second feature map.

[0016] In one possible design, the first parsing mask and the second parsing mask are in the form of one-hot encoding.

[0017] In a second aspect, the present disclosure provides an image processing device, comprising:

[0018] an acquisition module, configured to acquire an image to be processed, a first parsing mask, and a second parsing mask, wherein the image to be processed includes a target object, a part of the target object is blocked by an occluder, the first parsing mask is a parsing mask of a region of the target object not blocked by the occluder, and the second parsing mask is a parsing mask of a complete region of the target object;

[0019] The processing module is used to input the image to be processed, the first parsing mask and the second parsing mask into the neural network model, and complete the content of the area of ​​the target object blocked by the occluder based on the attention mechanism.

[0020] In one possible design, a processing module is specifically used to process an image to be processed through multiple network layers in a neural network model, and complete the content of an area of ​​a target object obscured by an occluder based on an attention mechanism; wherein the processing process of at least one of the multiple network layers includes: receiving a first feature map output by a previous network layer, wherein the first feature map is obtained based on the image to be processed, a first parsing mask, and a second parsing mask; obtaining a second feature map according to the first feature map, the first parsing mask, and the second parsing mask, wherein the second feature map is a feature after the content of the area of ​​the target object obscured by the occluder is enhanced, the sizes of the first feature map, the first parsing mask, and the second parsing mask are the same, and the number of channels of the first parsing mask and the second parsing mask are the same; and performing convolution processing on the second feature map.

[0021] In one possible design, the multiple network layers include a first network layer and a second network layer, the first feature map received by the first network layer has a different size from the first feature map received by the second network layer, the first parsing mask in the processing process of the first network layer has a different size from the first parsing mask in the processing process of the second network layer, and the second parsing mask in the processing process of the first network layer has a different size from the second parsing mask in the processing process of the second network layer.

[0022] In one possible design, a processing module is used to obtain a first attention result based on a first feature map, a first parsing mask, and a second parsing mask. The first attention result is used to represent the association relationship between each spatial position point of the target object in the feature map corresponding to the image to be processed and all the spatial position points, and / or obtain a second attention result based on the first feature map, the first parsing mask, and the second parsing mask. The second attention result is used to represent the semantic information and depth features of a sub-component area in an area obscured by an occluded object of the target object corresponding to the difference between the first parsing mask and the second parsing mask; and obtain a second feature map based on the first attention result and / or the second attention result.

[0023] In a possible design, a processing module is used to obtain an associated feature map based on a first feature map, a first parsing mask, and a second parsing mask, and use the associated feature map as a first attention result, wherein the associated feature map is used to represent the association relationship between the area of ​​the target object that is not obscured by the occluder and the complete area of ​​the target object; stacking the associated feature map and the first feature map to obtain a first stacking result; and performing dimensionality reduction processing on the first stacking result to obtain a second feature map.

[0024] In a possible design, the processing module is used to obtain a correlation feature map according to the first feature map, the first parsing mask and the second parsing mask, and use the correlation feature map as the first attention result, the correlation feature map is used to represent the correlation relationship between the area of ​​the target object that is not blocked by the occluder and the complete area of ​​the target object; perform channel number dimensionality reduction processing on the first feature map to obtain a reduced dimensionality feature map, the number of channels of the reduced dimensionality feature map is the same as the number of channels of the first parsing mask or the second parsing mask; perform dot multiplication processing on the reduced dimensionality feature map and the first parsing mask to obtain a first dot multiplication result, and the first dot multiplication result is used to represent the target object The semantic information and depth features of the sub-component area corresponding to the first parsing mask in the area not blocked by the occluder; perform dot multiplication on the dimensionality reduction feature map and the second parsing mask to obtain a second dot product result, and the second dot product result is used to represent the semantic information and depth features of the sub-component area corresponding to the second parsing mask in the complete area of ​​the target object; stack and convolve the first dot product result and the second dot product result to obtain a second attention result; stack the second attention result, the associated feature map and the first feature map to obtain a second stacking result; perform dimensionality reduction on the second stacking result to obtain a second feature map.

[0025] In one possible design, the processing module is used to stack the first feature map and the first parsing mask to obtain a third stacking result; stack the first feature map and the second parsing mask to obtain a fourth stacking result; and perform attention mechanism processing on the third stacking result and the fourth stacking result to obtain an associated feature map.

[0026] In one possible design, a processing module is used to perform channel number dimensionality reduction processing on the first feature map to obtain a reduced dimensionality feature map, wherein the number of channels of the reduced dimensionality feature map is the same as the number of channels of the first parsing mask or the second parsing mask; perform dot multiplication processing on the reduced dimensionality feature map and the first parsing mask to obtain a first dot multiplication result, and the first dot multiplication result is used to represent the semantic information and depth features of the sub-component area corresponding to the first parsing mask in the area of ​​the target object not obscured by the occluder; perform dot multiplication processing on the reduced dimensionality feature map and the second parsing mask to obtain a second dot multiplication result, and the second dot multiplication result is used to represent the semantic information and depth features of the sub-component area corresponding to the second parsing mask in the complete area of ​​the target object; stack and convolve the first dot multiplication result and the second dot multiplication result to obtain a second attention result; stack the second attention result and the first feature map to obtain a fifth stacking result; perform dimensionality reduction processing on the fifth stacking result to obtain a second feature map.

[0027] In one possible design, the first parsing mask and the second parsing mask are in the form of one-hot encoding.

[0028] The beneficial effects of the image processing device provided in the above-mentioned second aspect and each possible design of the above-mentioned second aspect can be referred to the beneficial effects brought about by the above-mentioned first aspect and each possible implementation method of the first aspect, and will not be repeated here.

[0029] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions in the memory so that the electronic device executes the image processing method in the first aspect and any possible design of the first aspect.

[0030] In a fourth aspect, the present disclosure provides a computer storage medium, including computer instructions, which, when executed on an electronic device, enables the electronic device to execute the image processing method in the first aspect and any possible design of the first aspect.

[0031] In a fifth aspect, the present disclosure provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the image processing method in the first aspect and any possible design of the first aspect.

[0032] In a sixth aspect, the present disclosure provides a chip system, the chip system comprising: a processor; when the processor executes computer instructions stored in a memory, the electronic device executes the image processing method in the first aspect and any possible design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0035] Figure 1 A schematic diagram of the structures of 19 parts of a human body provided in an embodiment of the present disclosure;

[0036] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0037] Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present disclosure;

[0038] Figure 4A A working principle diagram of the attention mechanism module provided in the embodiment of the present disclosure;

[0039] Figure 4B A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0040] Figure 5A A working principle diagram of the attention mechanism module provided in the embodiment of the present disclosure;

[0041] Figure 5B A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0042] Fig. 6A A working principle diagram of the attention mechanism module provided in the embodiment of the present disclosure;

[0043] Figure 6B A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0044] Figure 7 A schematic diagram of the structure of an image processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0047] Exemplarily, the present disclosure provides an image processing method, apparatus, device, computer storage medium and computer program product, which introduce a parsing mask of the target object, and an auxiliary attention mechanism guides the neural network model to capture the correct attention and focus on the to-be-completed area of ​​the target object, so as to accurately infer the content of the to-be-completed area of ​​the target object, which is conducive to achieving tasks such as target tracking, target detection, and image segmentation.

[0048] The image processing method of the present disclosure is performed by an electronic device. The electronic device may be a tablet computer, a mobile phone (such as a folding screen mobile phone, a large screen mobile phone, etc.), a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, a high-definition TV, a 4K TV, a smart speaker, a smart projector, and other Internet of Things (IOT) devices. The present disclosure does not impose any restrictions on the specific types of electronic devices.

[0049] The parsing mask of the target object can be understood as the code formed by different numbers corresponding to each sub-component region after the sub-component regions of the target object are segmented. For example, when the target object is a human body, the human body is segmented into N sub-component regions, and the segmented N sub-component regions are subjected to human parsing, that is, the N sub-component regions are marked with numbers 0-N respectively to obtain the parsing mask of the human body, where N is a positive integer.

[0050] When N = 19, Figure 1 As shown, the 19 sub-regions of the human body may include: head a1, neck a2, left shoulder a3, right shoulder a4, left upper arm a5, right upper arm a6, left lower arm a7, right lower arm a8, left hand a9, right hand a10, left hip a11, right hip a12, left thigh a13, right thigh a14, left calf a15, right calf a16, left foot a17, right foot a18 and body a19, and the parsing mask of the human body is a code formed by each of the 19 sub-regions of the human body corresponding to a number. In addition, the present disclosure is not limited to dividing the human body into 19 sub-regions.

[0051] Based on the foregoing description, the present disclosure will use an electronic device as an example in an embodiment to elaborate on the image processing method provided by the present disclosure in detail in combination with the accompanying drawings and application scenarios.

[0052] See also Figure 2 , Figure 2 Schematic diagram of the process flow of the image processing method provided by the embodiment of the present disclosure. Figure 2 As shown, the image processing method provided by the present disclosure may include:

[0053] S101, obtaining an image to be processed, a first parsing mask, and a second parsing mask.

[0054] The electronic device may obtain an image to be processed, wherein the image to be processed includes a target object, and a portion of the target object is blocked by an obstruction. The present disclosure does not limit parameters such as the size, format, and content of the image to be processed.

[0055] It should be noted that, in addition to the image to be processed, the electronic device can also obtain a complete input including the image to be processed, a first mask and a second mask, wherein the first mask is a mask of the area of ​​the target object that is not blocked by the blocking object, and the second mask is a mask of the complete area of ​​the target object.

[0056] Based on the foregoing description, the present disclosure divides the complete area of ​​the target object into an area of ​​the target object that is not blocked by the blocker and an area of ​​the target object that is blocked by the blocker.

[0057] The target object may include but is not limited to a human body, an animal or an object, etc. The present disclosure does not limit the size, shape, position and other parameters of the target object's area not blocked by the blocker and the target object's area blocked by the blocker.

[0058] The electronic device performs parsing processing on the sub-component area contained in the area of ​​the target object in the image to be processed that is not blocked by the occluder to obtain a first parsing mask, that is, the first parsing mask is a parsing mask of the area of ​​the target object that is not blocked by the occluder.

[0059] Correspondingly, the electronic device performs parsing processing on the sub-component regions included in the complete region of the target object in the image to be processed to obtain a second parsing mask, that is, the second parsing mask is a parsing mask of the complete region of the target object.

[0060] The complete region of the target object in the image to be processed is predicted. In addition, the sub-component regions included in the region of the target object not blocked by the blocking object and the sub-component regions included in the complete region of the target object adopt the same segmentation rule.

[0061] S102: input the image to be processed, the first parsing mask and the second parsing mask into a neural network model, and complete the content of the area of ​​the target object blocked by the occluder based on the attention mechanism.

[0062] The electronic device can adopt a neural network model, based on the image to be processed, the first parsing mask and the second parsing mask, and complete the content of the area of ​​the target object obscured by the occluder by focusing the attention mechanism of the neural network model on the target object, so as to output the image to be processed after the area of ​​the target object obscured by the occluder is completed.

[0063] The image processing method provided by the present invention introduces the parsing mask of the area of ​​the target object not obscured by the occluder and the parsing mask of the complete area of ​​the target object into the attention mechanism, so that the attention mechanism can focus on the target object, so as to accurately infer the content of the area of ​​the target object obscured by the occluder based on the characteristics of the area of ​​the target object not obscured by the occluder in the image to be processed and the characteristics of the predicted complete area of ​​the target object, thereby realizing the completion of the content of the area of ​​the target object obscured by the occluder in the image to be processed, and can accurately output the image to be processed after the area of ​​the target object obscured by the occluder is completed, which is conducive to the reliable completion of tasks such as target tracking, target detection, and image segmentation.

[0064] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of the neural network model provided in the embodiment of the present disclosure. Figure 3 As shown, the neural network model 10 provided by the present disclosure may include: an input layer 11, an intermediate layer 12, an output layer 13 and an attention mechanism module 14. The attention mechanism module 14 is arranged between the input layer 11 and the intermediate layer 12, and / or, the attention mechanism module 14 is arranged between the intermediate layer 12 and the output layer 13, and / or, the attention mechanism module 14 is arranged between multiple intermediate layers 12.

[0065] Furthermore, the present disclosure does not limit the number of intermediate layers 12 and attention mechanism modules 14. In some embodiments, the attention mechanism modules 14 are symmetrically arranged in the neural network model 10.

[0066] Among them, the input layer 11 is mainly used to receive the image to be processed ( Figure 3 The image to be processed is illustrated by the letter A1), and the feature map obtained after convolution and other processing is output to the intermediate layer 12 or the attention mechanism module 14.

[0067] Among them, the intermediate layer 12 is mainly used to receive feature maps from the input layer 11 or the attention mechanism module 14 or the intermediate layer 12, and extract features, convolution, dimensionality reduction, dimensionality increase and other processing on the feature maps to obtain feature maps and output them to the output layer 13 or the attention mechanism module 14 or the intermediate layer 12.

[0068] Among them, the output layer 13 is mainly used to receive the feature map from the intermediate layer 12 or the attention mechanism module 14, and perform convolution and other processing on the feature map to output the image to be processed after filling the area of ​​the target object blocked by the occluder ( Figure 3 The letter A2 is used to indicate this).

[0069] Among them, the attention mechanism module 14 is mainly used to receive the first parsing mask ( Figure 3) and the second parsing mask ( Figure 3 The feature map is received from the input layer 11 or the middle layer 12, and the feature map obtained after the attention mechanism is calculated is output to the middle layer 12 or the output layer 13.

[0070] Exemplarily, when the neural network model 10 includes multiple attention mechanism modules 14, such as a first attention mechanism module and a second attention mechanism module, it is assumed that the first parsing mask received by the first attention mechanism module is B1, and the second parsing mask is C1. The first parsing mask received by the second attention mechanism module is B2, and the second parsing mask is C2. Then, the sizes of B1 and C1 remain the same, the sizes of B2 and C2 remain the same, and the size of B1 is different from the size of B2, and the size of C1 is different from the size of C2. Thus. The attention mechanism of the parsing mask can be integrated into the layers of different stages in the neural network model 10 at different scales.

[0071] Based on the above description, the neural network model 10 may include: multiple network layers. One of the network layers may be an input layer 11, or one or more intermediate layers 12, or an attention mechanism module 14 and a layer connected to the output end side of the attention mechanism module 14 (i.e., the side close to the output layer 13). The layer connected to the output end side of the attention mechanism module 14 may be an intermediate layer 12 or an output layer 13. In addition, the neural network model 10 includes at least one network layer including an attention mechanism module 14, and the network layer includes an attention mechanism module 14 and a layer connected to the output end side of the attention mechanism module 14.

[0072] Thus, the electronic device can process the image to be processed through multiple network layers, continuously enhancing the features corresponding to the content of the area of ​​the target object blocked by the occluder, so as to complete the content of the area of ​​the target object blocked by the occluder.

[0073] For a network layer 1 among multiple network layers, network layer 1 is a network layer including an attention mechanism module 14, and network layer 1 is arranged between network layer 0 and network layer 2, network layer 0 is the previous network layer of network layer 1, and network layer 2 is the next network layer of network layer 1.

[0074] Network layer 1 can receive the first feature map output by network layer 0, wherein the first feature map is obtained based on the image to be processed, the first parsing mask and the second parsing mask. Network layer 1 can obtain the second feature map based on the first feature map, the first parsing mask and the second parsing mask, and then perform convolution processing on the second feature map, and transmit the feature map after convolution processing to network layer 2 (i.e., the first feature map output by network layer 1 to network layer 2).

[0075] The second feature map is a feature of the target object after content enhancement of the area blocked by the occluder, so as to accurately infer the characteristics of the area blocked by the occluder. The first feature map, the first parsing mask and the second parsing mask have the same size, and the first parsing mask and the second parsing mask have the same number of channels. Exemplarily, the first parsing mask and the second parsing mask are in the form of one-hot encoding.

[0076] Therefore, through the above process, network layer 1 can continuously enhance the features corresponding to the content of the area of ​​the target object obscured by the occluder based on the output second feature map, so as to accurately infer the content of the area of ​​the target object obscured by the occluder based on the feature map.

[0077] In some embodiments, the neural network model 10 may include a first network layer and a second network layer capable of performing the above process. When the size of the first feature map in the processing of the first network layer is different from the size of the first feature map in the processing of the second network layer, the first parsing mask and the second parsing mask are scaled to the same size as the first feature map received by the network layer, and then input into the attention mechanism module 14.

[0078] For example, assume that both network layer 1 and network layer 2 can perform the above process, and the size of the first feature map received by network layer 1 from network layer 0 is larger than the size of the first feature map received by network layer 2 from network layer 1. Since the first parsing mask, the second parsing mask, and the first feature map for performing the above process in network layer 1 remain the same, and the sizes of the first parsing mask, the second parsing mask, and the first feature map for performing the above process in network layer 2 remain responsive. Therefore, the size of the first parsing mask for performing the above process in network layer 1 is larger than the size of the first parsing mask for performing the above process in network layer 2, and the size of the second parsing mask for performing the above process in network layer 1 is larger than the size of the second parsing mask for performing the above process in network layer 2.

[0079] It should be noted that, in addition to the above-mentioned method of dividing one of the network layers, the present disclosure may also adopt the following division method: one of the network layers may be the attention mechanism module 14 and the layer connected to the input end side of the attention mechanism module 14 (i.e., the side close to the input layer 11), or may be one or more intermediate layers 12, or may be the input layer 11. The layer connected to the input end side of the attention mechanism module 14 may be the intermediate layer 12 or the input layer 11. Furthermore, the neural network model 10 includes at least one network layer, which includes the attention mechanism module 14 and the layer connected to the input end side of the attention mechanism module 14.

[0080] For a network layer 1 among multiple network layers, network layer 1 is a network layer including an attention mechanism module 14, and network layer 1 is arranged between network layer 0 and network layer 2, network layer 0 is the previous network layer of network layer 1, and network layer 2 is the next network layer of network layer 1.

[0081] Network layer 1 can receive the second feature map output by network layer 0, wherein the second feature map is obtained based on the image to be processed, the first parsing mask, and the second parsing mask. Network layer 1 performs convolution processing on the second feature map to obtain the first feature map, and obtains the second feature map based on the first feature map, the first parsing mask, and the second parsing mask, and then transmits the second feature map to network layer 2 (i.e., the second feature map output by network layer 1 to network layer 2).

[0082] In addition, the attention mechanism module 14 can also be set separately without being bound to other layers, that is, at least one attention mechanism module 14 is set between any two layers.

[0083] Based on the foregoing description, the electronic device may utilize the attention mechanism module 14 in at least one network layer in a variety of implementation methods to obtain a second feature map according to the first feature map, the first parsing mask, and the second parsing mask.

[0084] The electronic device may utilize the attention mechanism module 14 to obtain a first attention result and / or a second attention result according to the first feature map, the first parsing mask, and the second parsing mask.

[0085] The first attention result is used to represent the association between each spatial position point of the target object in the feature map corresponding to the image to be processed and all the spatial position points. The spatial position point here is a point in the feature map (H×W×C, H represents length, W represents width, and C represents the number of channels), the coordinates of the spatial position point are represented as (i, j), and the size of the spatial position point is 1×1×C.

[0086] In the actual calculation process, the attention mechanism module 14 can calculate the correlation between each spatial position point and all spatial position points in the feature map corresponding to the image to be processed, including the target object and the background area. In addition, the attention mechanism module 14 can also focus on the target object based on the first parsing mask and the second parsing mask to obtain a first attention result.

[0087] Among them, the second attention result is used to represent the semantic information and depth features of the sub-component area corresponding to the difference between the first parsing mask and the second parsing mask in the area of ​​the target object occluded by the occluder.

[0088] The semantic information here can be understood as the specific location of the sub-component region, such as the head or the left foot of the human body, etc. The depth feature here can be understood as the content of the sub-component region after being filled in the image to be processed, such as the content of the left foot after being filled in the image to be processed.

[0089] The attention mechanism module 14 can compare the semantic information and depth features of the sub-regions contained in the region of the target object occluded by the occluder based on the difference between the first parsing mask and the second parsing mask, so as to obtain a second attention result.

[0090] Thus, the electronic device can utilize the attention mechanism module 14 to obtain a second feature map according to the first attention result and / or the second attention result.

[0091] Next, combine Figure 4A-4B , Figure 5A-Figure 5B and Figure 6A-6B , introduces three feasible implementations in which the electronic device can use the attention mechanism module 14 to obtain the second feature map. For ease of explanation, Figure 4A-4B , Figure 5A-Figure 5B and Figure 6A-6B In the example, the first feature map uses F i in Take the example to illustrate that the first parsing mask adopts M m p As an example, the second parsing mask uses M a p As an example, the second feature map uses F i out As an example, the channel number dimensionality reduction processing takes 1×1 as an example, the convolution processing takes 3×3 as an example, the point product processing takes a point in a circle as an example, and the stacking processing takes a cross in a circle as an example.

[0092] See also Figure 4A and Figure 4B , Figure 4A A working principle diagram of the attention mechanism module provided in the embodiment of the present disclosure, Figure 4B Schematic diagram of the process flow of the image processing method provided by the embodiment of the present disclosure. Figure 4A-4B As shown, the electronic device of the present disclosure can use the attention mechanism module 14 to perform the following steps:

[0093] S201. Obtain an associated feature map according to the first feature map, the first parsing mask and the second parsing mask, and use the associated feature map as the first attention result.

[0094] The correlation feature map corresponds to the first attention result, that is, the correlation feature map and the first attention result have the same meaning, and the correlation feature map is used as the first attention result. The correlation feature map can be used to represent the correlation relationship between the area of ​​the target object that is not blocked by the occluder and the complete area of ​​the target object. Thus, the specific location and content of the area of ​​the target object blocked by the occluder can be inferred.

[0095] S202: Stack the associated feature map and the first feature map to obtain a first stacking result, thereby determining the missing partial area of ​​the target object, so as to infer features associated with the content of the area of ​​the target object blocked by the occluder in the first feature map.

[0096] S203, performing channel number dimensionality reduction processing on the first stacking result to obtain a second feature map. Thus, the first feature map and the second feature map can maintain the same spatial dimension, avoiding the influence of the attention mechanism model on the current operation, so that the attention mechanism module 14 can be seamlessly integrated into the layers of the neural network model 10 at different stages, without the need for the current network layer to adjust the spatial dimension of the second feature map, simplifying the process of the current network layer outputting the first feature map to the next network layer.

[0097] Based on the above description, the electronic device can use the attention mechanism module 14 to introduce a spatial position point attention mechanism based on the parsing mask to obtain a second feature map for representing the content enhanced features of the area of ​​the target object occluded by the occluder.

[0098] See also Figure 5A and Figure 5B , Figure 5A A working principle diagram of the attention mechanism module provided in the embodiment of the present disclosure, Figure 5B Schematic diagram of the process flow of the image processing method provided by the embodiment of the present disclosure. Figure 5A-Figure 5B As shown, the electronic device of the present disclosure can use the attention mechanism module 14 to perform the following steps:

[0099] S301. Obtain an associated feature map according to the first feature map, the first parsing mask and the second parsing mask, and use the associated feature map as the first attention result.

[0100] The specific implementation of step S301 can be found in Figure 4B The description of step S201 in is not repeated here.

[0101] S302, performing channel number dimensionality reduction processing on the first feature map to obtain a reduced-dimensionality feature map, so that the number of channels of the reduced-dimensionality feature map remains the same as the number of channels of the first parsing mask or the second parsing mask, thereby reducing the processing steps of the attention mechanism module 14.

[0102] For example, assuming that the number of channels of the first parsing mask or the second parsing mask is 19, since the number of channels of the reduced-dimensionality feature map is usually greater than 19, the attention mechanism module 14 executes step S302 to reduce the number of channels of the reduced-dimensionality feature map to 19.

[0103] S303: Perform a dot product process on the dimension reduction feature map and the first parsing mask to obtain a first dot product result.

[0104] The first dot product result corresponds to the second attention result. The first dot product result is used to represent the semantic information and depth features of the sub-component region corresponding to the first parsing mask in the area of ​​the target object that is not blocked by the occluder. The attention mechanism module 14 can determine the semantic information and depth features of the sub-component region contained in the area of ​​the target object that is not blocked by the occluder, that is, the first dot product result, based on the first parsing mask.

[0105] S304: Perform a dot product process on the dimension reduction feature map and the second parsing mask to obtain a second dot product result.

[0106] The second dot product result corresponds to the second attention result. The second dot product result is used to represent the semantic information and depth features of the sub-component region corresponding to the second parsing mask in the complete region of the target object. The attention mechanism module 14 can determine the semantic information and depth features of the sub-component region contained in the complete region of the target object based on the second parsing mask, that is, the second dot product result.

[0107] In addition, the present disclosure does not distinguish between the execution order of steps S303 and S304. Step S303 may be executed first and then step S304, or step S304 may be executed first and then step S303, or steps S303 and S304 may be executed simultaneously.

[0108] S305. Stack and convolve the first dot product result and the second dot product result to obtain a second attention result.

[0109] S306: stack the second attention result, the associated feature map and the first feature map to obtain a second stacking result.

[0110] Thus, the missing partial area of ​​the target object is determined, that is, the area of ​​the target object blocked by the occluder is separated, so as to infer the features associated with the content of the area of ​​the target object blocked by the occluder in the first feature map.

[0111] The first feature map can increase the redundancy of the above process, making the second stacking result more accurate. In addition, the first feature map can also be removed in step S306, and the second attention result and the associated feature map are stacked to obtain the second stacking result.

[0112] S307, performing channel number dimensionality reduction processing on the second stacking result to obtain a second feature map. Thus, the first feature map and the second feature map can maintain the same spatial dimension, avoiding the influence of the attention mechanism model on the operation of the current network, thereby eliminating the need for the current network layer to adjust the spatial dimension of the second feature map, and simplifying the process of the current network layer outputting the first feature map to the next network layer.

[0113] For example, the number of channels of the first feature map is C, and the number of channels of the second stacking result is 2C+38. Therefore, the attention mechanism module 14 executes step S307 to reduce the number of channels of the second stacking result to C.

[0114] Based on the above description, the electronic device can use the attention mechanism module 14 to introduce a spatial position point attention mechanism based on the parsing mask and a channel number attention mechanism based on the parsing mask to obtain a second feature map for representing the content enhanced features of the area obscured by the occluder of the target object.

[0115] exist Figure 4B Step S201 in or Figure 5B In step S301, the electronic device can use the attention mechanism module 14 in a variety of implementation methods to obtain an associated feature map based on the first feature map, the first parsing mask and the second parsing mask.

[0116] In some embodiments, the electronic device may utilize the attention mechanism module 14 to stack the first feature map and the first parsing mask to obtain a third stacking result, so as to infer features associated with the content of the area of ​​the target object in the first feature map that is not obscured by the occluder.

[0117] Correspondingly, the electronic device can utilize the attention mechanism module 14 to stack the first feature map and the second parsing mask to obtain a fourth stacking result, so as to infer features associated with the content of the complete area of ​​the target object in the first feature map.

[0118] Therefore, the electronic device can use the attention mechanism module 14 to perform attention mechanism processing on the third stacking result and the fourth stacking result to obtain an associated feature map, so as to infer features associated with the content of the area of ​​the target object obscured by the occluder in the first feature map.

[0119] See also Fig. 6A and Figure 6B , Fig. 6A A working principle diagram of the attention mechanism module provided in the embodiment of the present disclosure, Figure 6B Schematic diagram of the process flow of the image processing method provided by the embodiment of the present disclosure. Figure 6A-6B As shown, the electronic device of the present disclosure can use the attention mechanism module 14 to perform the following steps:

[0120] S401, performing channel number dimensionality reduction processing on the first feature map to obtain a reduced-dimensionality feature map, so that the number of channels of the reduced-dimensionality feature map remains the same as the number of channels of the first parsing mask or the second parsing mask, thereby reducing the processing steps of the attention mechanism module 14.

[0121] S402, perform dot multiplication on the dimension reduction feature map and the first analytical mask to obtain a first dot multiplication result. The specific implementation of step S402 can be found in Figure 5B The description of step S303 in is not repeated here. S403, perform dot multiplication on the dimension reduction feature map and the second parsing mask to obtain a second dot multiplication result. The specific implementation of step S403 can be found in Figure 5B The description of step S304 in will not be repeated here.

[0122] S404: Stack and convolve the first dot product result and the second dot product result to obtain a second attention result.

[0123] S405. Stack the second attention result and the first feature map to obtain a fifth stacking result.

[0124] Thus, the missing partial area of ​​the target object is determined, so as to infer features associated with the content of the area of ​​the target object blocked by the occluder in the first feature map.

[0125] S406: Perform channel number dimensionality reduction processing on the fifth stacking result to obtain a second feature map. Thus, the first feature map and the second feature map can maintain the same spatial dimension, avoiding the influence of the attention mechanism model on the operation of the current network, thereby eliminating the need for the current network layer to adjust the spatial dimension of the second feature map, and simplifying the process of the current network layer outputting the first feature map to the next network layer.

[0126] Based on the above description, the electronic device can use the attention mechanism module 14 to introduce a channel number attention mechanism based on the parsing mask to obtain a second feature map for representing the content enhanced features of the area of ​​the target object blocked by the occluder. Exemplarily, the present disclosure provides an image processing device.

[0127] See also Figure 7 , Figure 7 The image processing device of the present disclosure can be set in an electronic device to achieve the above Figure 1-6B The image processing method of the embodiment corresponds to the operation of the electronic device. Figure 7 As shown, the image processing device 700 provided by the present disclosure may include: an acquisition module 701 and a processing module 702 .

[0128] The acquisition module 701 is used to acquire an image to be processed, a first parsing mask and a second parsing mask, wherein the image to be processed includes a target object, a part of the target object is blocked by an occluder, the first parsing mask is a parsing mask of a region of the target object not blocked by the occluder, and the second parsing mask is a parsing mask of a complete region of the target object;

[0129] The processing module 702 is used to input the image to be processed, the first parsing mask and the second parsing mask into the neural network model, and complete the content of the area of ​​the target object blocked by the occluder based on the attention mechanism.

[0130] In some embodiments, the processing module 702 is specifically used to process the image to be processed through multiple network layers in the neural network model, and complete the content of the area blocked by the occluder of the target object based on the attention mechanism; wherein the processing process of at least one network layer among the multiple network layers includes: receiving a first feature map output by a previous network layer, wherein the first feature map is obtained based on the image to be processed, the first parsing mask and the second parsing mask; obtaining a second feature map according to the first feature map, the first parsing mask and the second parsing mask, wherein the second feature map is a feature after the content of the area blocked by the occluder of the target object is enhanced, the sizes of the first feature map, the first parsing mask and the second parsing mask are the same, and the number of channels of the first parsing mask and the second parsing mask are the same; performing convolution processing on the second feature map.

[0131] In some embodiments, the multiple network layers include a first network layer and a second network layer, the first feature map received by the first network layer has a different size from the first feature map received by the second network layer, the first parsing mask in the processing process of the first network layer has a different size from the first parsing mask in the processing process of the second network layer, and the second parsing mask in the processing process of the first network layer has a different size from the second parsing mask in the processing process of the second network layer.

[0132] In one possible design, the processing module 702 is used to obtain a first attention result based on the first feature map, the first parsing mask and the second parsing mask. The first attention result is used to represent the association relationship between each spatial position point of the target object in the feature map corresponding to the image to be processed and all the spatial position points, and / or obtain a second attention result based on the first feature map, the first parsing mask and the second parsing mask. The second attention result is used to represent the semantic information and depth features of the sub-component area in the area obscured by the occluder of the target object corresponding to the difference between the first parsing mask and the second parsing mask; obtain a second feature map based on the first attention result and / or the second attention result.

[0133] In some embodiments, the processing module 702 is used to obtain an associated feature map based on the first feature map, the first parsing mask and the second parsing mask, and use the associated feature map as the first attention result, the associated feature map is used to represent the association relationship between the area of ​​the target object that is not obscured by the occluder and the complete area of ​​the target object; stack the associated feature map and the first feature map to obtain a first stacking result; perform dimensionality reduction processing on the first stacking result to obtain a second feature map.

[0134] In some embodiments, the processing module 702 is used to obtain an associated feature map according to the first feature map, the first parsing mask and the second parsing mask, and use the associated feature map as the first attention result, and the associated feature map is used to represent the association relationship between the area of ​​the target object that is not blocked by the occluder and the complete area of ​​the target object; perform channel number dimensionality reduction processing on the first feature map to obtain a reduced dimensionality feature map, and the number of channels of the reduced dimensionality feature map is the same as the number of channels of the first parsing mask or the second parsing mask; perform dot multiplication processing on the reduced dimensionality feature map and the first parsing mask to obtain a first dot multiplication result, and the first dot multiplication result is used to represent the target object The semantic information and depth features of the sub-component area corresponding to the first parsing mask in the area not blocked by the occluder are obtained; the reduced dimension feature map and the second parsing mask are subjected to dot multiplication to obtain a second dot product result, and the second dot product result is used to represent the semantic information and depth features of the sub-component area corresponding to the second parsing mask in the complete area of ​​the target object; the first dot product result and the second dot product result are stacked and convolved to obtain a second attention result; the second attention result, the associated feature map and the first feature map are stacked to obtain a second stacking result; the second stacking result is subjected to dimensionality reduction to obtain a second feature map.

[0135] In some embodiments, the processing module 702 is used to stack the first feature map and the first parsing mask to obtain a third stacking result; stack the first feature map and the second parsing mask to obtain a fourth stacking result; and perform attention mechanism processing on the third stacking result and the fourth stacking result to obtain an associated feature map.

[0136] In some embodiments, the processing module 702 is used to perform channel number reduction processing on the first feature map to obtain a reduced dimension feature map, and the number of channels of the reduced dimension feature map is the same as the number of channels of the first parsing mask or the second parsing mask; perform dot multiplication processing on the reduced dimension feature map and the first parsing mask to obtain a first dot multiplication result, and the first dot multiplication result is used to represent the semantic information and depth features of the sub-component area corresponding to the first parsing mask in the area of ​​the target object that is not obscured by the occluder; perform dot multiplication processing on the reduced dimension feature map and the second parsing mask to obtain a second dot multiplication result, and the second dot multiplication result is used to represent the semantic information and depth features of the sub-component area corresponding to the second parsing mask in the complete area of ​​the target object; stack and convolve the first dot multiplication result and the second dot multiplication result to obtain a second attention result; stack the second attention result and the first feature map to obtain a fifth stacking result; perform dimensionality reduction processing on the fifth stacking result to obtain a second feature map.

[0137] In one possible design, the first parsing mask and the second parsing mask are in the form of one-hot encoding.

[0138] The image processing device provided by the present disclosure can execute the above method embodiments. Its specific implementation principles and technical effects can be found in the above method embodiments, and the present disclosure will not repeat them here.

[0139] Exemplarily, the present disclosure provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory; when the one or more processors execute the one or more computer programs, the electronic device implements the image processing method of the foregoing embodiments.

[0140] Illustratively, the present disclosure provides a chip system, which is applied to an electronic device including a memory and a sensor; the chip system includes: a processor; when the processor executes the image processing method of the above embodiment.

[0141] Exemplarily, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor in an electronic device, the image processing method of the foregoing embodiment is implemented.

[0142] Exemplarily, the present disclosure provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the image processing method of the foregoing embodiment.

[0143] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. Available media can be magnetic media, (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)), etc.

[0144] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0145] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, a first parsing mask, and a second parsing mask, wherein the image to be processed includes a target object, a partial area of ​​the target object is blocked by an occluder, the first parsing mask is a parsing mask of an area of ​​the target object that is not blocked by the occluder, and the second parsing mask is a parsing mask of a complete area of ​​the target object; Inputting the image to be processed, the first parsing mask and the second parsing mask into a neural network model, and completing the content of the area of ​​the target object blocked by the occluder based on an attention mechanism; Among them, the attention mechanism-based completion of the content of the area obscured by the occluder of the target object includes completing the content of the area obscured by the occluder of the target object according to the spatial position association relationship of the target object in the image to be processed, and / or the semantic information and depth features of the area obscured by the occluder of the target object.

2. The method according to claim 1, characterized in that The step of inputting the image to be processed, the first parsing mask and the second parsing mask into a neural network model, and completing the content of the area of ​​the target object blocked by the occluder based on an attention mechanism, comprises: Processing the image to be processed through multiple network layers in the neural network model, and completing the content of the area of ​​the target object blocked by the occluder based on the attention mechanism; The processing of at least one network layer among the multiple network layers includes: Receiving a first feature map output by a previous network layer, where the first feature map is obtained based on the image to be processed, the first parsing mask, and the second parsing mask; Obtaining a second feature map according to the first feature map, the first parsing mask, and the second parsing mask, wherein the second feature map is a feature of a region of the target object that is blocked by the occluder after content enhancement, the first feature map, the first parsing mask, and the second parsing mask have the same size, and the first parsing mask and the second parsing mask have the same number of channels; The second feature map is convolved and the convolved feature map is transmitted to the next network layer.

3. The method according to claim 2, characterized in that The multiple network layers include a first network layer and a second network layer. The first feature map received by the first network layer has a size different from that of the first feature map received by the second network layer. The first parsing mask in the processing of the first network layer has a size different from that of the first parsing mask in the processing of the second network layer. The second parsing mask in the processing of the first network layer has a size different from that of the second parsing mask in the processing of the second network layer.

4. The method according to claim 2 or 3, characterized in that: The step of obtaining a second feature map according to the first feature map, the first parsing mask and the second parsing mask comprises: According to the first feature map, the first parsing mask and the second parsing mask, a first attention result is obtained, wherein the first attention result is used to represent the association relationship between each spatial position point of the target object in the feature map corresponding to the image to be processed and all spatial position points, and / or according to the first feature map, the first parsing mask and the second parsing mask, a second attention result is obtained, wherein the second attention result is used to represent the semantic information and depth features of the sub-component area corresponding to the difference between the first parsing mask and the second parsing mask in the area blocked by the occluder of the target object; The second feature map is obtained according to the first attention result and / or the second attention result.

5. The method according to claim 4, characterized in that The obtaining a first attention result according to the first feature map, the first parsing mask and the second parsing mask comprises: Obtaining a correlation feature map according to the first feature map, the first parsing mask, and the second parsing mask, and using the correlation feature map as the first attention result, wherein the correlation feature map is used to represent a correlation relationship between an area of ​​the target object that is not blocked by the occluder and a complete area of ​​the target object; Performing stacking processing on the associated feature map and the first feature map to obtain a first stacking result; Perform dimensionality reduction processing on the first stacking result to obtain the second feature map.

6. The method according to claim 4, characterized in that The obtaining, according to the first feature map, the first parsing mask and the second parsing mask, a first attention result and a second attention result, comprises: Obtaining a correlation feature map according to the first feature map, the first parsing mask, and the second parsing mask, and using the correlation feature map as the first attention result, wherein the correlation feature map is used to represent a correlation relationship between an area of ​​the target object that is not blocked by the occluder and a complete area of ​​the target object; Performing channel number dimensionality reduction processing on the first feature map to obtain a reduced-dimensionality feature map, wherein the number of channels of the reduced-dimensionality feature map is the same as the number of channels of the first parsing mask or the second parsing mask; Performing a dot product process on the reduced dimension feature map and the first parsing mask to obtain a first dot product result, where the first dot product result is used to represent semantic information and depth features of a sub-component region corresponding to the first parsing mask in a region of the target object that is not blocked by the occluder; Performing a dot product process on the reduced dimension feature map and the second parsing mask to obtain a second dot product result, where the second dot product result is used to represent semantic information and depth features of a sub-component region corresponding to the second parsing mask in the complete region of the target object; Performing stacking and convolution processing on the first dot product result and the second dot product result to obtain a second attention result; The obtaining the second feature map according to the first attention result and the second attention result includes: Stacking the second attention result, the associated feature map, and the first feature map to obtain a second stacking result; Perform dimensionality reduction processing on the second stacking result to obtain the second feature map.

7. The method according to claim 5 or 6, characterized in that: The step of obtaining an associated feature map according to the first feature map, the first parsing mask and the second parsing mask comprises: Performing stacking processing on the first feature map and the first parsing mask to obtain a third stacking result; Performing stacking processing on the first feature map and the second parsing mask to obtain a fourth stacking result; Perform attention mechanism processing on the third stacking result and the fourth stacking result to obtain the associated feature map.

8. The method according to claim 4, characterized in that The obtaining a second attention result according to the first feature map, the first parsing mask and the second parsing mask comprises: Performing channel number dimensionality reduction processing on the first feature map to obtain a reduced-dimensionality feature map, wherein the number of channels of the reduced-dimensionality feature map is the same as the number of channels of the first parsing mask or the second parsing mask; Performing a dot product process on the reduced dimension feature map and the first parsing mask to obtain a first dot product result, where the first dot product result is used to represent semantic information and depth features of a sub-component region corresponding to the first parsing mask in a region of the target object that is not blocked by the occluder; Performing a dot product process on the reduced dimension feature map and the second parsing mask to obtain a second dot product result, where the second dot product result is used to represent semantic information and depth features of a sub-component region corresponding to the second parsing mask in the complete region of the target object; Performing stacking and convolution processing on the first dot product result and the second dot product result to obtain a second attention result; The obtaining the second feature map according to the second attention result includes: Stacking the second attention result and the first feature map to obtain a fifth stacking result; Perform dimensionality reduction processing on the fifth stacking result to obtain the second feature map.

9. The method according to any one of claims 1-3, 5-6, and 8, characterized in that: The first parsing mask and the second parsing mask are in a one-hot encoding form.

10. An image processing device, characterized in that: The device comprises: an acquisition module, configured to acquire an image to be processed, a first parsing mask, and a second parsing mask, wherein the image to be processed includes a target object, a part of the target object is blocked by an occluder, the first parsing mask is a parsing mask of a region of the target object that is not blocked by the occluder, and the second parsing mask is a parsing mask of a complete region of the target object; A processing module, configured to input the image to be processed, the first parsing mask and the second parsing mask into a neural network model, and complete the content of the area of ​​the target object blocked by the occluder based on an attention mechanism; Among them, the attention mechanism-based completion of the content of the area obscured by the occluder of the target object includes completing the content of the area obscured by the occluder of the target object based on the spatial position association information of the target object in the image to be processed, and / or the semantic information and depth features corresponding to the area obscured by the occluder of the target object.

11. An electronic device, comprising: one or more processors; Memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory; characterized in that when the one or more processors execute the one or more computer programs, the electronic device implements the image processing method as described in any one of claims 1-9.

12. A computer storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the image processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 9.

Images