Image processing method, device, electronic device, medium and computer program product
By using a target neural network model in image processing combined with preset and reference masks for training, the problem of low mask accuracy in target object completion is solved, and higher precision image processing results are achieved.
Patent Information
- Application Number
- CN202110246097.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-03-05
AI Technical Summary
In existing technologies, image processing methods do not achieve high accuracy in obtaining complete target masks when completing occluded target objects.
The first mask of the visible region of the target object in the original image is obtained, and then stacked with the original image and input into the target neural network model. The target complete mask is trained by the model based on the original image sample, the target object sample, the preset parsed mask and the reference complete mask. The model is then trained with two supervision signals to improve accuracy.
It improves the accuracy of the target integrity mask of the target object, thereby enhancing the precision and efficiency of image processing, especially in target object completion and image segmentation tasks.
Smart Images

Figure CN112967197B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, medium and computer program product. Background Art
[0002] In the field of image processing, it is usually necessary to complete some partially occluded target objects in an image. The target object can be a human body or other objects. Taking the human body as an example, if a part of the human body is occluded by an occluder in an image, a series of image processing is required to complete the occluded part of the human body and obtain a complete human body. Usually, in the process of completing the target object, it is usually necessary to predict the target complete mask of the target object.
[0003] In the prior art, the original image is usually input into an instance segmentation network, a mask of the visible area of the target object is obtained through the instance segmentation network, and then a complete target mask of the target object is obtained through a neural network model based on the mask of the visible area of the target object.
[0004] However, the accuracy of obtaining a complete target mask of a target object using the existing technology is not high. Summary of the invention
[0005] In order to solve the above technical problems, the present disclosure provides an image processing method, apparatus, electronic device, medium and computer program product.
[0006] A first aspect of the present disclosure provides an image processing method, comprising:
[0007] Acquire a first mask of a visible area of a target object in an original image, wherein the original image contains the target object and a part of the target object is blocked by an occluder;
[0008] The original image and the first mask are stacked and input into a target neural network model to obtain a target complete mask of the target object in the original image, wherein the target neural network model is trained based on the original image sample, a preset parsing mask of the target object sample, and a reference complete mask of the target object sample, the original image sample contains the target object sample, a partial area of the target object sample is obscured by an occluder, and the preset parsing mask contains semantic mask labels of each part of the target object sample.
[0009] Optionally, also include:
[0010] The first mask is subtracted from the target complete mask of the target object to obtain a mask of the invisible area of the target object.
[0011] Optionally, after stacking the original image and the first mask and inputting them into the target neural network model, the method further includes:
[0012] A complete parsed mask of the target object in the original image is output, where the complete parsed mask contains semantic mask labels of various parts of the target object.
[0013] Optionally, also include:
[0014] Subtracting the first mask from the target complete mask of the target object to obtain a mask of the invisible area of the target object;
[0015] The mask of the invisible region of the target object is multiplied by the complete parsed mask of the target object to obtain a semantic mask label of the invisible region of the target object.
[0016] Optionally, before acquiring the first mask of the visible area of the target object in the original image, the method further includes:
[0017] Acquire a second mask of the target object sample in the original image sample;
[0018] Inputting the original image sample and the second mask into a neural network model to obtain a target complete mask of a target object sample in the original image sample and a complete parsed mask of the target object sample;
[0019] The reference complete mask of the target object sample and the preset parsing mask of the target object sample are used as supervisory signals, and a loss function is obtained using the supervisory signal and the target complete mask of the target object sample and the complete parsing mask of the target object sample. The neural network model is trained until the neural network model converges, and the converged neural network model is used as the target neural network model.
[0020] Optionally, also include:
[0021] The original image sample is input into a target object parsing network model to obtain a preset parsing mask of the target object sample.
[0022] A second aspect of the present disclosure provides an image processing device, the device comprising:
[0023] An acquisition module, configured to acquire a first mask of a visible area of a target object in an original image, wherein the original image contains the target object and a part of the target object is blocked by an occluder;
[0024] A processing module is used to stack the original image and the first mask and input them into a target neural network model to obtain a target complete mask of the target object in the original image, wherein the target neural network model is trained based on the original image sample, a preset parsing mask of the target object sample and a reference complete mask of the target object sample, the original image sample contains the target object sample, part of the target object sample is blocked by an occluder, and the preset parsing mask contains semantic mask labels of various parts of the target object sample.
[0025] Optionally, the processing module is further configured to subtract the first mask from the target complete mask of the target object to obtain a mask of the invisible area of the target object.
[0026] Optionally, the processing module is further configured to output a complete parsed mask of the target object in the original image, wherein the complete parsed mask includes semantic mask labels of various parts of the target object.
[0027] Optionally, the processing module is also used to subtract the first mask from the target complete mask of the target object to obtain a mask of an invisible area of the target object; and multiply the mask of the invisible area of the target object by the complete parsed mask of the target object to obtain a semantic mask label of the invisible area of the target object.
[0028] Optionally, the mask of the visible area of the target object is the first mask.
[0029] Optionally, the acquisition module is further used to acquire a second mask of the target object sample in the original image sample;
[0030] The processing module is also used to input the original image sample and the second mask into the neural network model to obtain the target complete mask of the target object sample in the original image sample and the complete parsing mask of the target object sample; use the reference complete mask of the target object sample and the preset parsing mask of the target object sample as supervisory signals, use the supervisory signals and the target complete mask of the target object sample and the complete parsing mask of the target object sample to obtain a loss function, train the neural network model until the neural network model converges, and use the converged neural network model as the target neural network model.
[0031] Optionally, the processing module is further used to input the original image sample into a target object parsing network model to obtain a preset parsing mask of the target object sample.
[0032] A third aspect of the present disclosure provides an electronic device, comprising: a processor, wherein the processor is configured to execute a computer program stored in a memory, wherein the computer program implements the steps of the method described in the first aspect when executed by the processor.
[0033] A fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.
[0034] A fifth aspect of the present disclosure provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the image processing method described in the first aspect.
[0035] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0036] Since the target neural network model is trained based on the original image samples, the preset parsing masks of the target object samples and the reference complete masks of the target object samples, the semantic information of each part of the target object is taken into account. Therefore, by obtaining a first mask of the visible area of the target object in the original image, wherein the original image contains the target object and a partial area of the target object is obscured by an occluder; the original image and the first mask are stacked and input into the target neural network model to obtain the target complete mask of the target object in the original image, the accuracy of the target complete mask of the target object is higher. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0039] Figure 1 A schematic diagram of human body demasking provided by the present disclosure;
[0040] Figure 2 A schematic diagram of an image processing system provided by the present disclosure;
[0041] Figure 3 A schematic diagram of an image processing process provided by the present disclosure;
[0042] Figure 4 A schematic diagram of another image processing process provided by the present disclosure;
[0043] Figure 5 A schematic diagram of another image processing process provided by the present disclosure;
[0044] Figure 6 A schematic diagram of another image processing process provided by the present disclosure;
[0045] Figure 7 A structural schematic diagram of an image processing device is provided in the present disclosure. DETAILED DESCRIPTION
[0046] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0048] The original image of the present disclosure contains a target object, wherein a part of the target object is blocked by an occluder. For the convenience of description, the area of the target object blocked by the occluder is described as an invisible area of the target object. The area of the target object not blocked by the occluder is described as a visible area of the target object. The target object may be a human body, an animal or other object, which is not limited by the present disclosure.
[0049] In the process of completing the invisible area of the target object in the original image, it is necessary to predict the mask of the invisible area of the target object and complete the target object based on the mask of the invisible area of the target object, so as to achieve image processing tasks such as target tracking, target detection and image segmentation.
[0050] The mask of the invisible region of the target object is usually obtained by subtracting the target complete mask of the target object and the mask of the visible region of the target object. Therefore, the accuracy of the target complete mask of the target object directly affects the accuracy of the mask of the invisible region of the target object.
[0051] The parsing mask of the target object can be understood as a code formed by different numbers corresponding to each part after the target object is segmented.
[0052] The target object of the present disclosure may be a human body, an animal, or other objects. The following embodiments of the present disclosure are described and illustrated using the human body as an example. Other target objects are similar to the human body and are not described in detail.
[0053] Taking the human body as an example, human parsing is to segment the human body into N parts, where N is a positive integer.
[0054] Assume N = 19, such as Figure 1 As shown, Figure 1 A schematic diagram of a human body de-masking provided by the present disclosure, wherein the 19 parts of the human body may include: head a1, neck a2, left shoulder a3, right shoulder a4, left upper arm a5, right upper arm a6, left lower arm a7, right lower arm a8, left hand a9, right hand a10, left hip a11, right hip a12, left thigh a13, right thigh a14, left calf a15, right calf a16, left foot a17, right foot a18 and body a19, and the human body de-masking mask is a code formed by one number corresponding to each of the 19 parts of the human body. In addition, the present disclosure is not limited to dividing the human body into 19 parts.
[0055] By parsing the target object, the semantic information of each segmented part of the target object can be obtained.
[0056] In order to improve the accuracy of the target complete mask of the target object obtained, the present disclosure obtains the target complete mask of the target object through a target neural network model, wherein the target neural network model is trained based on the original image sample, the preset parsing mask of the target object sample and the reference complete mask of the target object sample. Wherein, the reference complete mask of the target object sample refers to the accurate complete mask of the acquired target object sample, and the preset parsing mask contains reliable semantic mask labels of each part of the target object sample, and these semantic mask labels are used as pseudo labels, that is, they are considered to be real semantic mask labels, and participate in the neural network training. Since in the process of training the target neural network model, not only the reference complete mask of the target object sample is used as a supervisory signal, but also the preset parsing mask of the target object sample is combined as a supervisory signal, since the parsing mask takes into account the semantic information of each part of the target object, the accuracy of the semantics of each part of the target object is considered during training, therefore, the accuracy of the target complete mask output by the target neural network model obtained by combining the two supervisory signals for training is higher.
[0057] The image processing method disclosed in the present invention is performed by an electronic device. The electronic device may be a tablet computer, a mobile phone (such as a folding screen mobile phone, a large screen mobile phone, etc.), a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, a high-definition TV, a 4K TV, a smart speaker, a smart projector, and other Internet of Things (IOT) devices. The present disclosure does not impose any restrictions on the specific types of electronic devices.
[0058] Figure 2 A schematic diagram of an image processing system provided by the present disclosure, such as Figure 2 As shown, the system includes:
[0059] The target neural network model 101 includes a plurality of network layers, some of which are used to predict the target complete mask of the target object, and some of which are used to predict the complete parsing mask of the target object. The target neural network model 101 may include an hourglass network 101a and a multi-layer convolutional network 101b (also called a segmentation head or a task head). The input signal of the target neural network model 101 is a first mask 103 of a visible area of a target object of an original image and a stacking result of the original image 102. The output of the target neural network model 101 is a target complete mask 104 and a complete parsing mask 105 of the target object, wherein the target complete mask 104 is a binary segmentation mask; the complete parsing mask 105 refers to a mask that can identify the semantics of each part of the target object, and the complete parsing mask includes semantic mask labels of each part of the target object. According to the semantic mask labels, it can be determined what each part of the target object is, for example, head, neck, left shoulder, right shoulder, left upper arm, right upper arm, left lower arm, right lower arm, left hand, right hand, left hip, right hip, left thigh, right thigh, left calf, right calf, left foot, right foot or body.
[0060] Figure 3 A schematic diagram of an image processing process provided by the present disclosure, combined with Figure 2 , the method of this embodiment is as follows:
[0061] S301: Obtain a first mask of a visible area of a target object in an original image.
[0062] The original image contains a target object, and a part of the target object is blocked by an occluder. The visible area of the target object refers to the area of the target object displayed in the original image. The original image generally contains a background, a target object, and an occluder.
[0063] by Figure 2 Take for example, where the target object is a human body, the occluder is grass, the legs and feet of the human body are blocked by the occluder, and the head, upper body and part of the legs are visible areas of the human body.
[0064] Exemplarily, the size of the original image may be H×W×3, where H is the height, W is the width, and 3 is the number of channels.
[0065] Optionally, the first mask of the visible area of the target object in the original image can be obtained through the instance segmentation network. The size of the first mask is H×W×1, and the values in the first mask obey the distribution of 0 to 1. However, since the instance segmentation network does not consider the occlusion situation, that is, the instance segmentation network is used to segment various objects and is not designed for target objects, the first mask of the visible area of the target object in the original image obtained by the instance segmentation network may be incomplete or partially wrong, and the accuracy is not high.
[0066] In this step, the original image can be recorded as: s , the instance segmentation network is recorded as: N i (), the first mask is recorded as: M i ,but,
[0067] M i =N i (I s )
[0068] S302: The original image and the first mask are stacked and input into a target neural network model to obtain a target complete mask corresponding to the target object in the original image.
[0069] Among them, the target complete mask corresponding to the target object in the original image refers to the mask of binary segmentation.
[0070] Specifically, the original image and the first mask may be stacked to obtain a stacked result, and the stacked result may be input into the target neural network model to obtain a target complete mask corresponding to the target object. Combined with the description in S301, the stacked result is an H×W×4 image, that is, the stacked result is a four-channel image.
[0071] The target neural network model is recorded as: The target complete mask corresponding to the target object: M a ,but:
[0072]
[0073] Since the target neural network model is trained based on the original image samples, the preset parsing masks of the target object samples and the reference complete masks of the target object samples, that is, two tasks are completed during the training process, one task is to obtain the target complete mask of the target object samples, and the other task is to obtain the complete parsing mask of the target object samples. Since the complete parsing mask of the target object samples contains the semantic mask labels of each part of the target object, therefore, when the target neural network model is trained with the preset parsing mask and the reference complete mask as supervisory signals, more dimensions are measured, thereby improving the accuracy of the target complete mask of the target object output by the target neural network model.
[0074] In this embodiment, since the target neural network model is trained based on the original image samples, the preset parsing masks of the target object samples and the reference complete masks of the target object samples, the semantic information of each part of the target object is taken into consideration. Therefore, by obtaining a first mask of the visible area of the target object in the original image, wherein the original image contains the target object and a partial area of the target object is obscured by an occluder; the original image and the first mask are stacked and input into the target neural network model to obtain the target complete mask of the target object in the original image, the accuracy of the target complete mask of the target object is higher.
[0075] Figure 4 A schematic diagram of another image processing process provided by the present disclosure, Figure 4 is Figure 3 On the basis of the embodiment shown, further, it also includes:
[0076] S303: Obtain a complete parsed mask of the target object in the original image.
[0077] Among them, the complete parsed mask contains the semantic mask labels of each part of the target object.
[0078] In some scenarios, the complete resolution mask of the target object may be used for image processing. For example, during image completion, the completion effect may be determined based on the complete resolution mask of the completed image.
[0079] Therefore, the present disclosure also outputs a complete parsed mask of the target object in the original image, so as to facilitate image processing in some scenarios and improve the accuracy of image processing.
[0080] Furthermore, a target complete mask and a complete resolution mask of a target object can be obtained simultaneously through a target neural network model. For a scenario where the target complete mask and the complete resolution mask of a target object are simultaneously applied to perform image processing, the efficiency of image processing can be improved.
[0081] Figure 5 A schematic diagram of another image processing process provided by the present disclosure is shown below: Figure 5 is Figure 3 or Figure 4 On the basis of the illustrated embodiment, further, in some scenarios where both the visible area mask of the target object and the target complete mask of the target object are applied to perform image processing, for example, in the scenario of target object completion, it is necessary to complete the target object based on the invisible area mask. Therefore, the method of the present disclosure may also include:
[0082] S304: Subtract the mask of the visible area of the target object from the target complete mask of the target object to obtain the mask of the invisible area of the target object.
[0083] A possible implementation manner is to use the first mask as a mask of a visible area of the target object, that is, to obtain a mask of an invisible area of the target object by subtracting a mask of the visible area of the target object from a complete target mask of the target object.
[0084] Another possible implementation method is: obtaining the accurate mask of the visible area of the target object by other means, for example, obtaining the accurate mask of the visible area of the target object by clipping, etc., subtracting the accurate mask of the visible area of the target object from the complete target mask of the target object to obtain the mask of the invisible area of the target object, thereby further improving the accuracy of the mask of the invisible area of the target object.
[0085] In the present disclosure, since the accuracy of the target complete mask is improved, the accuracy of the mask of the invisible area of the target object obtained by subtracting the target complete mask of the target object from the mask of the visible area of the target object will also be improved. Image completion and other processing can be performed based on the obtained mask of the invisible area, thereby further improving the effect of image completion.
[0086] exist Figure 5 On the basis of the illustrated embodiment, further, in some scenarios, a semantic mask label needs to be applied to the invisible area of the target object, and therefore, S305 may also be included:
[0087] S305: Multiply the mask of the invisible region of the target object by the complete parsed mask of the target object to obtain a semantic mask label of the invisible region of the target object.
[0088] Since the mask of the invisible area only has the value of 1 in the invisible area, and the values of the other areas are all 0, the semantic mask label of the invisible area of the target object can be obtained by multiplying the mask of the invisible area of the target object with the complete parsed mask of the target object. Thus, based on the semantic mask label of the invisible area of the target object, it is possible to determine which part of the target object the invisible area is, that is, which part of the target object is blocked by the occluder. Taking the target object as a human body as an example, it can be determined according to the semantic mask label whether it is the head, neck, left shoulder, right shoulder, left upper arm, right upper arm, left lower arm, right lower arm, left hand, right hand, left hip, right hip, left thigh, right thigh, left calf, right calf, left foot, right foot or body that is blocked. Optionally, the invisible area can be further completed according to the semantic mask label. For example, if it is determined that the right foot is blocked according to the semantic mask label, the invisible area can be completed according to the characteristics of the right foot. Optionally, the completion effect may be further measured based on the semantic mask label. For example, if the semantic mask label determines that the right foot is occluded, but the image after the invisible area is completed is the left foot, it can be determined that the completion accuracy is not high.
[0089] Figure 6 A schematic diagram of another image processing process provided by the present disclosure is shown below. Figure 6 is Figure 3-Figure 5 On the basis of any of the embodiments shown, further, it also includes: a process of obtaining a target neural network model, that is, a process of obtaining a target neural network model through machine learning, such as Figure 6 As shown:
[0090] S601: Obtain a second mask of a target object sample in an original image sample.
[0091] The original image sample includes the target object sample, and part of the target object sample is blocked by the occluder. The visible area of the target object sample refers to the area of the target object sample displayed in the original image sample. The original image sample generally includes the background, the target object sample and the occluder.
[0092] Optionally, a second mask of a visible area of a target object sample in the original image sample may be obtained through an instance segmentation network.
[0093] S602: The original image sample and the second mask are stacked and input into a target neural network model to obtain a target complete mask of the target object sample in the original image sample and a complete parsed mask of the target object sample.
[0094] That is, the target neural network model processes two tasks simultaneously, one task is to obtain the target complete mask of the target object sample, and the other task is to obtain the complete parsed mask of the target object sample.
[0095] S603: Using the reference complete mask of the target object sample and the preset parsing mask of the target object sample as supervisory signals, and using the supervisory signals and the target complete mask of the target object sample and the complete parsing mask of the target object sample to obtain a loss function, and training the neural network model until the neural network model converges, and using the converged neural network model as the target neural network model.
[0096] Optionally, the preset parsing mask of the target object sample may be obtained by, but not limited to, the following possible implementation methods:
[0097] Among them, a possible implementation method is: mark the semantic mask labels of various parts of the target object by manual labeling, and then obtain the preset parsing mask of the target object sample from the semantic mask label, and the preset parsing mask of the target object sample contains the semantic mask labels of various parts of the target object sample.
[0098] Another possible implementation manner is: inputting the original image sample into a target object parsing network model to obtain a preset parsing mask of the target object sample.
[0099] The reference complete mask of the target object sample and the preset parsing mask of the target object sample are used as supervisory signals. During the training of the neural network model, a first loss function is obtained according to the target complete mask of the target object sample output by the neural network model and the reference complete mask of the target object sample. A second loss function is obtained according to the complete parsing mask of the target object sample output by the neural network model and the preset parsing mask of the target object sample. The neural network model is trained according to the first loss function and the second loss function. That is, the parameters of the neural network model are adjusted according to the first loss function and the second loss function until the neural network model converges, that is, until the first loss function and the second loss function meet the preset requirements, that is, until the accuracy of the target complete mask of the output target object sample and the complete parsing mask of the target object sample meets the requirements, then the neural network model is considered to converge. The converged neural network model is used as the target neural network model.
[0100] Specifically, the neural network model can be trained based on the gradient descent method according to the first loss function and the second loss function until the neural network model converges. In this embodiment, the second mask of the target object sample in the original image sample is obtained, and the original image sample and the second mask are matched and input into the neural network model to obtain the target complete mask of the target object sample in the original image sample and the complete parsing mask of the target object sample; the reference complete mask of the target object sample and the preset parsing mask of the target object sample are used as supervisory signals to train the neural network model until the neural network model converges, and the converged neural network model is used as the target neural network model. Since the complete parsing mask of the target object sample contains the semantic mask labels of each part of the target object, when the target neural network model is trained with the preset parsing mask and the reference complete mask as supervisory signals, more dimensions are measured, thereby improving the accuracy of the target complete mask of the target object output by the target neural network model.
[0101] Figure 7 The present invention provides a structural schematic diagram of an image processing device, such as Figure 7 As shown, the device of this embodiment includes: an acquisition module 701 and a processing module 702, wherein:
[0102] An acquisition module 701 is used to acquire a first mask of a visible area of a target object in an original image, wherein the original image contains the target object and a part of the target object is blocked by an occluder;
[0103] The processing module 702 is used to stack the original image and the first mask and input them into the target neural network model to obtain a target complete mask of the target object in the original image, wherein the target neural network model is trained based on the original image sample, the preset parsing mask of the target object sample and the reference complete mask of the target object sample, the original image sample contains the target object sample, part of the target object sample is blocked by an occluder, and the preset parsing mask contains semantic mask labels of each part of the target object sample.
[0104] Optionally, the processing module 702 is further configured to subtract the first mask from the target complete mask of the target object to obtain a mask of the invisible area of the target object.
[0105] Optionally, the processing module 702 is further configured to output a complete parsed mask of the target object in the original image, where the complete parsed mask includes semantic mask labels of various parts of the target object.
[0106] Optionally, the processing module 702 is further used to subtract the first mask from the target complete mask of the target object to obtain a mask of the invisible area of the target object; and multiply the mask of the invisible area of the target object by the complete parsed mask of the target object to obtain a semantic mask label of the invisible area of the target object.
[0107] Optionally, the acquisition module 701 is further used to acquire a second mask of the target object sample in the original image sample;
[0108] The processing module 702 is also used to input the original image sample and the second mask into the neural network model to obtain the target complete mask of the target object sample in the original image sample and the complete parsing mask of the target object sample; use the reference complete mask of the target object sample and the preset parsing mask of the target object sample as supervisory signals, use the supervisory signals and the target complete mask of the target object sample and the complete parsing mask of the target object sample to obtain a loss function, train the neural network model until the neural network model converges, and use the converged neural network model as the target neural network model.
[0109] Optionally, the processing module 702 is further configured to input the original image sample into a target object parsing network model to obtain a preset parsing mask of the target object sample.
[0110] The device of this embodiment corresponds to the technical solution that can be used to execute the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.
[0111] The present disclosure also provides an electronic device, comprising: a processor, the processor is used to execute a computer program stored in a memory, the computer program is executed by the processor to achieve Figure 3-Figure 6 The steps of any of the method embodiments described in any one of the preceding claims. It should be noted that the processor may be a graphics processing unit (GPU), that is, the program algorithm of the present disclosure may be completely implemented by a GPU. Exemplarily, it may be implemented using PyTorch of a unified computing device architecture (CUDA).
[0112] The present disclosure provides a computer-readable storage medium having a computer program stored thereon, which implements Figure 3-Figure 6 The steps of any one of the method embodiments described above.
[0113] The present disclosure provides a computer program product, when the computer program product is run on a computer, so that the computer executes Figure 3-Figure 6 The steps of any one of the method embodiments described above.
[0114] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. Available media can be magnetic media, (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)), etc.
[0115] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0116] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing an image, It is characterized in that include: Acquire a first mask of a visible area of a target object in an original image, wherein the original image contains the target object and a part of the target object is blocked by an occluder; The original image and the first mask are stacked and input into a target neural network model to obtain a target complete mask of the target object in the original image, wherein the target neural network model is trained based on the original image sample, a preset parsing mask of the target object sample, and a reference complete mask of the target object sample, the original image sample contains the target object sample, a part of the target object sample is blocked by an occluder, and the preset parsing mask contains semantic mask labels of each part of the target object sample; The target neural network model is trained based on original image samples, preset parsing masks of target object samples and reference complete masks of target object samples, including: training the target neural network model using the preset parsing masks and the reference complete masks as supervisory signals.
2. The method according to claim 1, It is characterized in that Also includes: The first mask is subtracted from the target complete mask of the target object to obtain a mask of the invisible area of the target object.
3. The method according to claim 1, It is characterized in that After the stacking of the original image and the first mask is input into the target neural network model, the method further includes: A complete parsed mask of the target object in the original image is output, where the complete parsed mask contains semantic mask labels of various parts of the target object.
4. The method according to claim 3, It is characterized in that Also includes: Subtracting the first mask from the target complete mask of the target object to obtain a mask of the invisible area of the target object; The mask of the invisible region of the target object is multiplied by the complete parsed mask of the target object to obtain a semantic mask label of the invisible region of the target object.
5. The method according to any one of claims 1 to 4, It is characterized in that Before obtaining the first mask of the visible area of the target object in the original image, the method further includes: Acquire a second mask of the target object sample in the original image sample; Inputting the original image sample and the second mask into a neural network model to obtain a target complete mask of a target object sample in the original image sample and a complete parsed mask of the target object sample; The reference complete mask of the target object sample and the preset parsing mask of the target object sample are used as supervisory signals, and a loss function is obtained using the supervisory signal and the target complete mask of the target object sample and the complete parsing mask of the target object sample. The neural network model is trained until the neural network model converges, and the converged neural network model is used as the target neural network model.
6. The method according to claim 5, It is characterized in that Also includes: The original image sample is input into a target object parsing network model to obtain a preset parsing mask of the target object sample.
7. An image processing device, It is characterized in that The device comprises: An acquisition module, configured to acquire a first mask of a visible area of a target object in an original image, wherein the original image contains the target object and a part of the target object is blocked by an occluder; a processing module, configured to stack the original image and the first mask and input them into a target neural network model to obtain a target complete mask of the target object in the original image, wherein the target neural network model is trained based on the original image sample, a preset parsing mask of the target object sample, and a reference complete mask of the target object sample, the original image sample includes the target object sample, a part of the target object sample is blocked by an occluder, and the preset parsing mask includes semantic mask labels of various parts of the target object sample; The processing module is used to train the target neural network model using a preset parsing mask and a reference complete mask as supervisory signals.
8. An electronic device, It is characterized in that include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the method described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, It is characterized in that When the computer program product is run on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 6.
