Image processing method, device, equipment and storage medium

Abstract

The present disclosure provides an image processing method, device, equipment and storage medium. In some embodiments of the present disclosure, the image processing device acquires a target image, performs a feature extraction operation on a physical object region of the target image, and obtains a feature map of the physical object region contained in the target image; the image processing device determines the instance segmentation quality according to the feature map of the physical object region; when the instance segmentation quality does not meet the set quality condition, the image processing device uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the physical object region; when the segmentation effect of the target image using the instance segmentation method is poor, the target image is semantically segmented to improve the segmentation effect of the physical object in the target image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] Perform object detection on images, including semantic segmentation and instance segmentation. Instance segmentation is challenging because it requires correctly detecting all objects in the image while also accurately segmenting each instance. Therefore, it combines elements from the classic computer vision task of object detection, where the goal is to classify individual objects and locate each object using a bounding box. Semantic segmentation requires distinguishing every pixel in the image to distinguish different objects, and different instances of the same object do not need to be segmented separately.

[0003] The MaskRcnn framework is a classic framework in the field of instance segmentation, which evolved from FasterRcnn. Currently, the MaskRcnn framework has a poor effect on the segmentation of entity objects in the target image. Summary of the invention

[0004] The present disclosure provides an image processing method, apparatus, device and storage medium to at least solve the problem of poor instance segmentation effect of target images in related technologies. The technical solution of the present disclosure is as follows:

[0005] The present application provides an image processing method, including:

[0006] Get the target image;

[0007] Performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image;

[0008] Determining instance segmentation quality according to the feature map of the physical object region;

[0009] If the instance segmentation quality does not meet the set quality condition, a pre-trained semantic segmentation model is used to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the entity object area.

[0010] Optionally, performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image includes:

[0011] Performing a feature extraction operation on the target image to obtain a shared feature map;

[0012] Performing a box selection operation on the shared feature map for the physical object region to obtain a first candidate box corresponding to the physical object region;

[0013] Filtering out a second candidate frame that meets a set condition from the first candidate frame;

[0014] A feature aggregation operation is performed on the second candidate box to obtain a feature map of the entity object area contained in the target image.

[0015] Optionally, the filtering out a second candidate box that meets a set condition from the first candidate box includes:

[0016] Calculating, according to the first candidate box and the shared feature map, a probability value that the first candidate box is the physical object area;

[0017] A candidate box whose probability value is greater than a set probability threshold is selected from the first candidate boxes as the second candidate box.

[0018] Optionally, determining the instance segmentation quality according to the feature map of the physical object region includes:

[0019] Inputting the feature map of the entity object region into a segmentation quality prediction model, and obtaining the overlap between the predicted mask and the real mask of the entity object region through the convolution layer and the fully connected layer of the segmentation quality prediction model;

[0020] The instance segmentation quality is determined according to the true confidence predicted by the classification subnet of the segmentation quality prediction model and the overlap between the predicted mask and the true mask.

[0021] Optionally, after performing the entity object region feature extraction operation on the target image to obtain a feature map of the entity object region contained in the target image, the method further includes:

[0022] Inputting the feature map of the entity object region into a pre-trained instance segmentation model to obtain an instance segmentation result corresponding to the entity object region;

[0023] The instance segmentation result is subjected to boundary optimization processing to obtain an instance segmentation result after boundary optimization.

[0024] Optionally, performing boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization includes:

[0025] Upsampling the instance mask corresponding to the instance segmentation result to obtain feature probability values ​​of multiple feature sub-graphs corresponding to the instance mask;

[0026] Selecting a target feature subgraph whose feature probability value falls within a set probability range from a plurality of feature subgraphs corresponding to the instance mask;

[0027] Performing a linear transformation on the features on the target feature subgraph to obtain an optimized feature probability value;

[0028] The optimized feature probability values ​​are used to replace the feature probability values ​​of the target feature subgraph to obtain an instance segmentation result after boundary optimization.

[0029] Optionally, before performing semantic segmentation on the target image using a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area, the method further includes:

[0030] Determining, according to the instance segmentation quality, whether the instance segmentation quality meets a set quality condition;

[0031] If the instance segmentation quality does not meet the set quality condition, the target image is input into a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area.

[0032] Optionally, the determining, according to the instance segmentation quality, whether the segmentation quality meets a set quality condition includes:

[0033] If the instance segmentation quality is greater than or equal to the set quality threshold, the segmentation quality meets the set quality condition;

[0034] If the instance segmentation quality is less than the set quality threshold, the segmentation quality does not meet the set quality condition.

[0035] The present application also provides an image processing device, including:

[0036] An acquisition module, used for acquiring a target image;

[0037] A feature extraction module, used to perform a feature extraction operation on a physical object region of the target image to obtain a feature map of the physical object region contained in the target image;

[0038] A determination module, configured to determine instance segmentation quality according to a feature map of the entity object region;

[0039] The semantic segmentation module uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image if the instance segmentation quality does not meet the set quality conditions, so as to obtain a semantic segmentation result corresponding to the entity object area.

[0040] Optionally, the feature extraction module includes: a feature extraction submodule, a frame selection submodule, a screening submodule and a feature aggregation submodule;

[0041] The feature extraction submodule is used to perform a feature extraction operation on the target image to obtain a shared feature map;

[0042] The box selection submodule is used to perform a box selection operation on the shared feature map for the physical object area to obtain a first candidate box corresponding to the physical object area;

[0043] The screening submodule is used to screen out second candidate boxes that meet set conditions from the first candidate boxes;

[0044] The feature aggregation submodule is used to perform a feature aggregation operation on the second candidate box to obtain a feature map of the entity object area contained in the target image.

[0045] Optionally, when the screening submodule screens out a second candidate box that meets the set conditions from the first candidate box, it is used to calculate the probability value of the first candidate box being the entity object area based on the first candidate box and the shared feature map; and select a candidate box whose probability value is greater than a set probability threshold from the first candidate box as the second candidate box.

[0046] Optionally, when determining the instance segmentation quality based on the feature map of the entity object area, the determination module is used to input the feature map of the entity object area into a segmentation quality prediction model, and obtain the overlap between the predicted mask and the true mask of the entity object area through the convolutional layer and the fully connected layer of the segmentation quality prediction model; and determine the instance segmentation quality based on the true confidence predicted by the classification subnet of the segmentation quality prediction model and the overlap between the predicted mask and the true mask.

[0047] Optionally, the feature extraction module includes: an instance segmentation submodule and a boundary optimization submodule;

[0048] The instance segmentation submodule is used to input the feature map of the entity object area into a pre-trained instance segmentation model to obtain an instance segmentation result corresponding to the entity object area;

[0049] The boundary optimization module is used to perform boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization.

[0050] Optionally, when the boundary optimization module performs boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization, it is used to upsample the instance mask corresponding to the instance segmentation result to obtain feature probability values ​​of multiple feature subgraphs corresponding to the instance mask; select a target feature subgraph whose feature probability value belongs to a set probability range from the multiple feature subgraphs corresponding to the instance mask; perform linear transformation on the features on the target feature subgraph to obtain optimized feature probability values; use the optimized feature probability values ​​to replace the feature probability values ​​of the target feature subgraph to obtain the instance segmentation result after boundary optimization.

[0051] Optionally, before using a pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the entity object area, it is also used to: determine whether the instance segmentation quality meets the set quality conditions based on the instance segmentation quality; if the instance segmentation quality does not meet the set quality conditions, input the target image into the pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area.

[0052] Optionally, the semantic segmentation module determines whether the instance segmentation quality meets a set quality condition according to the instance segmentation quality, and is used to:

[0053] If the instance segmentation quality is greater than or equal to the set quality threshold, the segmentation quality meets the set quality condition;

[0054] If the instance segmentation quality is less than the set quality threshold, a judgment result that semantic segmentation needs to be performed on the target image is generated.

[0055] The present application also provides an electronic device, including:

[0056] processor;

[0057] a memory for storing instructions executable by the processor;

[0058] The processor is configured to execute the instructions to implement each step in the above method.

[0059] The embodiment of the present application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute each step in the above method.

[0060] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0061] In some embodiments of the present disclosure, an image processing device obtains a target image, performs a physical object region feature extraction operation on the target image, and obtains a feature map of the physical object region contained in the target image; the image processing device determines the instance segmentation quality based on the feature map of the physical object region; when the instance segmentation quality does not meet the set quality conditions, the image processing device performs semantic segmentation on the target image using a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the physical object region; when the segmentation effect of the target image using instance segmentation is poor, the target image is semantically segmented to improve the segmentation effect of the physical object in the target image.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0064] Figure 1 It is a structural block diagram of the prior art MaskRcnn model.

[0065] Figure 2 A flowchart of an image processing method provided by an exemplary embodiment of the present disclosure;

[0066] Figure 3 A framework diagram of a MaskRcnn model provided for an exemplary embodiment of the present disclosure;

[0067] Figure 4 A flowchart of another image processing method provided for an exemplary embodiment of the present disclosure;

[0068] Figure 5 A flowchart of another image processing method provided for an exemplary embodiment of the present disclosure;

[0069] Figure 6a A structural block diagram of an image processing device provided for an exemplary embodiment of the present disclosure;

[0070] Figure 6b A structural block diagram of an image processing device provided for an exemplary embodiment of the present disclosure;

[0071] Figure 7 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0072] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0074] Figure 1 : is a structural block diagram of the prior art MaskRcnn model 10. Figure 1 As shown, the MaskRcnn model 10 includes a feature extraction module 11, a candidate box extraction module 12, a feature aggregation module 13, a classification module 14, a regression module 15 and an instance segmentation module 16.

[0075] The feature extraction module 11 is used to perform a feature extraction operation on the target image input to the MaskRcnn model 10 to obtain a shared feature map;

[0076] A candidate box extraction module 12 is used to perform a box selection operation on the shared feature map for the physical object area to obtain a candidate box corresponding to the physical object area;

[0077] A feature aggregation module 13 is used to perform a feature aggregation operation on the image features in the candidate frame to obtain a feature map of the entity object area contained in the target image;

[0078] A classification module 14, used to classify the entity objects contained in the target image according to the feature map of the entity object area contained in the target image, and obtain a classification result;

[0079] A regression module 15 is used to adjust the position of the candidate box corresponding to the entity object area to obtain a candidate box corresponding to each entity object;

[0080] The instance segmentation module 16 is used to perform instance segmentation on the entity objects contained in the target image to obtain an instance mask corresponding to each entity object.

[0081] In the above embodiment, the feature extraction module 11 is a BackBone model, which is a series of convolutional layers used to extract feature maps of images to obtain shared feature maps. The implementation of the BackBone model includes, for example, VGG16, VGG19, GooLeNet, ResNet50, and ResNet101.

[0082] In the above embodiment, the candidate box extraction module 12 is an RPN network, which is used to extract candidate boxes from the shared feature map. In order to achieve better feature map fusion, the RPN network is different from other networks of the RPN network, which use the feature map of the last layer. The feature map of the last layer has strong semantics, but the position and resolution are relatively low, and it is easy to fail to detect relatively small objects. The RPN network fuses the feature maps from the bottom layer to the high layer, thereby making full use of the features extracted at each stage.

[0083] In the above embodiment, the feature aggregation module is a RoiAlign module.

[0084] In the above embodiment, the output size of the instance segmentation module is limited, and MaskRcnn uses a 28*28 output, which causes the boundary of the segmented entity object to be relatively rough. In addition, when the entity objects in the target image overlap, the instance segmentation module will still try to segment a single instance, resulting in a poor segmentation effect on the entity objects in the target image.

[0085] In response to the above-mentioned technical problems, in some embodiments of the present disclosure, an image processing device obtains a target image, performs an entity object region feature extraction operation on the target image, and obtains a feature map of the entity object region contained in the target image; the image processing device determines the instance segmentation quality based on the feature map of the entity object region; if the instance segmentation quality does not meet the set quality conditions, a pre-trained semantic segmentation model is used to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the entity object region. When the segmentation effect of the target image using instance segmentation is poor, semantic segmentation is performed on the target image to improve the segmentation effect of the entity object in the target image.

[0086] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0087] Figure 2 FIG. 1 is a flow chart of an image processing method provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0088] S201: Acquire a target image;

[0089] S202: performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image;

[0090] S203: Determine instance segmentation quality according to the feature map of the entity object area;

[0091] S204: If the instance segmentation quality does not meet the set quality condition, a pre-trained semantic segmentation model is used to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the entity object area.

[0092] In this embodiment, the execution subject of the above method is an image processing device, and the image processing device may be a smart terminal or a server.

[0093] When the image processing device is a smart terminal, the smart terminal includes but is not limited to any of the following: a personal computer, a tablet computer, a laptop computer, a mobile phone, a smart bracelet, a smart TV, a smart watch, and a smart speaker.

[0094] When the image processing device is a server, this embodiment does not limit the implementation form of the server, and the server can be a conventional server, a cloud server, a cloud host, a virtual center, etc. The server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture.

[0095] In this embodiment, the image processing device acquires a target image; the image processing device performs a feature extraction operation on the target image to obtain a feature map of the target image's target object region; the image processing device determines the instance segmentation quality based on the feature map of the target object region. When the instance segmentation quality does not meet the set quality condition, the image processing device uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the target object region. When the segmentation effect of the target image using the instance segmentation method is poor, the target image is semantically segmented to improve the segmentation effect of the target image's target object.

[0096] In the above embodiment, after the instance segmentation device acquires the target image, it performs a feature extraction operation on the target image to obtain a feature map of the entity object area contained in the target image. One achievable method is that the instance segmentation device performs a feature extraction operation on the target image to obtain a shared feature map; the instance segmentation device performs a box selection operation on the shared feature map to obtain a first candidate box corresponding to the entity object area; the instance segmentation device selects a second candidate box that meets the set conditions from the first candidate box; the instance segmentation device performs a feature aggregation operation on the second candidate box to obtain a feature map of the entity object area contained in the target image. The embodiment of the present application performs a box selection on the shared feature map to obtain a first candidate box corresponding to the entity object area, and the first candidate box is screened, and a feature aggregation operation is performed on the screened second candidate box, thereby reducing the amount of calculation and improving the image processing speed.

[0097] For example, the instance segmentation device uses the BackBone model to perform feature extraction operations on the target image to obtain a shared feature map; uses the RPN network to perform a box selection operation on the shared feature map to obtain a first candidate box corresponding to the entity object area; selects a second candidate box that meets the set conditions from the first candidate box; uses the RoiAlign module to perform a feature aggregation operation on the second candidate box to obtain a feature map of the entity object area contained in the target image.

[0098] In an optional embodiment, the instance segmentation device selects a second candidate box that meets the set conditions from the first candidate box. One achievable method is to calculate the probability value that the first candidate box is the entity object area based on the first candidate box and the shared feature map; and select a candidate box whose probability value is greater than a set probability threshold from the first candidate box as the second candidate box. It should be noted that the present application does not limit the setting of the probability threshold, and the setting of the probability threshold can be adjusted according to the actual situation. For example, the setting of the probability threshold can be any one of 0.9, 0.95 and 0.92.

[0099] For example, the instance segmentation device calculates the probability value that the first candidate box is the entity object area based on the first candidate box and the shared feature map; and selects a candidate box whose probability value is greater than a set probability threshold of 0.9 from the first candidate box as the second candidate box.

[0100] In the above embodiment, the instance segmentation device determines the instance segmentation quality based on the feature map of the entity object area, including but not limited to the following determination methods:

[0101] Determination method 1: Input the feature map of the entity object area into a pre-trained segmentation quality prediction model to obtain the instance segmentation quality.

[0102] Determination method 2: Based on the feature map of the entity object area, the instance segmentation quality is calculated using the instance segmentation quality algorithm.

[0103] In the above determination method 1, before using the instance segmentation quality model, it is first necessary to train the instance segmentation quality model. The following is an explanation of the training process of the instance segmentation quality model: First, a large number of image samples are collected to cover as many different entity objects that may be encountered in various image target detection scenarios as possible to improve the sample coverage. Then, the entity objects on the image samples can be annotated to obtain the actual distribution of the entity objects on the target image. Then, the annotated image samples can be subjected to the entity object region feature extraction operation to obtain the feature map of the entity object region contained in the target image, and the feature map of the entity object region is input into a neural network model. Inside the neural network model, the training samples can be subjected to feature extraction and calculation operations according to the model parameters, and the instance segmentation results are output by the output layer of the neural network model. Then, the loss function layer of the neural network model can calculate the loss function according to the difference between the instance segmentation result output by the output layer and the real entity object on the image sample. If the loss function does not meet the set requirements, the model parameters can be adjusted and the iterative training continues. When the loss function of the neural network model meets the set requirements, a trained segmentation quality prediction model can be obtained.

[0104] After inputting the feature map of the entity object region into the segmentation quality prediction model, the overlap between the predicted mask and the real mask of the entity object region is obtained through the convolution layer and the fully connected layer of the segmentation quality prediction model; the instance segmentation quality is determined according to the true confidence predicted by the classification subnet of the segmentation quality prediction model and the overlap between the predicted mask and the real mask. For example, inputting the feature map of the entity object region into the segmentation quality prediction model, the intersection-over-union ratio between the predicted mask and the real mask of the entity object region is obtained through the convolution layer and the fully connected layer of the segmentation quality prediction model; the instance segmentation quality is obtained by multiplying the true confidence predicted by the classification subnet of the segmentation quality prediction model with the intersection-over-union ratio between the predicted mask and the real mask.

[0105] In the second determination method, the Mask Iou algorithm is used to calculate the instance segmentation quality of the target image based on the feature map of the entity object area. When the instance segmentation method is adopted, the instance segmentation quality is estimated in advance to decide whether to switch to semantic segmentation to improve the instance segmentation effect.

[0106] In the above embodiment, when the instance segmentation quality does not meet the set quality condition, the instance segmentation device uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the entity object area. One achievable method is to determine whether the instance segmentation quality meets the set quality condition based on the instance segmentation quality; if the instance segmentation quality does not meet the set quality condition, the target image is input into the pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area.

[0107] It should be noted that, in the embodiment of the present disclosure, when the instance segmentation quality reaches the set quality condition, the instance segmentation method is adopted to obtain the instance segmentation result. When the instance segmentation quality does not reach the set quality condition, the semantic segmentation method is adopted to obtain the semantic segmentation result. Obviously, other segmentation methods can also be used for the target image to ensure a better image segmentation result.

[0108] In some embodiments of the present disclosure, it is determined whether the segmentation quality meets the set quality conditions based on the instance segmentation quality. One achievable method is that if the instance segmentation quality is greater than or equal to the set quality threshold, the segmentation quality meets the set quality conditions; if the instance segmentation quality is less than the set quality threshold, the segmentation quality does not meet the set quality conditions. For example, the instance segmentation device obtains an instance segmentation quality of 95 points for the target image, and the instance segmentation quality of 95 points is greater than the set quality threshold of 90 points, then an instance segmentation classification result that does not require semantic segmentation of the target image is generated. It should be noted that the present application does not limit the set quality threshold, and the set quality threshold can be adjusted according to actual conditions.

[0109] In some embodiments of the present disclosure, after the instance segmentation device performs an entity object region feature extraction operation on the target image to obtain a feature map of the entity object region contained in the target image, the feature map of the entity object region is input into a pre-trained instance segmentation model to obtain an instance segmentation result corresponding to the entity object region, and performs boundary optimization processing on the instance segmentation result to obtain an instance segmentation effect after boundary optimization.

[0110] Among them, the instance segmentation result is subjected to boundary optimization processing to obtain the instance segmentation result after boundary optimization. The instance mask corresponding to the instance segmentation result is upsampled to obtain the feature probability values ​​of multiple feature subgraphs corresponding to the instance mask; the target feature subgraph whose feature probability value belongs to the set probability range is selected from the multiple feature subgraphs corresponding to the instance mask; the features on the target feature subgraph are linearly transformed to obtain the optimized feature probability value; the feature probability value of the target feature subgraph is replaced with the optimized feature probability value to obtain the instance segmentation result after boundary optimization. For example, the instance mask corresponding to the instance segmentation result is upsampled twice by bilinear interpolation to obtain the feature probability values ​​of multiple feature subgraphs corresponding to the instance mask; the target feature subgraph whose feature probability value belongs to the set probability range is selected from the multiple feature subgraphs corresponding to the instance mask; the features on the target feature subgraph are linearly transformed to obtain the optimized feature probability value; the feature probability value of the target feature subgraph is replaced with the optimized feature probability value to obtain the instance segmentation result after boundary optimization.

[0111] Optionally, the instance segmentation result is input into a pre-trained boundary optimization model to obtain an instance segmentation result after boundary optimization. The embodiment of the present application performs boundary optimization processing on the instance segmentation result of instance segmentation again to obtain an instance segmentation result after boundary optimization, thereby achieving the effect of fine-tuning the boundary and increasing the effect of instance segmentation. For example, the instance segmentation result is input into a pre-trained pointrend model to obtain an instance segmentation result after boundary optimization, and the pointrend model is used to gradually upsample the image to find the uncertain position for segmentation prediction, thereby achieving the effect of fine-tuning the boundary.

[0112] Figure 3 A framework diagram of a MaskRcnn model 30 provided for an exemplary embodiment of the present disclosure. Figure 3 The image processing method of the embodiment of the present application is described. Figure 3 As shown, the MaskRcnn model 30 includes a feature extraction module 31, a candidate box extraction module 32, a feature aggregation module 33, a classification module 34, a regression module 35, an instance segmentation module 36, a boundary optimization module 37 and a segmentation quality prediction module 38.

[0113] The feature extraction module 31 is used to perform a feature extraction operation on the target image input to the MaskRcnn model 30 to obtain a shared feature map;

[0114] A candidate box extraction module 32 is used to perform a box selection operation on the shared feature map for the physical object area to obtain a candidate box corresponding to the physical object area;

[0115] A feature aggregation module 33 is used to perform a feature aggregation operation on the image features in the candidate frame to obtain a feature map of the entity object area contained in the target image;

[0116] A classification module 34, used to classify the entity objects contained in the target image according to the feature map of the entity object area contained in the target image, and obtain a classification result;

[0117] A regression module 35 is used to adjust the position of the candidate box corresponding to the entity object area to obtain a candidate box corresponding to each entity object;

[0118] An instance segmentation module 36 is used to perform instance segmentation on the entity objects contained in the target image to obtain an instance mask corresponding to each entity object;

[0119] A boundary optimization module 37 is used to perform boundary optimization processing on the instance mask corresponding to each entity object to obtain an instance segmentation result after boundary optimization;

[0120] The segmentation quality prediction module 38 is used to predict the instance segmentation quality according to the feature map of the entity object area to obtain the instance segmentation quality.

[0121] In one feasible manner, the feature extraction module 31 is a BackBone model, the candidate box extraction module 32 is an RPN network; the feature aggregation module 33 is a RoiAlign module, the boundary optimization module 37 is a pointrend model, and the segmentation quality prediction module 38 is a Mask Iou algorithm module.

[0122] For example, the BackBone model in the MaskRcnn model 30 performs a feature extraction operation on the input target image to obtain a shared feature map; the RPN network performs a box selection operation on the shared feature map to obtain a candidate box corresponding to the entity object area; the RoiAlign module performs a feature aggregation operation on the image features in the candidate box to obtain a feature map of the entity object area contained in the target image; the classification module 34 is used to classify the entity objects contained in the target image according to the feature map of the entity object area contained in the target image to obtain a classification result; the regression module 35 is used to adjust the position of the candidate box corresponding to the entity object area to obtain a candidate box corresponding to each entity object; the instance segmentation module 36 is used to perform instance segmentation on the entity objects contained in the target image to obtain an instance mask corresponding to each entity object; the pointrend model is used to perform boundary optimization processing on the instance mask corresponding to each entity object to obtain an instance segmentation result after boundary optimization, thereby improving the instance segmentation result; The iou algorithm module predicts the instance segmentation quality based on the feature map of the entity object area to obtain the instance segmentation quality; based on the instance segmentation quality, it generates the instance segmentation classification result; if the instance segmentation classification result requires semantic segmentation of the target image, the target image is input into a pre-trained semantic segmentation model to obtain the semantic segmentation result corresponding to the entity object area.

[0123] Based on the description of the above embodiments, Figure 4 FIG. 1 is a flow chart of another image processing method provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the method includes:

[0124] S401: Acquire a target image;

[0125] S402: performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image;

[0126] S403: Determine instance segmentation quality according to the feature map of the entity object region;

[0127] S404: Generate instance segmentation classification results according to the instance segmentation quality;

[0128] S405: If the instance segmentation classification result indicates that semantic segmentation is required for the target image, the target image is input into a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area;

[0129] S406: Output semantic segmentation results.

[0130] In this embodiment, the execution subject of the above method is an image processing device, and the image processing device may be a smart terminal or a server.

[0131] When the image processing device is a smart terminal, the smart terminal includes but is not limited to any of the following: a personal computer, a tablet computer, a laptop computer, a mobile phone, a smart bracelet, a smart TV, a smart watch, and a smart speaker.

[0132] When the image processing device is a server, this embodiment does not limit the implementation form of the server, and the server can be a conventional server, a cloud server, a cloud host, a virtual center, etc. The server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture.

[0133] In this embodiment, the implementation of each step of the above-mentioned image processing method can refer to the description of the above-mentioned embodiments, and will not be repeated in this embodiment. Accordingly, this embodiment can also achieve the effective effects of the above-mentioned embodiments.

[0134] Based on the description of the above embodiments, Figure 5 FIG. 1 is a flow chart of another image processing method provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the method includes:

[0135] S501: Acquire a target image;

[0136] S502: performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image;

[0137] S503: Determine instance segmentation quality according to the feature map of the entity object area;

[0138] S504: Generate instance segmentation classification results according to the instance segmentation quality;

[0139] S505: If the instance segmentation classification result indicates that semantic segmentation is required for the target image, the target image is input into a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area;

[0140] S506: Perform image synthesis using the semantic segmentation result.

[0141] In this embodiment, the execution subject of the above method is an image processing device, and the image processing device may be a smart terminal or a server.

[0142] When the image processing device is a smart terminal, the smart terminal includes but is not limited to any of the following: a personal computer, a tablet computer, a laptop computer, a mobile phone, a smart bracelet, a smart TV, a smart watch, and a smart speaker.

[0143] When the image processing device is a server, this embodiment does not limit the implementation form of the server, and the server can be a conventional server, a cloud server, a cloud host, a virtual center, etc. The server device mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture.

[0144] In this embodiment, the implementation of steps S501 to S505 in the above image processing method can refer to the description of the above embodiments, which will not be repeated here.

[0145] In the above embodiment, the image processing device obtains the semantic segmentation result corresponding to the entity object region, that is, obtains the instance mask corresponding to each entity object contained in the target image. The image processing device extracts the regional image of each entity object from the target image and synthesizes the regional image of each entity object with other images.

[0146] In the above-mentioned method embodiment of the present disclosure, an image processing device obtains a target image, performs a physical object region feature extraction operation on the target image, and obtains a feature map of the physical object region contained in the target image; the image processing device determines the instance segmentation quality based on the feature map of the physical object region; when the instance segmentation quality does not meet the set quality conditions, the image processing device uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the physical object region, and when the segmentation effect of the target image using the instance segmentation method is poor, the target image is semantically segmented to improve the segmentation effect of the physical object in the target image.

[0147] Figure 6a FIG. 6 is a structural block diagram of an image processing device 60 provided by an exemplary embodiment of the present disclosure. Figure 6a As shown, the image processing device 60 includes: an acquisition module 61, a feature extraction module 62, a determination module 63 and a semantic segmentation module 64.

[0148] Wherein, the acquisition module 61 is used to acquire the target image;

[0149] A feature extraction module 62 is used to perform a feature extraction operation on a physical object region of a target image to obtain a feature map of the physical object region contained in the target image;

[0150] A determination module 63, configured to determine the instance segmentation quality according to the feature map of the entity object region;

[0151] The semantic segmentation module 64 performs semantic segmentation on the target image using a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area if the instance segmentation quality does not meet the set quality condition.

[0152] Alternatively, if Figure 6b As shown, the feature extraction module 62 includes: a feature extraction submodule 621, a frame selection submodule 622, a screening submodule 623 and a feature aggregation submodule 624;

[0153] The feature extraction submodule 621 is used to perform a feature extraction operation on the target image to obtain a shared feature map;

[0154] A box selection submodule 622 is used to perform a box selection operation on the shared feature map for the physical object area to obtain a first candidate box corresponding to the physical object area;

[0155] A screening submodule 623 is used to screen out second candidate boxes that meet a set condition from the first candidate boxes;

[0156] The feature aggregation submodule 624 is used to perform a feature aggregation operation on the second candidate box to obtain a feature map of the entity object area contained in the target image.

[0157] Optionally, when filtering out a second candidate box that meets the set conditions from the first candidate box, the screening submodule 623 is used to calculate the probability value of the first candidate box being the entity object area based on the first candidate box and the shared feature map; and select a candidate box whose probability value is greater than a set probability threshold from the first candidate box as the second candidate box.

[0158] Optionally, when determining the instance segmentation quality based on the feature map of the entity object area, the determination module 63 is used to input the feature map of the entity object area into the segmentation quality prediction model, and obtain the overlap between the predicted mask and the true mask of the entity object area through the convolution layer and the fully connected layer of the segmentation quality prediction model; determine the instance segmentation quality based on the true confidence predicted by the classification subnet of the segmentation quality prediction model and the overlap between the predicted mask and the true mask.

[0159] Alternatively, if Figure 6b As shown, the feature extraction module 62 further includes: an instance segmentation submodule 625 and a boundary optimization submodule 626;

[0160] An instance segmentation submodule 625 is used to input the feature map of the entity object region into a pre-trained instance segmentation model to obtain an instance segmentation result corresponding to the entity object region;

[0161] The boundary optimization submodule 626 is used to perform boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization.

[0162] Optionally, when the boundary optimization submodule 626 performs boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization, it is used to upsample the instance mask corresponding to the instance segmentation result to obtain feature probability values ​​of multiple feature subgraphs corresponding to the instance mask; select a target feature subgraph whose feature probability value belongs to a set probability range from the multiple feature subgraphs corresponding to the instance mask; perform a linear transformation on the features on the target feature subgraph to obtain the optimized feature probability value; replace the feature probability value of the target feature subgraph with the optimized feature probability value to obtain the instance segmentation result after boundary optimization.

[0163] Optionally, before using a pre-trained semantic segmentation model to perform semantic segmentation on the target image and obtaining a semantic segmentation result corresponding to the entity object area, the semantic segmentation module 64 is also used to: determine whether the instance segmentation quality meets the set quality conditions based on the instance segmentation quality; if the instance segmentation quality does not meet the set quality conditions, input the target image into the pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area.

[0164] Optionally, when the semantic segmentation module 64 determines whether the segmentation quality meets the set quality conditions based on the instance segmentation quality, it is used to: if the instance segmentation quality is greater than or equal to the set quality threshold, the segmentation quality meets the set quality conditions; if the instance segmentation quality is less than the set quality threshold, the segmentation quality does not meet the set quality conditions.

[0165] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0166] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0167] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 707 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0168] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 707, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 707, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a text processing method. For example, in some embodiments, the text processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 707. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the text processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the text processing method in any other appropriate manner (e.g., by means of firmware).

[0170] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0172] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0174] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0175] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0176] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores the computer program and the computer program is executed by one or more processors, the one or more processors execute Figure 2 Each step in the method embodiment.

[0177] Accordingly, the embodiment of the present application also provides a computer program product, the computer program product includes a computer program / instruction, and the computer program / instruction is executed by a processor. Figure 2 Each step in the method embodiment.

[0178] In the above-mentioned device, equipment, storage medium and computer program product embodiments of the present disclosure, an image processing device obtains a target image, performs a physical object region feature extraction operation on the target image, and obtains a feature map of the physical object region contained in the target image; the image processing device determines the instance segmentation quality based on the feature map of the physical object region; when the instance segmentation quality does not meet the set quality conditions, the image processing device uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the physical object region; when the segmentation effect of the target image using the instance segmentation method is poor, the target image is semantically segmented to improve the segmentation effect of the physical object in the target image.

[0179] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0180] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, characterized in that: include: Get the target image; Performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image; Determining instance segmentation quality according to the feature map of the physical object region; If the instance segmentation quality does not meet the set quality condition, a pre-trained semantic segmentation model is used to perform semantic segmentation on the target image to obtain a semantic segmentation result corresponding to the entity object area.

2. The method according to claim 1, characterized in that The step of performing a physical object region feature extraction operation on the target image to obtain a feature map of the physical object region contained in the target image includes: Performing a feature extraction operation on the target image to obtain a shared feature map; Performing a box selection operation on the shared feature map for the physical object region to obtain a first candidate box corresponding to the physical object region; Filtering out a second candidate frame that meets a set condition from the first candidate frame; A feature aggregation operation is performed on the second candidate box to obtain a feature map of the entity object area contained in the target image.

3. The method according to claim 2, characterized in that The step of selecting a second candidate frame that meets a set condition from the first candidate frame includes: Calculating, according to the first candidate box and the shared feature map, a probability value that the first candidate box is the physical object area; A candidate box whose probability value is greater than a set probability threshold is selected from the first candidate boxes as the second candidate box.

4. The method according to claim 1, characterized in that: The determining the instance segmentation quality according to the feature map of the entity object region includes: Inputting the feature map of the entity object region into a segmentation quality prediction model, and obtaining the overlap between the predicted mask and the real mask of the entity object region through the convolution layer and the fully connected layer of the segmentation quality prediction model; The instance segmentation quality is determined according to the true confidence predicted by the classification subnet of the segmentation quality prediction model and the overlap between the predicted mask and the true mask.

5. The method according to claim 1, characterized in that After performing the entity object region feature extraction operation on the target image to obtain a feature map of the entity object region contained in the target image, the method further includes: Inputting the feature map of the entity object region into a pre-trained instance segmentation model to obtain an instance segmentation result corresponding to the entity object region; The instance segmentation result is subjected to boundary optimization processing to obtain an instance segmentation result after boundary optimization.

6. The method according to claim 5, characterized in that The performing boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization includes: Upsampling the instance mask corresponding to the instance segmentation result to obtain feature probability values ​​of multiple feature sub-graphs corresponding to the instance mask; Selecting a target feature subgraph whose feature probability value falls within a set probability range from a plurality of feature subgraphs corresponding to the instance mask; Performing a linear transformation on the features on the target feature subgraph to obtain an optimized feature probability value; The optimized feature probability values ​​are used to replace the feature probability values ​​of the target feature subgraph to obtain an instance segmentation result after boundary optimization.

7. The method according to claim 1, characterized in that Before performing semantic segmentation on the target image using the pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area, the method further includes: Determining, according to the instance segmentation quality, whether the instance segmentation quality meets a set quality condition; If the instance segmentation quality does not meet the set quality condition, the target image is input into a pre-trained semantic segmentation model to obtain a semantic segmentation result corresponding to the entity object area.

8. The method according to claim 7, characterized in that The determining, according to the instance segmentation quality, whether the segmentation quality meets a set quality condition includes: If the instance segmentation quality is greater than or equal to the set quality threshold, the segmentation quality meets the set quality condition; If the instance segmentation quality is less than the set quality threshold, the segmentation quality does not meet the set quality condition.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring a target image; A feature extraction module, used to perform a feature extraction operation on a physical object region of the target image to obtain a feature map of the physical object region contained in the target image; A determination module, configured to determine instance segmentation quality according to a feature map of the entity object region; The semantic segmentation module uses a pre-trained semantic segmentation model to perform semantic segmentation on the target image if the instance segmentation quality does not meet the set quality conditions, and obtains a semantic segmentation result corresponding to the entity object area.

10. The image processing device according to claim 9, characterized in that: The feature extraction module includes: a feature extraction submodule, a frame selection submodule, a screening submodule and a feature aggregation submodule; The feature extraction submodule is used to perform a feature extraction operation on the target image to obtain a shared feature map; The box selection submodule is used to perform a box selection operation on the shared feature map for the physical object area to obtain a first candidate box corresponding to the physical object area; The screening submodule is used to screen out second candidate boxes that meet set conditions from the first candidate boxes; The feature aggregation submodule is used to perform a feature aggregation operation on the second candidate box to obtain a feature map of the entity object area contained in the target image.

11. The image processing device according to claim 10, characterized in that: When the screening submodule screens out the second candidate box that meets the set conditions from the first candidate box, the screening submodule is used to calculate the probability value of the first candidate box being the entity object area based on the first candidate box and the shared feature map; and select a candidate box whose probability value is greater than a set probability threshold from the first candidate box as the second candidate box.

12. The image processing device according to claim 9, characterized in that: When determining the instance segmentation quality based on the feature map of the entity object area, the determination module is used to input the feature map of the entity object area into the segmentation quality prediction model, obtain the overlap between the predicted mask and the real mask of the entity object area through the convolution layer and the fully connected layer of the segmentation quality prediction model; and determine the instance segmentation quality based on the real confidence predicted by the classification subnet of the segmentation quality prediction model and the overlap between the predicted mask and the real mask.

13. The image processing device according to claim 9, characterized in that: The feature extraction module includes: an instance segmentation submodule and a boundary optimization submodule; The instance segmentation submodule is used to input the feature map of the entity object area into a pre-trained instance segmentation model to obtain an instance segmentation result corresponding to the entity object area; The boundary optimization module is used to perform boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization.

14. The image processing device according to claim 13, characterized in that: When the boundary optimization module performs boundary optimization processing on the instance segmentation result to obtain the instance segmentation result after boundary optimization, it is used to upsample the instance mask corresponding to the instance segmentation result to obtain feature probability values ​​of multiple feature subgraphs corresponding to the instance mask; select a target feature subgraph whose feature probability value falls within a set probability range from the multiple feature subgraphs corresponding to the instance mask; perform linear transformation on the features on the target feature subgraph to obtain the optimized feature probability value; use the optimized feature probability value to replace the feature probability value of the target feature subgraph to obtain the instance segmentation result after boundary optimization.

15. The image processing device according to claim 9, characterized in that: Before using the pre-trained semantic segmentation model to perform semantic segmentation on the target image to obtain the semantic segmentation result corresponding to the entity object area, it is also used to: determine whether the instance segmentation quality meets the set quality conditions based on the instance segmentation quality; if the instance segmentation quality does not meet the set quality conditions, input the target image into the pre-trained semantic segmentation model to obtain the semantic segmentation result corresponding to the entity object area.

16. The image processing device according to claim 15, characterized in that: When determining whether the segmentation quality meets a set quality condition according to the instance segmentation quality, the semantic segmentation module is used to: If the instance segmentation quality is greater than or equal to the set quality threshold, the segmentation quality meets the set quality condition; If the instance segmentation quality is less than the set quality threshold, the segmentation quality does not meet the set quality condition.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement each step in the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute each step in the method according to any one of claims 1-8.

Images