An image processing method, device and readable storage medium

Abstract

The application provides an image processing method and device and a readable storage medium, and relates to the technical field of communication. The method comprises the following steps: segmenting an original image to obtain a plurality of subgraph masks, each of the plurality of subgraph masks corresponding to a target object, and the target objects corresponding to each of the subgraph masks being different from each other; and fusing the plurality of subgraph masks to determine a panoramic segmentation mask, the panoramic segmentation mask having no regional overlap or gap. The scheme of the application solves the problem of errors in current segmentation data labeling.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to an image processing method, device, and readable storage medium. Background Technology

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses various technologies such as multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing to simulate and apply computer-generated text, images, 3D models, music, videos, and other virtual information to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0003] However, the panoptic segmentation task for AR videos requires that each pixel in the image must be assigned a semantic label and an instance ID. The semantic label refers to the object's category, while the instance ID corresponds to a different number for objects of the same category.

[0004] However, current segmentation data annotation requires manual marking of multiple points on the boundary of the target object to form an irregular polygonal boundary, and then the region is segmented and labeled based on this boundary, which results in a large annotation error. Summary of the Invention

[0005] The purpose of this invention is to provide an image processing method, device, and readable storage medium to effectively solve the problem of errors in current segmentation data annotation.

[0006] To achieve the above objectives, embodiments of the present invention provide an image processing method, comprising:

[0007] The original image is segmented to obtain multiple sub-image masks. Each sub-image mask corresponds to a target object, and the target objects corresponding to each sub-image mask are different.

[0008] The multiple sub-image masks are fused together to determine a panoramic segmentation mask, which has no overlapping or gaps.

[0009] Optionally, the segmentation of the original image to obtain multiple sub-image masks includes:

[0010] Obtain the location information of the user operation corresponding to the original image;

[0011] The original image and the location information are input into the segmentation model to obtain the multiple sub-image masks output by the segmentation model;

[0012] The segmentation model is used to segment the original image one by one based on the target objects included in the original image.

[0013] Optionally, the method further includes:

[0014] Based on the sample images, determine the location information of the user operations used in the training;

[0015] The segmentation model is trained based on the sample image, the sub-image mask of the sample image, and the location information.

[0016] Optionally, determining the location information of the user operations used for training based on the sample images includes:

[0017] Erosion is performed on the predicted error region of the sample image to obtain the target region;

[0018] Select a target location within the target area;

[0019] The location information of the target location is used as the location information of user operations for training.

[0020] Optionally, the location information includes location coordinate information and location label information, wherein the location label information is used to characterize whether the current location is foreground.

[0021] Optionally, the step of inputting the original image and the location information into the segmentation model to obtain the plurality of sub-image masks output by the segmentation model includes:

[0022] During the segmentation process of the segmentation model for a target object, the model is processed in conjunction with a reference subgraph mask of the target object. The reference subgraph mask can be: an all-zero mask, a subgraph mask obtained from the previous segmentation, or a predicted subgraph mask.

[0023] Optionally, fusing the multiple sub-image masks to determine the panoramic segmentation mask includes:

[0024] Based on the multiple sub-image masks and the original image, multiple prediction masks and classification prediction results are obtained;

[0025] The k valid targets in the classification prediction results are added to the multiple prediction masks to obtain the panoramic segmentation mask;

[0026] Where k equals the number of target objects in the original image.

[0027] Optionally, the step of obtaining multiple prediction masks and classification prediction results based on the multiple sub-image masks and the original image includes:

[0028] Obtain the feature map of the original image;

[0029] The classification prediction result is determined based on the feature map and the multiple sub-map masks;

[0030] Based on the feature map, determine the mask features of the original image;

[0031] Multiple prediction masks are determined based on the mask features of the original image and the classification features corresponding to the multiple sub-image masks.

[0032] Optionally, in determining the plurality of prediction masks, the prediction loss includes the cross-entropy loss of the mask and the cross-entropy loss of the non-deterministic region in the mask, wherein the non-deterministic region is the region outside the region where the target object is located in the current mask.

[0033] To achieve the above objectives, embodiments of the present invention provide an image processing apparatus, comprising:

[0034] The first processing module is used to segment the original image to obtain multiple sub-image masks. Each sub-image mask corresponds to a target object, and the target objects corresponding to each sub-image mask are different from each other.

[0035] The second processing module is used to fuse the multiple sub-image masks to determine a panoramic segmentation mask, wherein the panoramic segmentation mask has no overlapping areas or gaps.

[0036] To achieve the above objectives, embodiments of the present invention provide an image processing device, including a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the image processing method described above.

[0037] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the steps in the image processing method described above.

[0038] The beneficial effects of the above-mentioned technical solution of the present invention are as follows:

[0039] The method of this invention, after segmenting the original image into sub-image masks corresponding to different target objects, further eliminates regional overlap or gaps caused by segmentation through the fusion processing of multiple sub-image masks, and obtains a panoramic segmentation mask without regional overlap or gaps, thereby achieving effective image annotation and reducing annotation errors. Attached Figure Description

[0040] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the C2M algorithm.

[0042] Figure 3 This is a schematic diagram of the fusion model.

[0043] Figure 4 This is a schematic diagram of the module structure of the image processing device according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention;

[0045] Figure 6 One of the schematic diagrams for obtaining a subgraph mask;

[0046] Figure 7 The second schematic diagram for obtaining the subgraph mask. Detailed Implementation

[0047] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0048] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0049] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0051] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0052] like Figure 1 As shown, an image processing method according to an embodiment of the present invention includes:

[0053] Step 101: Segment the original image to obtain multiple sub-image masks. Each sub-image mask corresponds to a target object, and the target objects corresponding to each sub-image mask are different from each other.

[0054] Here, the target objects are individual targets logically divided according to the target structure in the original image. The mask not only includes the image of the corresponding region, but also the annotation information of each pixel in that region, such as semantic labels and instance identifiers (ids). The semantic labels are used to indicate the category of the target object, and the instance IDs are used to indicate different numbers for corresponding objects of the same category.

[0055] In this step, the original image is segmented based on the target object, which can obtain sub-images Mask corresponding to different target objects, thus achieving the annotation of different target objects.

[0056] Step 102: The multiple sub-image masks are fused together to determine the panoramic segmentation mask, which has no overlapping or gaps.

[0057] In this step, the multiple sub-image masks obtained in step 101 are merged to eliminate overlapping or gaps in the segmented regions, thus determining a panoramic segmentation mask without overlapping or gaps. For example, Figure 6 As shown, the original image contains two target objects, and sub-images Mask1 and Mask2 are obtained through segmentation. However, gaps remain between the sub-images Mask1 and Mask2. The fusion process fills these gaps to obtain a seamless panoramic segmentation Mask. Figure 7 As shown, there are two target objects in the original image, and sub-images Mask1 and Mask2 are obtained through segmentation. However, there is an overlap between the sub-images Mask. At this time, the fusion will merge the overlapping positions to obtain a panoramic segmentation Mask with no overlapping regions.

[0058] Thus, according to steps 101-102 above, the method of this embodiment of the invention, after segmenting the original image into sub-image masks corresponding to different target objects, further eliminates the region overlap or gap caused by segmentation through the fusion processing of multiple sub-image masks, and obtains a panoramic segmentation mask without region overlap or gap, thereby achieving effective image annotation and reducing annotation error.

[0059] Optionally, in this embodiment, segmenting the original image to obtain multiple sub-image masks includes:

[0060] Obtain the location information of the user operation corresponding to the original image;

[0061] The original image and the location information are input into the segmentation model to obtain the multiple sub-image masks output by the segmentation model;

[0062] The segmentation model is used to segment the original image one by one based on the target objects included in the original image.

[0063] Here, the segmentation model is a pre-trained model used to segment the input image (i.e., the original image). This segmentation model can also be understood as an interactive single-object segmentation model, whose input includes the original image and the positional information of user operations, and which uses the interaction of human intervention and deep learning algorithms to achieve high-precision segmentation of a single target object.

[0064] The user operations corresponding to the original image can be operations such as multiple clicks or dragging lines on the original image with the mouse. After being input into the segmentation model, these multiple operations can be used in multiple iterations according to different target objects. Each iteration segments only one target object, and multiple interactive iterations are used to segment multiple target objects within the original image.

[0065] For example, for a target object A, the segmentation model can segment the target object A into sub-image Masks based on the user operations corresponding to the target object A, until the sub-image Masks of the target object A meet the requirements; then, it is determined whether all target objects of the original image have been segmented. If so, the fusion of the multiple sub-image Masks is performed; otherwise, a new round of iteration is performed for the next target object.

[0066] Optionally, in this embodiment, the location information of the user operation is the cumulative location information of all user operations.

[0067] Optionally, in this embodiment, the location information includes location coordinate information and location label information, wherein the location label information is used to characterize whether the current location is foreground.

[0068] For example, a user's mouse click action can provide location information (x, y, is). positive ), where x and y represent the coordinates of the position on the image where the mouse clicks, is positive True indicates that the position is in the foreground, is positive Setting it to False indicates that the location is the background. If the user drags the mouse to draw a line, multiple location information can be obtained based on this operation. This location information is the same as above and will not be repeated here.

[0069] Optionally, in this embodiment, the step of inputting the original image and the location information into the segmentation model to obtain the plurality of sub-image masks output by the segmentation model includes:

[0070] During the segmentation process of the segmentation model for a target object, the model is processed in conjunction with a reference subgraph mask of the target object. The reference subgraph mask can be: an all-zero mask, a subgraph mask obtained from the previous segmentation, or a predicted subgraph mask.

[0071] In other words, after inputting the original image and the location information of user actions into the segmentation model, the model, during the segmentation process for a specific target object, also combines the reference sub-image mask of the target object to determine its sub-image mask. For example, in this iterative matting process for the target object, the original image, the location information of user actions, and a zero-based mask are used as inputs for the first matting step; in the next matting step, the original image, the location information of user actions, and the result of the first matting step (the sub-image mask obtained from the previous segmentation) are used as inputs for the current matting step; ...; until multiple matting steps are performed, a sub-image mask that meets the requirements for the target object is obtained. Alternatively, in this target object matting process, the iterative matting process is simulated first to determine the predicted sub-image mask, which is equivalent to the result of multiple iterative matting steps. Then, the original image, the location information of user actions, and the predicted sub-image mask are directly used to obtain the sub-image mask for the target object. In this case, the obtained sub-image mask meets the requirements, and the number of iterations is reduced.

[0072] Alternatively, in this embodiment, the method may further include:

[0073] Based on the sample images, determine the location information of the user operations used in the training;

[0074] The segmentation model is trained based on the sample image, the sub-image mask of the sample image, and the location information.

[0075] Here, the sample images (ground mask) and sub-masks of the sample images can be obtained from manually annotated panoptic segmentation datasets. Location information is obtained by simulating user actions based on the sample images. In this way, a segmentation model with higher quality output results can be obtained through training.

[0076] It is important to know that during training, the loss function can be calculated using the prediction confidence and focusing parameters (i.e., the hyperparameters of the algorithm) based on the location information of user operations used in training.

[0077] Optionally, determining the location information of the user operations used for training based on the sample images includes:

[0078] Erosion is performed on the predicted error region of the sample image to obtain the target region;

[0079] Select a target location within the target area;

[0080] The location information of the target location is used as the location information of user operations for training.

[0081] Here, the predicted error region is determined based on the segmentation results obtained from manually annotating the sample images. For example, if the segmented target object is incomplete after the user clicks on region 1 of the sample image, then region 1 is the predicted error region. Therefore, the target region obtained after erosion of the predicted error region can better simulate user operations. Specifically, the erosion on the predicted error region is an erosion operation centered on the position of the corresponding user operation with a radius of a preset distance (e.g., 5mm). In the target region after erosion, a target position can be randomly selected as the position of the simulated user operation, that is, the position of the user operation used for training. If the randomly selected position is not in the foreground of the label, the is... positive If True, then Is positive It is False.

[0082] In addition, during the training process, the maximum number of simulated user operations can be limited by setting parameters.

[0083] Optionally, in this embodiment, the segmentation model can adopt the Clicks mode of mouse click + C2M algorithm. Clicks mode is an interactive matting mode where the user provides matting cues to the C2M algorithm using mouse clicks. The C2M algorithm takes the original image (matted image), user operation location information (user click cues), and the previous matting result (zero-mask, the sub-image mask obtained from the previous segmentation) as input, and outputs the matting result through C2M model inference.

[0084] The C2M algorithm structure is as follows: Figure 2 As shown, select frame F i For the original image to be segmented, HRNET-OCR is used for image matting. The user's click location information is used as input to provide additional clues to the algorithm; the mask from the previous matting result is also used as input, providing additional clues. The mask is set to all zeros on the first call. The user click and the previous matting result are convolved and then fed into the HRNET-OCR network. The original HRNET-OCR network structure does not need to be modified; the pre-trained weight parameters of the HRNET-OCR network model can be used to initialize the model during training. Figure 2 The algorithm model shown accelerates the convergence speed of the model and improves the training effect. Specifically, it processes the encoded user mouse clicks (Clicks) and the previous image matting result (the sub-image mask obtained in the previous segmentation process for segmenting a target object) M. i After passing through a 3x3 convolution kernel with a stride of 2, a 64-channel feature map is input, and then this 64-channel feature map is added to the left network.

[0085] Before the user's location information is input into the algorithm model, the location data needs to be converted into a two-channel feature map using a Gaussian blur algorithm with a radius of 3 (not limited to Gaussian blur algorithm). One channel contains is positive The location of True indicates the foreground area; another channel contains is positive The location marked as False indicates the background area.

[0086] The model training process for the Clicks mode of the mouse click + C2M algorithm is as follows:

[0087] 1) Training data: Training is performed on our own manually annotated panoramic segmentation dataset.

[0088] 2) Encoding method for Clicks: User mouse clicks are represented as (x, y, is) positive x and y represent the coordinates of the image where the mouse clicked. positive True indicates that the position is in the foreground, is positive False indicates that the location is background. Before inputting the user's mouse click clues into the algorithm model, the point data needs to be converted into a two-channel feature map using a Gaussian blur algorithm with a radius of 3 (not limited to Gaussian blur algorithm). One channel contains, is positive The location of True indicates the foreground area, which is contained in another channel. positive The location marked as False indicates the background area. It's worth noting that all user mouse click locations are accumulated and then used as model input.

[0089] 3) Simulation of user clicks: During the training process, through N iters The parameter limits the maximum number of simulated user clicks, set to 20. The method for simulating user clicks is as follows: First, perform an erosion operation (erosion radius of 5) on the area where prediction was incorrect. Then, randomly select a location within the eroded area as the simulated user's position. If the randomly selected location is in the foreground of the label, the is... positive Set to True, otherwise the is value at the clicked location. positive It is False.

[0090] 4) Previous matting result: During training and inference, if the current frame does not have a previous matting result, the previous matting result is set to a zero matrix with the same length and width as the input frame (i.e., a zero mask). Otherwise, the previous matting result is used.

[0091] 5) Model Output: Here, output refers to the model output before upsampling, which is a 2-channel feature map. During training, the probability matrix of the foreground and background can be calculated using methods such as softmax on this feature map. During inference, when the resolution of the original image is much larger than the resolution of the input model image, resizing the 2-channel feature map to the original image size and then performing an argmax operation to obtain the final masking result can avoid edge blurring or jagged edges.

[0092] 6) Loss Function: The loss does not fundamentally affect the algorithm's performance. Possible loss functions include, but are not limited to, BCE, FL, and NFL. The following uses NFL as an example to illustrate the calculation of the loss function during algorithm model training: Where, p x，y This represents the prediction confidence at coordinates (x, y). γ1 is called the focusing parameter and is a hyperparameter of the algorithm, set to 2.

[0093] Optionally, in this embodiment, the segmentation model can adopt the Scribble mode of mouse drag-and-drop line drawing + S2M algorithm. In Scribble mode, the user provides matting cues to the S2M algorithm using mouse dragging. The S2M algorithm takes the original image (matted image), user operation position information (overlay of user dragging trajectory cues), and the predicted sub-image mask as input, and outputs the matting result through S2M model inference. This is an interactive matting mode.

[0094] Compared to the Clicks mode of mouse click + C2M algorithm, the line drawing simulation algorithm itself is highly complex and cannot be simulated on GPUs using existing training framework libraries. Furthermore, simulating line drawing using OpenCV can only run on the CPU, and iteratively simulating user line drawing behavior requires multiple data transfers between the CPU and GPU, resulting in excessive time consumption (reaching minutes per iteration). Therefore, the Scribble mode of mouse drag-and-drop line drawing + S2M algorithm decouples the correlation between user line drawing iterations, moving the iterative simulation pre-processing to the training data preparation stage. That is, during the segmentation of a target object, the reference subgraph mask of the target object is used as the predicted subgraph mask.

[0095] Correspondingly, in the Scribble mode using mouse drag-and-drop line drawing and the S2M algorithm,

[0096] 1) Training phase: Training is also performed on our own manually annotated panoptic segmentation dataset. To simulate user operations, erosion processing is applied to the ground truth mask. Of course, dilation, occlusion, and other morphological processing can also be performed. Then, the processing results are compared with the ground truth mask to determine which regions to simulate user drawing lines. OpenCV is then used to simulate user drawing lines in these regions, and finally, the simulated lines are used as the input for training.

[0097] 2) Inference Phase: The user's multiple line drawing results are merged into the "encoded Scribbles" as input for the new iteration. Note that this is not necessary in Clicks mode. Merging multiple line drawing results into the "encoded Scribbles" can simulate multiple iterations to obtain the predicted subgraph mask, which serves as input for the subgraph mask determination process of the target object.

[0098] By decoupling the correlation between user line drawing iterations using the above methods, the simulation of iterations is moved to the training data preparation stage, avoiding the time overhead of simulating user line drawing and frequent data transfer between CPU and GPU during the training process, thus making the line drawing mode technically feasible.

[0099] Furthermore, in this embodiment, the original image is segmented to obtain multiple sub-image masks, which can also be achieved through the Free mode with arbitrary mouse annotation or the Local mode that reduces the size of the cutout area.

[0100] Free mode refers to providing direct indication of whether the current position is foreground or background through user interaction. It is used for fine-tuning and serves as a fallback mode to Clicks and Scribble modes, but is used less frequently. Free mode is implemented by the system front-end and back-end and does not require algorithmic support. It is important to note that the mask obtained from Free mode should be passed to other algorithmic models that require a pre-mask.

[0101] Local mode allows users to directly select a rectangular area with the mouse. Subsequent image cutout operations are performed only within this selected area. After cutout of the selected area, the result is restored to the cutout result of the entire image. Local mode is implemented by the system front-end and back-end and does not require algorithm support. It is important to note that the full-image mask obtained in Local mode should be passed to other algorithm models that require a pre-mask. Another method to implement automatic Local mode is to use a cropped image that expands the current mask by 1.4 times as input instead of using the original input image for inference after more than 3 clicks.

[0102] It should be understood that single-target segmentation results suffer from edge errors (over- or under-segmentation of local regions), leading to overlap or gaps when all segmentation results are directly superimposed. This results in errors such as a single pixel corresponding to multiple instances or not corresponding to any instance, violating the rule that each pixel in a panoramic segmentation image corresponds to and uniquely corresponds to one category and instance. In this application, after segmenting the original image to obtain multiple sub-image masks, a panoramic segmentation mask without regional overlap or gaps can be output through fusion.

[0103] Optionally, fusing the multiple sub-image masks to determine the panoramic segmentation mask includes:

[0104] Based on the multiple sub-image masks and the original image, multiple prediction masks and classification prediction results are obtained;

[0105] The k valid targets in the classification prediction results are added to the multiple prediction masks to obtain the panoramic segmentation mask;

[0106] Where k equals the number of target objects in the original image.

[0107] Specifically, the multiple sub-image masks and the original image are input into the fusion model to obtain the panoramic segmentation mask output by the fusion model;

[0108] The fusion model obtains multiple prediction masks and classification prediction results based on the multiple sub-image masks and the original image. Then, it adds the k valid targets in the classification prediction results to the multiple prediction masks to obtain a panoramic segmentation mask.

[0109] Here, the fusion model is trained with the aim of outputting a panoramic segmentation mask without overlapping or gaps in the region. During the inference phase, the input to the fusion model is the original image and multiple sub-image masks output by the segmentation model; during the training phase, the fusion model is trained using sample images and panoramic segmentation masks of the sample images as the training sample set.

[0110] In this embodiment, such as Figure 3As shown, the fusion model can be configured to include a pixel processing module and a self-attention (Transformer) module. The pixel processing module consists of a feature encoder (using a backbone network) and a feature decoder. The feature encoder receives the image input, uses a ResNet50 Backbone network as the encoding component, and outputs features extracted from the input image (i.e., feature maps). These features are then used as input to the feature decoder. The feature decoder contains three decoding components, which respectively use deconvolution or resize+convolution to double the resolution of the feature maps. The features decoded by each component are input to the Transformer module. The last decoding component uses a resize operation to increase the resolution of the feature maps to the same size as the input image, serving as mask embeddings. The Transformer module consists of three Transformer components. The input to each component is a concatenation of scaled object masks of the same resolution and the features output from the feature decoder. Each Transformer component has N kernels (representing the maximum number of objects it supports segmentation). The output of the last Transformer component is passed through an MLP component (composed of two fully connected layers) to output the classification prediction result (i.e., the predicted classification shown in the figure) and classification features (embeddings).

[0111] Among them, Scaled Object Masks are multiple sub-image masks obtained by segmenting the original image.

[0112] Optionally, the step of obtaining multiple prediction masks and classification prediction results based on the multiple sub-image masks and the original image includes:

[0113] Obtain the feature map of the original image;

[0114] The classification prediction result is determined based on the feature map and the multiple sub-map masks.

[0115] Thus, combined as Figure 3 As shown, the original image is processed by the pixel processing module's feature encoder to output initial features. These initial features are then processed by the three decoding components of the feature decoder to obtain the feature map of the original image, including the feature map decoded by each component. Furthermore, for the Transformer module composed of three Transformer components, the input to each component is a concatenation of scaled object masks of the same resolution and the feature map output by the feature decoder. The output of the last Transformer component is then processed by the MLP component to output the classification prediction result. The MLP component can also output classification embeddings.

[0116] Optionally, the step of obtaining multiple prediction masks and classification prediction results based on the multiple sub-image masks and the original image includes:

[0117] Based on the feature map, determine the mask features of the original image;

[0118] Multiple prediction masks are determined based on the mask features of the original image and the classification features corresponding to the multiple sub-image masks.

[0119] Here, the feature map used to determine the mask features of the original image is the feature map decoded by the last decoding component of the feature decoder. This feature map is resized to the same size as the original image, thus determining the mask embeddings of the original image. Then, based on the mask embeddings of the original image and the classification embeddings corresponding to the multiple sub-image masks, multiple predicted masks can be obtained.

[0120] The prediction mask is obtained by multiplying the mask embeddings and the classification embeddings, and is a combination of N masks (each mask represents a prediction target).

[0121] in this way, Figure 3 The post-processing shown can extract k effective targets from the number of target objects k in the original image based on the predicted N Masks and the classification prediction results. The k effective target Masks are added together to output a panoramic segmentation Mask without regional overlap or gaps.

[0122] Optionally, in this embodiment, during the process of determining the plurality of prediction masks, the prediction loss includes the cross-entropy loss of the mask and the cross-entropy loss of the undetermined region in the mask, wherein the undetermined region is the region outside the region where the target object is located in the current mask.

[0123] For example, predicting loss calculation: Loss(Mask) = CE(Mask) + γCE(Mask*~Mask) mask )

[0124] The first part, CE(Mask), represents the cross-entropy loss of the mask, and the second part, CE(Mask*~Mask), represents the cross-entropy loss of the mask. mask The cross-entropy loss represents the non-deterministic region in the mask. mask This is a 0-1 matrix used to represent the regions in the truth mask that are definitely target objects. 0 indicates regions that are not definitely target objects, and 1 indicates regions that are definitely target objects. mask The calculation method is as follows:

[0125] From the ground truth mask of panoptic segmentation, separate the masks of N target objects, and then use OpenCV to perform random erosion on the masks of the objects, that is, reduce the region representing the object in the mask to generate the mask. i If i belongs to [1,2,3…N], then

[0126] ~Mask mask It's a mask. mask Invert the values; 1 represents a non-defined region, and 0 represents a defined region. Mask*~Mask mask The region involved in loss calculation is limited to a non-deterministic region. γ is a hyperparameter, set to 1.

[0127] Of course, when using a fusion model to determine the panoramic segmentation mask, the prediction loss is used in the process of determining multiple prediction masks.

[0128] In summary, the method of this invention provides an interactive method for annotating panoramic segmentation data.

[0129] like Figure 4 As shown, an image processing apparatus according to an embodiment of the present invention includes:

[0130] The first processing module 410 is used to segment the original image to obtain multiple sub-image masks, each of the multiple sub-image masks corresponding to a target object, and the target objects corresponding to each sub-image mask are different from each other.

[0131] The second processing module 420 is used to fuse the multiple sub-image masks to determine a panoramic segmentation mask, wherein the panoramic segmentation mask has no overlapping areas or gaps.

[0132] This device, after segmenting the original image into sub-image masks corresponding to different target objects, further eliminates regional overlap or gaps caused by segmentation through the fusion processing of multiple sub-image masks, and obtains a panoramic segmentation mask without regional overlap or gaps, thus achieving effective image annotation and reducing annotation errors.

[0133] Optionally, the first processing module includes:

[0134] The first acquisition unit is used to acquire the location information of the user operation corresponding to the original image;

[0135] The first processing unit is used to input the original image and the location information into the segmentation model to obtain the multiple sub-image masks output by the segmentation model;

[0136] The segmentation model is used to segment the original image one by one based on the target objects included in the original image.

[0137] Optionally, the device further includes:

[0138] The determination module is used to determine the location information of user operations used in training based on sample images;

[0139] The training module is used to train the segmentation model based on the sample image, the sub-image mask of the sample image, and the location information.

[0140] Optionally, the determining module is further configured to:

[0141] Erosion is performed on the predicted error region of the sample image to obtain the target region;

[0142] Select a target location within the target area;

[0143] The location information of the target location is used as the location information of user operations for training.

[0144] Optionally, the location information includes location coordinate information and location label information, wherein the location label information is used to characterize whether the current location is foreground.

[0145] Optionally, the first processing unit is further configured to:

[0146] During the segmentation process of the segmentation model for a target object, the model is processed in conjunction with a reference subgraph mask of the target object. The reference subgraph mask can be: an all-zero mask, a subgraph mask obtained from the previous segmentation, or a predicted subgraph mask.

[0147] Optionally, the second processing module includes:

[0148] The second processing unit is used to obtain multiple prediction masks and classification prediction results based on the multiple sub-image masks and the original image;

[0149] The k valid targets in the classification prediction results are added to the multiple prediction masks to obtain the panoramic segmentation mask;

[0150] Where k equals the number of target objects in the original image.

[0151] Optionally, the second processing unit is further configured to:

[0152] Obtain the feature map of the original image;

[0153] The classification prediction result is determined based on the feature map and the multiple sub-map masks;

[0154] Based on the feature map, determine the mask features of the original image;

[0155] Multiple prediction masks are determined based on the mask features of the original image and the classification features corresponding to the multiple sub-image masks.

[0156] Optionally, in determining the plurality of prediction masks, the prediction loss includes the cross-entropy loss of the mask and the cross-entropy loss of the non-deterministic region in the mask, wherein the non-deterministic region is the region outside the region where the target object is located in the current mask.

[0157] It should be noted that this device is an apparatus that applies the above-described image processing method, and the implementation of the above-described method embodiments is applicable to this device and can achieve the same technical effect.

[0158] like Figure 5 As shown, an image processing device according to an embodiment of the present invention includes a transceiver 510, a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500; when the processor 500 executes the program or instructions, it implements the above-described image processing method.

[0159] The transceiver 510 is used to receive and send data under the control of the processor 500.

[0160] Among them, Figure 5 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture can also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium.

[0161] The processor 500 is responsible for managing the bus architecture and general processing, while the memory 520 can store the data used by the processor 500 when performing operations.

[0162] An embodiment of the present invention provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the image processing method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0163] The processor is the processor in the image processing device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0164] It should be further noted that many of the functional components described in this specification are referred to as modules in order to emphasize the independence of their implementation.

[0165] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0166] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0167] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0168] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0169] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: The original image is segmented to obtain multiple sub-image masks. Each sub-image mask corresponds to a target object, and the target objects corresponding to each sub-image mask are different. The multiple sub-image masks are fused to determine a panoramic segmentation mask, which has no overlapping regions or gaps; wherein, the target object is a single target divided according to the target logic in the original image; The step of fusing the multiple sub-image masks to determine the panoramic segmentation mask includes: Based on the multiple sub-image masks and the original image, multiple prediction masks and classification prediction results are obtained; The k valid targets in the classification prediction results are added to the multiple prediction masks to obtain the panoramic segmentation mask; Where k equals the number of target objects in the original image; Based on the multiple sub-image masks and the original image, multiple prediction masks and classification prediction results are obtained, including: Obtain the feature map of the original image; The classification prediction result is determined based on the feature map and the multiple sub-map masks; Based on the feature map, determine the mask features of the original image; Multiple prediction masks are determined based on the mask features of the original image and the classification features corresponding to the multiple sub-image masks.

2. The method according to claim 1, characterized in that, The process of segmenting the original image to obtain multiple sub-image masks includes: Obtain the location information of the user operation corresponding to the original image; The original image and the location information are input into the segmentation model to obtain the multiple sub-image masks output by the segmentation model; The segmentation model is used to segment the original image one by one based on the target objects included in the original image.

3. The method according to claim 2, characterized in that, Also includes: Based on the sample images, determine the location information of the user operations used in the training; The segmentation model is trained based on the sample image, the sub-image mask of the sample image, and the location information.

4. The method according to claim 3, characterized in that, The step of determining the location information of user operations used in training based on sample images includes: Erosion is performed on the predicted error region of the sample image to obtain the target region; Select a target location within the target area; The location information of the target location is used as the location information of user operations for training.

5. The method according to claim 2 or 3, characterized in that, The step of inputting the original image and the location information into the segmentation model to obtain the multiple sub-image masks output by the segmentation model includes: During the segmentation process of the segmentation model for a target object, the model is processed in conjunction with a reference subgraph mask of the target object. The reference subgraph mask can be: an all-zero mask, a subgraph mask obtained from the previous segmentation, or a predicted subgraph mask.

6. The method according to claim 1, characterized in that, In determining the plurality of prediction masks, the prediction loss includes the cross-entropy loss of the mask and the cross-entropy loss of the non-deterministic region in the mask, wherein the non-deterministic region is the region outside the region where the target object is located in the current mask.

7. An image processing apparatus, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the image processing method as described in any one of claims 1-6.

8. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the image processing method as described in any one of claims 1-6.

Images