Image segmentation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image segmentation method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining image data including a to-be-segmented target and a weak tag target frame; and inputting the image data containing the to-be-segmented target and the weak label target frame into a pre-trained image segmentation model, and reprocessing a binary image obtained in the image segmentation model according to the characteristics of the target, the LOSS function in the image segmentation model is obtained through calculation based on a probability graph and the reprocessed binary image; and obtaining an image segmentation result according to the pre-trained image segmentation model. According to the invention, on one hand, effective segmentation under the condition of weak supervision is realized, and on the other hand, the universality of image segmentation is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image segmentation for autonomous driving, and in particular to an image segmentation method, device, electronic device, and storage medium. Background Art

[0002] Image segmentation is to divide an image into several specific regions with unique properties, and assign the same number to the pixels in the same connected region. Image segmentation can be divided into two categories: traditional methods and deep learning methods.

[0003] Traditional image segmentation includes threshold-based segmentation, region-based segmentation, edge-based segmentation, etc. However, based on traditional image segmentation algorithms, the segmentation effect is often poor, and the edge details are often not ideal.

[0004] Segmentation methods that rely on deep learning include: single-mode segmentation and multi-mode segmentation algorithms. For segmentation algorithms based on single-mode deep learning, although the effect is good and can achieve refined segmentation, it is heavily dependent on labeled labels, and does not have the ability to segment unlabeled data, and its application scenarios are severely limited. For multi-mode segmentation large models, not only can accurate segmentation effects be achieved on labeled data, but also segmentation capabilities can be achieved on open datasets, greatly improving the universality of the segmentation model. Therefore, segmentation algorithms based on single-mode deep learning have good effects and can achieve refined segmentation, but they are heavily dependent on labeled labels, and do not have the ability to segment unlabeled data, and their application scenarios are severely limited. Summary of the invention

[0005] The embodiments of the present application provide an image segmentation method, device, electronic device, and storage medium to achieve effective segmentation under weak supervision.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides an image segmentation method, wherein the method comprises:

[0008] Obtain image data containing the target to be segmented and the weakly labeled target frame;

[0009] The image data containing the target to be segmented and the weakly labeled target frame are input into a pre-trained image segmentation model, the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image;

[0010] An image segmentation result is obtained according to the pre-trained image segmentation model.

[0011] In some embodiments, the binary image obtained in the image segmentation model is processed according to the characteristics of the target and includes at least one of the following steps:

[0012] The binary image obtained in the image segmentation model is subjected to morphological processing to remove noise points;

[0013] The outer contour of each connected area is extracted from the binary image obtained in the image segmentation model, and the convex hull area and area of ​​each contour are obtained. The area is compared with a preset threshold. If it is less than the preset threshold, it is set as the background, otherwise it is set as the foreground.

[0014] In some embodiments, the pre-trained image segmentation model includes at least an Image Encoder, a Prompt Encoder, and a Mask Decoder;

[0015] The Image Encoder is used to encode the image data and then extract features;

[0016] The Prompt Encoder is used to encode the weakly labeled target frame information and then extract features;

[0017] The Mask Decoder is used to fuse the features extracted after encoding by the Image Encoder and the Prompt Encoder to obtain a prediction result;

[0018] The Image Encoder includes three branches, namely, an Anchor branch, a Teacher branch, and a Student branch. A weak data enhancement method is used for the Anchor branch and the Teacher branch, and a strong data enhancement method is used for the Student branch.

[0019] In some embodiments, the binary image obtained in the image segmentation model is processed based on the characteristics of the target, including:

[0020] Binary graph based on Anchor branch and the binary graph of the Teacher branch , according to the characteristics of the target itself, the binary graph of the new Teacher branch is obtained by reprocessing And the binary graph of the Anchor branch ,

[0021] The target's own characteristics include at least one of the following: a closed area characteristic of the target, and a noise characteristic of a connected area of ​​the target.

[0022] In some embodiments, the Teacher branch and the Student branch output probability maps respectively and The LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary map, including:

[0023] use and Calculate the focal loss and dice loss of the corresponding branch;

[0024] use Respectively and , Calculate the dice loss of the corresponding branch;

[0025] Based on binary image Get the feature map Each instance feature , based on binary graph Get the feature map Each instance feature , calculate the instance features of the Teacher branch and the Anchor branch and The similarity loss on the feature map It is generated by the Teacher branch after encoding in the Image Encoder. The feature map The Anchor branch uses the original structure of the Image Encoder and first freezes the structure and then encodes it;

[0026] All losses are increased by a specified coefficient and summed up, and the gradient is calculated to determine the LOSS function.

[0027] In some embodiments, the method further comprises:

[0028] In the Image Encoder, LoRA parameters are added to the Teacher branch and the Student branch for training, and the LoRA parameters are shared and the original Image Encoder structure is frozen;

[0029] Input the weakly labeled target frame into the Prompt Encoder for feature extraction;

[0030] The Mask Decoder receives feature information output from the three branches of the Image Encoder and the Prompt Encoder branch respectively to perform feature fusion, and outputs feature maps of the three branches after the feature fusion.

[0031] In some embodiments, obtaining an image segmentation result according to the pre-trained image segmentation model includes:

[0032] The pre-trained image segmentation model merges the trained LoRA parameters into the ImageEncoder;

[0033] Perform preset processing operations on the input image based on the longest and shortest sides, and perform preset adjustment operations on the weak label target box;

[0034] The image is input into the merged Image Encoder network, the weak label target box is input into the Prompt Encoder network structure, and the binary image Mask image is output through the Mask Decoder.

[0035] In some embodiments, after obtaining the image segmentation result according to the pre-trained image segmentation model, the method further includes:

[0036] Obtaining target pixel coordinate information according to the target information in the image segmentation result;

[0037] Convert the target pixel coordinate information into a world coordinate system to obtain the target coordinate information in the world coordinate system;

[0038] The roadside camera external parameters are calibrated or corrected according to the coordinate information of the target in the world coordinate system and the target pixel coordinate information in the roadside camera.

[0039] In a second aspect, an embodiment of the present application further provides an image segmentation device, wherein the device comprises:

[0040] An acquisition module is used to acquire image data containing the target to be segmented and a weakly labeled target frame;

[0041] A processing module, used for inputting the image data containing the target to be segmented and the weakly labeled target frame into a pre-trained image segmentation model, wherein the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image;

[0042] The segmentation module is used to obtain an image segmentation result according to the pre-trained image segmentation model.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0045] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: image data containing the target to be segmented and a weakly labeled target frame are obtained, and then the image data containing the target to be segmented and the weakly labeled target frame are input into a pre-trained image segmentation model. The binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image. The accuracy of the predicted binary image is optimized by reprocessing the binary image. At the same time, through the calculation of the loss loss function, the gradient only updates the LoRA parameters in the Image Encoder. Finally, according to the pre-trained image segmentation model, the image segmentation result is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 Schematic diagram of the process of image segmentation method in the embodiment of the present application;

[0048] Figure 2 This is a schematic diagram of the structure of an image segmentation device in an embodiment of the present application;

[0049] Figure 3 Schematic diagram of the processing flow of the image segmentation model in the image segmentation method in the embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0052] Traditional image segmentation algorithms include threshold-based segmentation, region-based segmentation, edge-based segmentation, etc. Segmentation methods that rely on deep learning include: single-mode segmentation and multi-mode segmentation algorithms. Single-mode segmentation algorithms include U-Net, Mask R-CNN, etc. Multi-mode segmentation algorithms include SAM, weSAM, segGPT, etc.

[0053] The segmentation effect of traditional image segmentation algorithms is often poor, and the edge details are often not ideal. The segmentation algorithm based on single-mode deep learning has better effect and can also achieve refined segmentation, but it is heavily dependent on annotation labels and has no ability to segment unlabeled data, which severely limits its application scenarios.

[0054] In view of the shortcomings of the above-mentioned image segmentation method, a multi-modal segmentation large model is provided in the embodiments of the present application, which can not only achieve accurate segmentation effect on labeled data, but also realize segmentation capability on open set data sets.

[0055] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0056] The present application embodiment provides an image segmentation method, such as Figure 1 As shown, a schematic diagram of the image segmentation method in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S130:

[0057] Step S110, obtaining image data containing the target to be segmented and a weakly labeled target frame.

[0058] In the model prediction stage, image data containing the target to be segmented is obtained. In particular, for weakly supervised segmentation, a weakly labeled target box needs to be input, which is consistent with the input in the model training stage.

[0059] Step S120, input the image data containing the target to be segmented and the weakly labeled target frame into a pre-trained image segmentation model, the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image.

[0060] The pre-trained image segmentation model preferably adopts the weSAM segmentation model. weSAM inherits the overall idea of ​​SAM and mainly has three parts: Image Encoder, Prompt Encoder, and Mask Decoder.

[0061] Furthermore, the binary image obtained in the image segmentation model is processed by the target's own characteristics, mainly for the purpose of further optimizing the accuracy of the predicted binary image, and thus relies on the geometric characteristics of the target itself. For example, when the target is a zebra crossing, it relies on the zebra crossing's own geometric characteristics.

[0062] Furthermore, the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary map. The LOSS function in the image segmentation model is obtained by statistically analyzing multiple losses.

[0063] Specifically, in machine learning or deep learning, when multiple loss functions need to be weighted summed to calculate the total loss and then optimized by gradient descent, this can be achieved by specifying a coefficient for each loss function. This is often used for multi-task learning, where the model needs to learn multiple related or unrelated tasks at the same time. The loss function of each task can have a specific weight so that their contribution to the total loss can be adjusted according to the importance or difficulty of the task.

[0064] It can be understood that the loss function, also known as the cost function, is an important concept in evaluating model performance in machine learning and deep learning. It is used to quantify the difference or error between the model's predicted value and the true value. The main purpose of the loss function is to guide the model's training process: by minimizing the loss function, we can find the optimal value of the model parameters, thereby improving the model's prediction accuracy. During the training process, by constantly adjusting the model parameters (such as the weights in the neural network) to reduce the value of the loss function, the model gradually learns the inherent laws and characteristics of the data, so that it can make more accurate predictions.

[0065] Step S130, obtaining an image segmentation result according to the pre-trained image segmentation model.

[0066] The pre-trained image segmentation model performs corresponding processing operations on the input image data containing the target to be segmented based on the longest side and the shortest side. At the same time, the weak label target box that is input together is adjusted accordingly. Then, in the merged Image Encoder network of the image input in the weSAM segmentation model, the weak label rectangular box is input into the Prompt Encoder network structure, and finally the Mask Decoder outputs the binary image Mask image.

[0067] Through the above steps, the image data containing the target to be segmented and the weakly labeled target frame are input into a pre-trained image segmentation model. And according to the pre-trained image segmentation model, an image segmentation result is obtained. The image segmentation model is optimized for the calculation of the binary image and the loss function, so that the above method can achieve effective segmentation of the image target under weak supervision conditions.

[0068] Different from the related art, the method in the embodiment of the present application has general universality and provides a solution and optimization method for the weakly supervised image segmentation algorithm. Compared with the segmentation algorithm of single-mode deep learning, it does not rely on the annotation label, and because the weakly supervised label does not have the ability to segment the unlabeled data page, it improves the scope of application scenarios.

[0069] In one embodiment of the present application, the binary image obtained in the image segmentation model is processed according to the inherent characteristics of the target and includes at least one of the following: morphological processing is performed on the binary image obtained in the image segmentation model to remove noise points; the outer contour of each connected area of ​​the binary image obtained in the image segmentation model is extracted to obtain the convex hull area and area of ​​each contour, and the area area is compared with a preset threshold value. If it is less than the preset threshold value, it is set as the background, otherwise it is set as the foreground.

[0070] like Figure 2 As shown in the figure, in order to further optimize the accuracy of the predicted binary image, we rely on the geometric characteristics of the target itself. Taking the zebra crossing as an example, the set characteristics mainly include:

[0071] Firstly, each zebra crossing is a single closed area, so there will be no problem of area vacancies in the closed area of ​​the zebra crossing.

[0072] Secondly, considering that in the process of image segmentation, there may be a very small area near the zebra crossing, which is very small compared to the zebra crossing, so there will be no noise points in the very small connected area.

[0073] Based on the above two points, the Teacher binary graph and Anchor binary map to be processed.

[0074] Case 1: Perform morphological opening operation to remove noise points.

[0075] Case 2: Extract the outer contour of each connected region of the binary graph, and obtain the convex hull region and area of ​​each contour. Specifically, in step b1), the convex hull algorithm or connected region search algorithm is used to obtain the convex hull region and area. b2) By comparing the area of ​​the region with the specified threshold, if it is less than the threshold, it is set as the background (non-zebra crossing), otherwise it is set as the foreground (zebra crossing). That is, in the model, to obtain a new Teacher binary graph and Anchor binary map .

[0076] In one embodiment of the present application, the pre-trained image segmentation model includes at least ImageEncoder, Prompt Encoder and Mask Decoder; the Image Encoder is used to extract features after encoding image data; the Prompt Encoder is used to extract features after encoding weak label target box information; the Mask Decoder is used to fuse the features extracted after encoding by the Image Encoder and the Prompt Encoder, and obtain a prediction result; the Image Encoder includes three branches, namely, Anchor branch, Teacher branch, and Student branch, weak data enhancement method is used for the Anchor branch and the Teacher branch, and strong data enhancement method is used for the Student branch.

[0077] Specifically, in the image segmentation model, Image Encoder encodes image information to extract features. Prompt Encoder encodes the rectangular box information of the data to extract features. In addition, points or rough segmentation masks can be used as input. Mask Decoder fuses the features encoded by Image Encoder and Prompt Encoder and gives the prediction results. weSAM has three Image Encoder branches, namely Anchor branch, Teacher branch, and Student branch. It should be noted that during the training process, LoRA parameters are added on the basis of Teacher branch and Student branch and learned and updated, and the original Image Encoder parameters are frozen.

[0078] Preferably, if Figure 2As shown, it also includes two forms of data enhancement for image data, namely weak data enhancement weak aug and strong data enhancement strong aug. Weak data enhancement uses, but is not limited to, horizontal flip and vertical flip operations with a probability of 0.5. Strong data enhancement includes, but is not limited to: data formatting to uint8 type, histogram equalization, image sharpening, random changes in image brightness and contrast, simulated image shadows, etc.

[0079] It can be understood that the above weak data enhancement weak aug and strong data enhancement strong aug are only examples and are not used to limit the protection scope of the embodiments of the present application. Specifically, the Anchor branch and the Teacher branch use weak data enhancement, and the Student branch uses strong data enhancement.

[0080] In one embodiment of the present application, the binary image obtained in the image segmentation model is processed based on the characteristics of the target, including: and the binary graph of the Teacher branch , according to the characteristics of the target itself, the binary graph of the new Teacher branch is obtained by reprocessing And the binary graph of the Anchor branch The target's own characteristics include at least one of the following: the closed area characteristics of the target, the noise characteristics of the connected area of ​​the target.

[0081] According to the closed area characteristics of the target, for example, the closed area of ​​the zebra crossing. And according to the noise characteristics of the connected area of ​​the target, for example, there will be no noise points in the extremely small connected area in the zebra crossing.

[0082] Binary graph based on Anchor branch and the binary graph of the Teacher branch , after reprocessing according to the characteristics of the target itself, a new binary graph of the Teacher branch is obtained And the binary graph of the Anchor branch .

[0083] In one embodiment of the present application, the Teacher branch and the Student branch respectively output probability maps and The LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary map, including: using and Calculate the focal loss and dice loss of the corresponding branch; use Respectively and , Calculate the dice loss of the corresponding branch; based on the binary image Get the feature map Each instance feature , based on binary graph Get the feature map Each instance feature , calculate the instance features of the Teacher branch and the Anchor branch and The similarity loss on the feature map It is generated by the Teacher branch after encoding in the Image Encoder. The feature map The Anchor branch uses the original structure of the Image Encoder and freezes the structure before encoding. All losses are summed up by adding a specified coefficient, and the gradient is calculated to determine the LOSS function.

[0084] (1) Based on the binary image of the Teacher branch and the probability image of the Student branch, calculate the focal loss and diceloss. Figure 2 As shown, L st .

[0085] (2) Based on the binary graph of the Anchor branch And the probability graph of the Student branch and the probability graph of the Teacher branch, calculate the dice loss. Figure 2 As shown, L anchor .

[0086] (3) Based on binary image Get the feature map Each instance feature , based on binary graph Get the feature map Each instance feature That is, before entering the Mask Decoder, the similarity loss of the Teacher branch and the Anchor branch on the instance features is calculated.

[0087] For more information, see Figure 2 ,use and Calculate focal loss and dice loss. Use Respectively and , Calculate dice loss. Based on binary image Get the feature map Each instance feature , based on binary graph Get the feature map Each instance feature . Add the specified coefficient to all losses and calculate the gradient. The gradient only updates the LoRA parameters in the Image Encoder. Based on the back propagation of the LoRA parameters, when updating, only the LoRA parameters are updated, and the frozen parameters are not updated.

[0088] It is understandable that Focal Loss is a loss designed specifically for class imbalance. It was proposed in the paper "Focal Loss for Dense Object Detection" and applied to the training of object detection tasks. However, this is actually an idea that has nothing to do with a specific field and can be applied to any class-imbalanced data. Dice Loss comes from the paper "Dice Loss for Data-imbalanced NLP Tasks".

[0089] In one embodiment of the present application, the method also includes: adding LoRA parameters to the Teacher branch and the Student branch in the Image Encoder for training, while sharing the LoRA parameters and freezing the original Image Encoder structure; inputting the weak label target box into the Prompt Encoder for feature extraction; the Mask Decoder respectively receives the feature information output from the three branches of the Image Encoder and the Prompt Encoder branch for feature fusion, and outputs feature maps of the three branches after the features are fused.

[0090] like Figure 2 As shown, the Image Encoder includes, from top to bottom: Student branch, Teacher branch and Anchor branch. In the embodiment of the present application, LoRA parameters are added to the Student branch and the Teacher branch, but no LoRA parameters are added to the Anchor branch.

[0091] First, LoRA parameters are added to the Teacher branch and Student branch in the Image Encoder network for training, and the LoRA parameters are shared, and the original Image Encoder structure is frozen.

[0092] Then, the Teacher branch generates a feature map after encoding by Image Encoder The Anchor branch uses the original Image Encoder structure, and the structure is frozen. The encoded feature map is .

[0093] Finally, the three branches also share parameters for the frozen Image Encoder.

[0094] Specifically, the target is a zebra crossing as an example for explanation.

[0095] The zebra crossing rectangle frame information is input into the Prompt Encoder structure for feature extraction; the Mask Decoder network receives feature information from the three Image Encoder branches and the Prompt Encoder branch output respectively, and performs feature fusion. After the features are fused, the feature maps of the three branches are output.

[0096] Among them, the Teacher branch and the Student branch output probability map and , the Teacher branch outputs an additional binary map based on the probability map (Foreground with probability value greater than 0.5, otherwise it is background), Anchor branch outputs binary image (Foreground with probability value greater than 0.5, otherwise it is background).

[0097] In one embodiment of the present application, an image segmentation result is obtained according to the pre-trained image segmentation model, including: the pre-trained image segmentation model merges the trained LoRA parameters into the ImageEncoder; performs a preset processing operation on the input image based on the longest side and the shortest side while performing a preset adjustment operation on the weak label target box; inputs the image into the merged Image Encoder network, inputs the weak label target box into the Prompt Encoder network structure, and outputs a binary image Mask image through the Mask Decoder.

[0098] It can be understood that the image segmentation model includes three branches during the training phase: the Anchor branch is used to generate true values, the Teacher branch is used as supervision, and the Student branch is used for learning and is biased towards the true value. During the inference phase, the image segmentation model only includes the Student branch, which processes the input image data containing the target to be segmented and the weakly labeled target box. At the same time, during iteration, it only needs to update the LoRA parameters in the Image Encoder.

[0099] When the image segmentation model is inferencing, the image segmentation model will merge the trained LoRA parameters into the ImageEncoder. The input image includes but is not limited to resizing to the specified size based on the longest side and padding the shortest side. At the same time, the weak label rectangle input together is adjusted accordingly. Finally, in the merged Image Encoder network of the image input, the weak label rectangle is input into the Prompt Encoder network structure, and the binary image Mask image is output through MaskDecoder.

[0100] In one embodiment of the present application, after obtaining the image segmentation result according to the pre-trained image segmentation model, it also includes: obtaining target pixel coordinate information according to the target information in the image segmentation result; converting the target pixel coordinate information to a world coordinate system to obtain the coordinate information of the target in the world coordinate system; calibrating or correcting the external parameters of the roadside camera according to the coordinate information of the target in the world coordinate system and the target pixel coordinate information in the roadside camera.

[0101] When used in specific scenarios, to solve the problem of weak supervision, the image segmentation method can be based on the zebra crossing image data obtained by the vehicle-side or road-side camera for effective segmentation.

[0102] Taking the zebra crossing as an example, in order to achieve accurate calibration of the external parameters of the roadside camera, the zebra crossing information recognized in the vehicle-side RTK information and the vehicle-mounted camera perception data is relied on. The pixel coordinates of the zebra crossing are converted to the world coordinate system by combining VSLAM (Visual Simultaneous Localization and Mapping) technology.

[0103] Combined with the zebra crossing pixel information in the roadside camera and the corresponding zebra crossing world coordinate information provided by the vehicle side, the calibration and correction of the roadside camera external parameters are assisted.

[0104] VSLAM technology is a vision-based SLAM technology that uses cameras to obtain environmental image information, while simultaneously performing robot self-positioning and environmental map construction.

[0105] The present application embodiment also provides an image segmentation device 300, such as Figure 3 As shown, a schematic diagram of the structure of an image segmentation device in an embodiment of the present application is provided, wherein the image segmentation device 300 at least includes: an acquisition module 310, a processing module 320, and a segmentation module 330, wherein:

[0106] In one embodiment of the present application, the acquisition module 310 is specifically used to: acquire image data containing the target to be segmented and a weak label target frame.

[0107] In the model prediction stage, image data containing the target to be segmented is obtained. In particular, for weakly supervised segmentation, a weakly labeled target box needs to be input, which is consistent with the input in the model training stage.

[0108] In one embodiment of the present application, the processing module 320 is specifically used to: input the image data containing the target to be segmented and the weak label target frame into a pre-trained image segmentation model, the binary image obtained in the image segmentation model is reprocessed according to the inherent characteristics of the target, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image.

[0109] The pre-trained image segmentation model preferably adopts the weSAM segmentation model. weSAM inherits the overall idea of ​​SAM and mainly has three parts: Image Encoder, Prompt Encoder, and Mask Decoder.

[0110] Furthermore, the binary image obtained in the image segmentation model is processed by the target's own characteristics, mainly for the purpose of further optimizing the accuracy of the predicted binary image, and thus relies on the geometric characteristics of the target itself. For example, when the target is a zebra crossing, it relies on the zebra crossing's own geometric characteristics.

[0111] Furthermore, the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary map. The LOSS function in the image segmentation model is obtained by statistically analyzing multiple losses.

[0112] Specifically, in machine learning or deep learning, when multiple loss functions need to be weighted summed to calculate the total loss and then optimized by gradient descent, this can be achieved by specifying a coefficient for each loss function. This is often used for multi-task learning, where the model needs to learn multiple related or unrelated tasks at the same time. The loss function of each task can have a specific weight so that their contribution to the total loss can be adjusted according to the importance or difficulty of the task.

[0113] It can be understood that the loss function, also known as the cost function, is an important concept in evaluating model performance in machine learning and deep learning. It is used to quantify the difference or error between the model's predicted value and the true value. The main purpose of the loss function is to guide the model's training process: by minimizing the loss function, we can find the optimal value of the model parameters, thereby improving the model's prediction accuracy. During the training process, by constantly adjusting the model parameters (such as the weights in the neural network) to reduce the value of the loss function, the model gradually learns the inherent laws and characteristics of the data, so that it can make more accurate predictions.

[0114] In one embodiment of the present application, the segmentation module 330 is specifically used to obtain an image segmentation result according to the pre-trained image segmentation model.

[0115] The pre-trained image segmentation model performs corresponding processing operations on the input image data containing the target to be segmented based on the longest side and the shortest side. At the same time, the weak label target box that is input together is adjusted accordingly. Then, in the merged Image Encoder network of the image input in the weSAM segmentation model, the weak label rectangular box is input into the Prompt Encoder network structure, and finally the Mask Decoder outputs the binary image Mask image.

[0116] In one embodiment of the present application, the processing module 320 is further used to process the binary image obtained in the image segmentation model according to the characteristics of the target, including at least one of the following:

[0117] The binary image obtained in the image segmentation model is subjected to morphological processing to remove noise points;

[0118] The outer contour of each connected area is extracted from the binary image obtained in the image segmentation model, and the convex hull area and area of ​​each contour are obtained. The area is compared with a preset threshold. If it is less than the preset threshold, it is set as the background, otherwise it is set as the foreground.

[0119] In one embodiment of the present application, the processing module 320 is further used to divide the pre-trained image segmentation model into Image Encoder, Prompt Encoder and Mask Decoder;

[0120] The Image Encoder is used to encode image information and extract features;

[0121] The Prompt Encoder is used to encode the target frame information of the data and extract features;

[0122] The Mask Decoder is used to fuse the features encoded by the Image Encoder and the Prompt Encoder to obtain a prediction result;

[0123] The Image Encoder includes three branches, namely, an Anchor branch, a Teacher branch, and a Student branch. A weak data enhancement method is used for the Anchor branch and the Teacher branch, and a strong data enhancement method is used for the Student branch.

[0124] In one embodiment of the present application, the processing module 320 is also used to

[0125] Binary graph based on Anchor branch and the binary graph of the Teacher branch , according to the characteristics of the target itself, the binary graph of the new Teacher branch is obtained by reprocessing And the binary graph of the Anchor branch ;

[0126] The target's own characteristics include at least one of the following: a closed area characteristic of the target, and a noise characteristic of a connected area of ​​the target.

[0127] In one embodiment of the present application, the Teacher branch and the Student branch output probability maps and The processing module 320 is also used for

[0128] use and Calculate the focal loss and dice loss of the corresponding branch;

[0129] use Respectively and , Calculate the dice loss of the corresponding branch;

[0130] Based on binary image Get the feature map Each instance feature , based on binary graph Get the feature map Each instance feature , calculate the instance features of the Teacher branch and the Anchor branch and Similarity loss on ;

[0131] All losses are increased by a specified coefficient and summed up, and the gradient is calculated to determine the LOSS function.

[0132] In one embodiment of the present application, an update module is also included for

[0133] In the Image Encoder, LoRA parameters are added to the Teacher branch and the Student branch for training, and the LoRA parameters are shared and the original Image Encoder structure is frozen;

[0134] Input the weakly labeled target frame into the Prompt Encoder for feature extraction;

[0135] The Mask Decoder receives feature information output from the three branches of the Image Encoder and the Prompt Encoder branch respectively to perform feature fusion, and outputs feature maps of the three branches after the feature fusion.

[0136] In one embodiment of the present application, the segmentation module 330 is also used to

[0137] The pre-trained image segmentation model merges the trained LoRA parameters into the ImageEncoder;

[0138] Perform preset processing operations on the input image based on the longest and shortest sides, and perform preset adjustment operations on the weak label target box;

[0139] The image is input into the merged Image Encoder network and the weak label target box is input into the Prompt Encoder network structure, and the binary image Mask image is output through the Mask Decoder.

[0140] In one embodiment of the present application, a post-processing module is also included for

[0141] Obtaining target pixel coordinate information according to the target information in the image segmentation result;

[0142] Convert the target pixel coordinate information into a world coordinate system to obtain the target coordinate information in the world coordinate system;

[0143] The roadside camera external parameters are calibrated or corrected according to the coordinate information of the target in the world coordinate system and the target pixel coordinate information in the roadside camera.

[0144] It can be understood that the above-mentioned image segmentation device can implement each step of the image segmentation method provided in the above-mentioned embodiment, and the relevant explanations about the image segmentation method are applicable to the image segmentation device, which will not be repeated here.

[0145] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0146] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0148] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an image segmentation device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0149] Obtain image data containing the target to be segmented and the weakly labeled target frame;

[0150] The image data containing the target to be segmented and the weakly labeled target frame are input into a pre-trained image segmentation model, the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image;

[0151] An image segmentation result is obtained according to the pre-trained image segmentation model.

[0152] The above application Figure 1 The method performed by the image segmentation device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0153] The electronic device may also perform Figure 1 The method is performed by the image segmentation device in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0154] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the image segmentation device in the illustrated embodiment is specifically used to perform:

[0155] Obtain image data containing the target to be segmented and the weakly labeled target frame;

[0156] The image data containing the target to be segmented and the weakly labeled target frame are input into a pre-trained image segmentation model, the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image;

[0157] An image segmentation result is obtained according to the pre-trained image segmentation model.

[0158] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0163] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0164] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0165] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0167] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image segmentation method, wherein: The method comprises: Obtain image data containing the target to be segmented and the weakly labeled target frame; The image data containing the target to be segmented and the weakly labeled target frame are input into a pre-trained image segmentation model, the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image; An image segmentation result is obtained according to the pre-trained image segmentation model.

2. The method of claim 1, wherein: The binary image obtained in the image segmentation model is processed according to the characteristics of the target and includes at least one of the following: The binary image obtained in the image segmentation model is subjected to morphological processing to remove noise points; The outer contour of each connected area is extracted from the binary image obtained in the image segmentation model, and the convex hull area and area of ​​each contour are obtained. The area is compared with a preset threshold. If it is less than the preset threshold, it is set as the background, otherwise it is set as the foreground.

3. The method of claim 1, wherein: The pre-trained image segmentation model at least includes ImageEncoder, Prompt Encoder and Mask Decoder; The Image Encoder is used to encode the image data and then extract features; The Prompt Encoder is used to encode the weakly labeled target frame information and then extract features; The Mask Decoder is used to fuse the features extracted after encoding by the Image Encoder and the Prompt Encoder to obtain a prediction result; The Image Encoder includes three branches, namely, an Anchor branch, a Teacher branch, and a Student branch. A weak data enhancement method is used for the Anchor branch and the Teacher branch, and a strong data enhancement method is used for the Student branch.

4. The method of claim 3, wherein: The binary image obtained in the image segmentation model is processed again according to the characteristics of the target itself, including: Binary graph based on Anchor branch and the binary graph of the Teacher branch , according to the characteristics of the target itself, the binary graph of the new Teacher branch is obtained by reprocessing And the binary graph of the Anchor branch ; The target's own characteristics include at least one of the following: a closed area characteristic of the target, and a noise characteristic of a connected area of ​​the target.

5. The method of claim 4, wherein: The Teacher branch and the Student branch output probability maps respectively and The LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary map, including: use and Calculate the focal loss and dice loss of the corresponding branch; use Respectively and , Calculate the dice loss of the corresponding branch; Based on binary image Get the feature map Each instance feature , based on binary graph Get the feature map Each instance feature , calculate the instance features of the Teacher branch and the Anchor branch and The similarity loss on the feature map It is generated by the Teacher branch after encoding in the Image Encoder. The feature map The Anchor branch uses the original structure of the Image Encoder and first freezes the structure and then encodes it; All losses are increased by a specified coefficient and summed up, and the gradient is calculated to determine the LOSS function.

6. The method of claim 3, wherein: The method further comprises: In the Image Encoder, LoRA parameters are added to the Teacher branch and the Student branch for training, and the LoRA parameters are shared and the original Image Encoder structure is frozen; Input the weakly labeled target frame into the Prompt Encoder for feature extraction; The Mask Decoder receives feature information output from the three branches of the Image Encoder and the Prompt Encoder branch respectively to perform feature fusion, and outputs feature maps of the three branches after the feature fusion.

7. The method of claim 3, wherein: According to the pre-trained image segmentation model, an image segmentation result is obtained, including: The pre-trained image segmentation model merges the trained LoRA parameters into the Image Encoder; Perform preset processing operations on the input image based on the longest and shortest sides, and perform preset adjustment operations on the weakly labeled target box; The image is input into the merged Image Encoder network, the weak label target box is input into the Prompt Encoder network structure, and the binary image Mask image is output through the Mask Decoder.

8. The method of claim 1, wherein: After obtaining the image segmentation result according to the pre-trained image segmentation model, the method further includes: Obtaining target pixel coordinate information according to the target information in the image segmentation result; Convert the target pixel coordinate information into a world coordinate system to obtain the target coordinate information in the world coordinate system; The roadside camera external parameters are calibrated or corrected according to the coordinate information of the target in the world coordinate system and the target pixel coordinate information in the roadside camera.

9. An image segmentation device, wherein: The device comprises: An acquisition module is used to acquire image data containing the target to be segmented and a weakly labeled target frame; A processing module, used for inputting the image data containing the target to be segmented and the weakly labeled target frame into a pre-trained image segmentation model, wherein the binary image obtained in the image segmentation model is reprocessed according to the characteristics of the target itself, and the LOSS function in the image segmentation model is calculated based on the probability map and the reprocessed binary image; The segmentation module is used to obtain an image segmentation result according to the pre-trained image segmentation model.

10. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

11. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enables the electronic device to execute any one of the methods of claims 1 to 7.

Images