Semi-automatic annotation method and device for flotation foam images based on prompt learning

Abstract

The present application provides a method and device for semi-automatic annotation of flotation foam images based on prompt learning, which belongs to the field of ore flotation. The method includes: training a second target detection model according to an incompletely annotated target detection data set to obtain a second target detection model; repeating the step of training the second target detection model according to an updated incompletely annotated target detection data set, and finally obtaining an optimized second target detection model and an updated incompletely annotated target detection data set; training a second segmentation model according to an incompletely annotated segmented image data set to obtain a second segmentation model; repeating the training of the second segmentation model according to an updated incompletely annotated segmented image data set, and finally obtaining an optimized second segmentation model and an updated incompletely annotated segmented image data set. By obtaining visual cues of manual annotation and optimizing the detection model and the segmentation model, the annotation efficiency of the contours of instances in flotation foam images is improved.

Description

Technical Field

[0001] The present application relates to the field of ore flotation, and in particular to a method and device for semi-automatic annotation of flotation foam images based on prompt learning. Background Art

[0002] In the field of mineral resource development and processing, image analysis of flotation foam is one of the key research areas. Flotation foam analysis is mainly used to evaluate the flotation effect of minerals. These analyses are of great significance for optimizing mineral processing technology and improving resource utilization. However, the analysis tasks in flotation foam images usually require a lot of annotation work to provide high-quality data for subsequent deep learning algorithms. This process is not only time-consuming and labor-intensive, but also easily affected by human subjective factors, which may affect the accuracy and consistency of data annotation. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present application provide a method and device for semi-automatic annotation of flotation foam images based on prompt learning.

[0004] In a first aspect, an embodiment of the present application provides a semi-automatic annotation method for flotation foam images based on prompt learning, the method comprising: collecting a flotation foam image, preprocessing the flotation foam image, and obtaining a preprocessed flotation foam image; constructing a first target detection model, obtaining a first flotation foam image in which an initial visual prompt is manually annotated for a single instance in the preprocessed flotation foam image, and annotating visual prompts for the instance of the first flotation foam image through the first target detection model and the initial visual prompt to obtain a second flotation foam image; obtaining a third flotation foam image in which visual prompts are manually supplemented for the second flotation foam image, and constructing an incompletely annotated target detection dataset based on the third flotation foam image; training a second target detection model based on the incompletely annotated target detection dataset to obtain a second target detection model; and repeatedly performing: annotating visual prompts for new flotation foam images through the second target detection model, updating the annotated flotation foam images to the incompletely annotated target detection dataset, and re-annotating the image based on the updated incompletely annotated target detection dataset. The step of retraining the second target detection model is repeated until a preset number of executions is reached, and finally an optimized second target detection model and an updated incompletely labeled target detection data set are obtained; the labeled instances of the flotation foam image in the incompletely labeled target detection data set are used as visual cues for contour segmentation by the first segmentation model to obtain a first incompletely labeled segmented image; a second incompletely labeled segmented image is obtained by manually supplementing and correcting the labeled instances in the first incompletely labeled segmented image, and an incompletely labeled segmented image data set is constructed according to the second incompletely labeled segmented image; a second segmentation model is trained according to the incompletely labeled segmented image data set to obtain a second segmentation model; and the following steps are repeated: contour segmentation is performed on the new flotation foam image by the second segmentation model, the flotation foam image with contour segmentation completed is updated to the incompletely labeled segmented image data set, and the second segmentation model is repeatedly trained according to the updated incompletely labeled segmented image data set until a preset number of executions is reached, and finally an optimized second segmentation model and an updated incompletely labeled segmented image data set are obtained.

[0005] In one embodiment, the preprocessing of the flotation foam image to obtain the preprocessed flotation foam image includes: calculating the blurriness of each collected flotation foam image based on a blurriness evaluation function, screening the flotation foam images according to the blurriness to obtain a clear flotation foam image; and performing adaptive brightness control on the clear flotation foam image and then performing contrast enhancement to obtain the preprocessed flotation foam image.

[0006] In one embodiment, the clear flotation foam image is subjected to adaptive brightness and then contrast enhancement, including: dividing the input clear flotation foam image into non-overlapping sub-blocks of equal size; calculating the histogram and clipping threshold of each sub-block; reallocating the redundant pixels in each sub-block that are greater than the clipping threshold to complete adaptive brightness; performing histogram equalization on each sub-block after adaptive brightness, and reconstructing the grayscale values ​​of the pixels of each sub-block after the histogram equalization by using bilinear interpolation to complete contrast enhancement.

[0007] In one embodiment, the visual cue includes a rectangular box, a dot or a mask, and the visual cue is annotated on the instance of the first flotation foam image through the first target detection model and the initial visual cue to obtain the second flotation foam image, including: the first target detection model annotates the instances in the first flotation foam image that are similar to the annotated instances with corresponding rectangular boxes, points or masks based on the rectangular boxes, points or masks of the annotated instances in the first flotation foam image.

[0008] In one embodiment, training a second target detection model based on the incompletely labeled target detection dataset includes: constructing a backbone network structure, a feature fusion structure and an output head of the second target detection model based on the incompletely labeled target detection dataset; inputting the incompletely labeled target detection dataset into the backbone network structure of the second target detection model to obtain multiple feature maps; inputting multiple feature maps into the feature fusion structure to obtain a feature fusion map; inputting the feature fusion map into the output head of the second target detection model to output visual cues of the instance.

[0009] In one embodiment, updating the labeled flotation foam image to the incompletely labeled target detection dataset includes: adding the output instance with a confidence level higher than a confidence threshold as the labeling result of the flotation foam image to the original incompletely labeled target detection dataset to obtain an updated incompletely labeled target detection dataset.

[0010] In a second aspect, an embodiment of the present application provides a semi-automatic flotation foam image annotation device based on prompt learning, and the semi-automatic flotation foam image annotation device based on prompt learning includes: an acquisition module, which is used to acquire flotation foam images, pre-process the flotation foam images, and obtain pre-processed flotation foam images; an annotation module, which is used to construct a first target detection model, obtain a first flotation foam image in which an initial visual prompt is manually annotated on a single instance in the pre-processed flotation foam image, and annotate the instance of the first flotation foam image with a visual prompt through the first target detection model and the initial visual prompt to obtain a second flotation foam image; a first acquisition module, which is used to obtain a third flotation foam image in which visual prompts are manually supplemented for the second flotation foam image, and construct an incompletely annotated target detection data set according to the third flotation foam image; a first training module, which is used to train a second target detection model according to the incompletely annotated target detection data set to obtain a second target detection model; and a first loop module, which is used to repeatedly execute: annotating visual prompts on new flotation foam images through the second target detection model, updating the annotated flotation foam images to the incompletely annotated target detection data set, and updating the annotated flotation foam images according to the updated The incompletely labeled target detection data set repeatedly trains the second target detection model until a preset number of executions is reached, and finally an optimized second target detection model and an updated incompletely labeled target detection data set are obtained; a segmentation module is used to use the first segmentation model to perform contour segmentation on the labeled instances of the flotation foam image in the incompletely labeled target detection data set as visual cues to obtain a first incompletely labeled segmented image; a second acquisition module is used to obtain a second incompletely labeled segmented image that is manually supplemented and corrected by the labeled instances in the first incompletely labeled segmented image, and an incompletely labeled segmented image data set is constructed according to the second incompletely labeled segmented image; a second training module is used to train a second segmentation model according to the incompletely labeled segmented image data set to obtain a second segmentation model; a second loop module is used to repeatedly execute: contour segmentation of a new flotation foam image by the second segmentation model, updating the flotation foam image with completed contour segmentation to the incompletely labeled segmented image data set, and repeatedly training the second segmentation model according to the updated incompletely labeled segmented image data set until a preset number of executions is reached, and finally an optimized second segmentation model and an updated incompletely labeled segmented image data set are obtained.

[0011] In a third aspect, an embodiment of the present application provides an electronic device including a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is run on the processor, the semi-automatic flotation foam image labeling method based on prompt learning provided in the first aspect is executed.

[0012] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed on a processor, executes the semi-automatic flotation foam image labeling method based on prompt learning provided in the first aspect.

[0013] The semi-automatic flotation foam image labeling method based on prompt learning provided in the present application collects a flotation foam image, preprocesses the flotation foam image, and obtains a preprocessed flotation foam image; constructs a first target detection model, obtains a first flotation foam image in which an initial visual prompt is manually labeled for a single instance in the preprocessed flotation foam image, and labels the instance of the first flotation foam image with a visual prompt through the first target detection model and the initial visual prompt to obtain a second flotation foam image; obtains a third flotation foam image in which visual prompts are manually supplemented for the second flotation foam image, and constructs an incompletely labeled target detection dataset based on the third flotation foam image; trains a second target detection model based on the incompletely labeled target detection dataset to obtain a second target detection model; and repeatedly executes: labeling visual prompts for new flotation foam images through the second target detection model, updating the labeled flotation foam image to the incompletely labeled target detection dataset, and repeatedly training the second target according to the updated incompletely labeled target detection dataset. The steps of detecting the model are performed until the preset number of executions is reached, and finally the optimized second target detection model and the updated incompletely labeled target detection data set are obtained; the labeled instances of the flotation foam image in the incompletely labeled target detection data set are used as visual cues for contour segmentation by the first segmentation model to obtain the first incompletely labeled segmented image; a second incompletely labeled segmented image is obtained to supplement and correct the labeled instances in the first incompletely labeled segmented image manually, and an incompletely labeled segmented image data set is constructed according to the second incompletely labeled segmented image; a second segmentation model is trained according to the incompletely labeled segmented image data set to obtain a second segmentation model; and the following steps are repeated: contour segmentation of the new flotation foam image is performed by the second segmentation model, the flotation foam image with the completed contour segmentation is updated to the incompletely labeled segmented image data set, and the second segmentation model is repeatedly trained according to the updated incompletely labeled segmented image data set until the preset number of executions is reached, and finally the optimized second segmentation model and the updated incompletely labeled segmented image data set are obtained. By obtaining the manually labeled visual cues and optimizing the detection model and the segmentation model by the semi-supervised method, the contour labeling efficiency of the instances in the flotation foam image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope of protection of the present application. In each of the drawings, similar components are numbered similarly.

[0015] Figure 1 A schematic diagram of a process of a semi-automatic annotation method for flotation foam images based on prompt learning provided in an embodiment of the present application is shown;

[0016] Figure 2 Another schematic diagram of the process of the semi-automatic annotation method of flotation foam images based on prompt learning provided in an embodiment of the present application is shown;

[0017] Figure 3 A schematic diagram of visual prompts of an example provided by an embodiment of the present application is shown;

[0018] Figure 4 A schematic diagram of the network structure of a first target detection model provided in an embodiment of the present application is shown;

[0019] Figure 5 A schematic diagram of the network structure of a second target detection model provided in an embodiment of the present application is shown;

[0020] Figure 6 Another schematic diagram of the process of the semi-automatic flotation foam image annotation method based on prompt learning provided in an embodiment of the present application is shown;

[0021] Figure 7 A schematic diagram of a process for optimizing a second detection model provided in an embodiment of the present application is shown;

[0022] Figure 8 A schematic diagram of the network structure of the first segmentation model provided in an embodiment of the present application is shown;

[0023] Fig. 9 A schematic diagram of contour marking of an example provided in an embodiment of the present application is shown;

[0024] Fig.10 A schematic diagram of the network structure of the second segmentation model provided in an embodiment of the present application is shown;

[0025] Fig.11 Another schematic diagram of the process of the semi-automatic flotation foam image annotation method based on prompt learning provided in an embodiment of the present application is shown;

[0026] Fig.12 A structural schematic diagram of a flotation foam image semi-automatic annotation device based on prompt learning provided in an embodiment of the present application is shown.

[0027] Icon: 1200 - semi-automatic annotation device of flotation foam image based on prompt learning, 1201 - acquisition module, 1202 - annotation module, 1203 - first acquisition module, 1204 - first training module, 1205 - first circulation module, 1206 - segmentation module, 1207 - second acquisition module, 1208 - second training module, 1209 - second circulation module. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0029] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0030] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0031] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0032] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0033] Example 1

[0034] The embodiment of the present application provides a semi-automatic annotation method for flotation foam images based on prompt learning.

[0035] See also Figure 1, the semi-automatic annotation method of flotation foam image based on prompt learning includes steps S101-S109:

[0036] S101: collecting a flotation foam image, and preprocessing the flotation foam image to obtain a preprocessed flotation foam image.

[0037] In this embodiment, an image acquisition device is built based on a high-definition industrial camera, a dome light source and a light shield, and the image acquisition device is used to capture flotation foam images. Among them, the resolution of the high-definition industrial camera is not less than 2 million pixels. In order to eliminate the interference (such as shadows) caused by the uneven surface of the captured image and ensure the light intensity of the shooting, a strobe dome light with a power of not less than 40W is used to reduce the light intensity attenuation caused by the working environment temperature; a light shield is used to cover 2-3 times the shooting area as much as possible to reduce the influence of external ambient light on the color and texture characteristics of the image. Based on the flotation foam image obtained by the image acquisition device, an image database is established, and the collected flotation foam image is preprocessed until a clear flotation foam image appears, and then the clear flotation foam image is contrast enhanced.

[0038] See also Figure 2 In one embodiment, the preprocessing of the flotation foam image includes S1011-S1012:

[0039] S1011: Calculating the fuzziness of each flotation foam image based on a fuzziness evaluation function, and screening the flotation foam images according to the fuzziness to obtain clear flotation foam images.

[0040] In this embodiment, since the spatial domain evaluation function presents an obvious single-peak characteristic for images collected under normal illumination, a combination of energy gradient (TenenGrad) evaluation function, square gradient evaluation function and Vollath evaluation function is proposed to adaptively filter blurred images from the original image. The blurriness of the image is obtained by weighted summation using the above three blur evaluation functions. For defocus blur, a blur threshold range is set. If the image blur exceeds the threshold, an alarm is triggered to remind manual maintenance of the image acquisition device. For motion blur, the blurry frames are removed by analyzing continuous frame images until a clear image appears.

[0041] S1012: After adaptively adjusting the brightness of the clear flotation foam image, contrast enhancement is performed to obtain the target flotation foam image.

[0042] In this embodiment, the clear flotation foam image after screening is subjected to adaptive illumination compensation and contrast enhancement using a limited contrast adaptive histogram equalization algorithm.

[0043] In one embodiment, S1012 includes: dividing the input clear flotation foam image into non-overlapping sub-blocks of equal size; calculating the histogram and clipping threshold of each sub-block; reallocating the redundant pixels in each sub-block that are larger than the clipping threshold to complete adaptive brightness; performing histogram equalization on each sub-block after adaptive brightness, and reconstructing the grayscale values ​​of the pixels of each sub-block after histogram equalization by using bilinear interpolation to complete contrast enhancement.

[0044] In this embodiment, since different areas of the image have different brightness and contrast characteristics, in order to locally process the image, the input clear flotation foam image is divided into equal-sized and non-overlapping sub-blocks; the grayscale histogram of each sub-block is calculated, and the grayscale histogram shows the number of pixels at each grayscale level in the sub-block, thereby providing information about the brightness distribution of the sub-block; then, the clipping threshold is calculated based on the grayscale histogram of each sub-block, and the clipping threshold is used to distinguish between "important" pixels (i.e., pixels that have important contributions to the content of the flotation foam image) and "redundant" pixels (i.e., pixels generated due to noise or overexposure) in the sub-block; for each sub-block, all "redundant" pixels greater than the clipping threshold are reallocated, that is, the grayscale value of the pixel is reduced or remapped to a lower grayscale level to reduce the excessively bright part of the sub-block, thereby achieving adaptive brightness adjustment.

[0045] Furthermore, after completing the adaptive brightness adjustment, histogram equalization is performed on each sub-block. Histogram equalization can redistribute the grayscale to enhance the contrast of the image, making the details in the image clearer. In order to avoid discontinuity at the sub-block boundary, bilinear interpolation is used to reconstruct the grayscale value of the pixel points of each sub-block, effectively reducing the artifacts and distortion generated in the image segmentation and reconstruction process.

[0046] S102: construct a first target detection model, obtain a first flotation foam image in which initial visual cues are manually annotated for a single instance in the preprocessed flotation foam image, and annotate visual cues for instances of the first flotation foam image using the first target detection model and the initial visual cues to obtain a second flotation foam image.

[0047] In this embodiment, see Figure 3 , is a visual cue (rectangular box) of a manually annotated reference instance, and a schematic diagram of the annotation result output by the first object detection model. The first object detection model is a large model of interactive visual cue object detection. This model annotates unannotated instances with visual cues based on manually annotated reference instances. For the specific process of annotating flotation foam images based on visual cues (rectangular boxes or dots, etc.), see Figure 4 ,First, given the target image input (i.e. the preprocessed flotation foam image) , the image encoder (ImageEncoder) of the large model of object detection with interactive visual cues first extracts visual features ,Right now , using visual cues drawn by human interaction (including but not limited to boxes, points or masks, etc.) as the prompt P of the reference instance, the prompt encoder (PromptEncoder) extracts the image visual features from the image Extracting encoded visual cues , ; BoxDecoder converts the visual features of the target flotation foam image and visual cues for encoding The combination is taken as input and the detected rectangular box B and the associated confidence score S of the rectangular box and the rectangular box of the reference instance are output, that is, , a predetermined score threshold is applied to filter the detected boxes. The final model detects all instances similar to the reference instances among the unlabeled instances in the preprocessed flotation foam image, and labels them with visual cues. Under the guidance of visual feedback from the visual cue target detection large model, the instance detection results such as rectangular boxes or points output by the target detection are interactively optimized by prompting misdetected or missing objects, and finally the second flotation foam image is obtained.

[0048] In one embodiment, the visual cue includes a rectangular box, a point or a mask, and S102 includes: the first target detection model marks the instances in the first flotation foam image that are similar to the labeled instances as corresponding rectangular boxes, points or masks based on the rectangular boxes, points or masks of the labeled instances in the first flotation foam image.

[0049] In this embodiment, first, visual cue annotations are manually performed on instances (such as bubbles) in the target flotation foam image to obtain a reference instance. The visual cue annotation can be a rectangular box (a rectangular box selects a single bubble), a point (annotated at the center of the bubble) or a mask (covering the entire bubble area). Then, the interactive visual cue target detection model detects unlabeled instances similar to the reference instance in the entire image based on the rectangular box, point or mask of the reference instance, and annotates them with corresponding rectangular boxes, points or masks. That is, if the visual cue input is a point, the detection result output is a point in the center of each bubble. If it is a rectangular box, the detection result output is a regular rectangular box around each bubble.

[0050] S103: Acquire a third flotation foam image that is manually supplemented with visual cues for annotating the second flotation foam image, and construct an incompletely annotated target detection dataset based on the third flotation foam image.

[0051] In this embodiment, due to the blurred boundaries caused when taking the picture, or very small bubbles cannot be automatically labeled by the large model of target detection with interactive visual prompts, that is, the flotation foam image may still contain unlabeled instances that are difficult to identify. At this time, it is necessary to manually supplement or correct the annotations of these unlabeled instances. After the points are manually supplemented and corrected, they cannot be manually labeled due to factors such as resolution, blur, and unclear boundaries. What is obtained is an incompletely labeled target flotation foam image. The foam flotation image contains both instances automatically labeled by the model and instances manually supplemented and labeled. Multiple (100-200) incompletely labeled target flotation foam images are combined to form an incompletely labeled target detection data set. Although not all instances in each image are labeled, they cover multiple types of instances and different labeling situations as a whole.

[0052] Specifically, assuming that there are 500 bubbles in a flotation foam image, we first manually mark a rectangular box on the image. The first target detection model can automatically identify 400 bubbles, and then manually add 50 bubbles. The remaining 50 bubbles cannot be marked due to resolution, blur, unclear boundaries and other factors. In this way, we get a (450 / 500) flotation foam image.

[0053] S104: Training a second object detection model according to the incompletely labeled object detection data set to obtain a second object detection model.

[0054] In this embodiment, see Figure 5 , which is the network structure diagram of the second target detection model. The second target detection network includes three parts: the backbone network, the feature fusion layer and the output head. The output head includes the classification head, the regression head and the fine-tuning branch. The network structure design of each part is built around small samples (an incompletely labeled target detection data set composed of a small number of flotation foam images) and dense instance detection in the image. By expanding the data volume through the semi-supervised learning method, the second detection model can achieve better visual cue labeling effect of instances (bubbles) under the premise of a small amount of labeled data.

[0055] See also Figure 6 In one embodiment, the S104 includes steps S1041-S1044:

[0056] S1041: Constructing the backbone network structure, feature fusion structure and output head of the second target detection model according to the incompletely labeled target detection dataset.

[0057] In this embodiment, for an incompletely labeled target detection dataset, a second target detection model for small samples and dense targets is designed and trained, as well as a backbone network structure, feature fusion structure and output head of the second target detection model.

[0058] S1042: Input the incompletely labeled target detection data set into the backbone network structure of the second target detection model to obtain multiple feature maps.

[0059] In this embodiment, the backbone network of the second target detection model adopts a hybrid model including but not limited to CNN, Transformer and CNN-Transformer, wherein the key components include but are not limited to C2fCIB, ELAN, GELAN, etc.; its main core lies in designing a backbone network that can retain high-resolution features (such as adding more shallow feature maps) and multi-scale feature representation.

[0060] S1043: Input the plurality of feature maps into the feature fusion structure to obtain a feature fusion map.

[0061] In this embodiment, the feature fusion part of the second target detection model is divided into hierarchical feature fusion and adaptive feature fusion. The hierarchical feature fusion strategy realizes the shallow high-resolution features and the deep high-level semantic features through a multi-scale feature pyramid, including but not limited to FPN, BiFPN, AFPN, PAFPN, NASFPN, CARAFE, ASFF, DRFPN, etc.; the adaptive feature fusion mechanism is realized through the attention mechanism, including but not limited to SENet, CBAM, SKNet, DANet, OCNet, GCNet, etc.

[0062] S1044: Input the feature fusion map into the output head of the second object detection model, and output the visual prompt of the instance.

[0063] In one embodiment, the output head of the second target detection model decouples classification and positioning in an anchor-free manner, and adds a fine-tuning branch for the star representation of the prediction box. The star representation is a more flexible and accurate bounding box representation method that more accurately describes the shape and size of the target instance by adjusting multiple key points (such as the vertices of the star); the traditional rectangular box is expanded to a star representation, and a fine-tuning branch is added to the output head to fine-tune visual cues, such as the position of the key points of each instance, such as making the predicted rectangular box of each instance fit the actual target more accurately.

[0064] Optionally, in dense target instance detection, due to the large number of target instances, the small distance between target instances, the overlap and interference between target instances, the loss function should focus on solving the problems of imbalanced positive and negative sample ratios, blurred boundaries and confusion between target instances. The loss function includes but is not limited to boundary-aware loss, VariFocal loss+GIOU, DIOU, CIOU, and alpha-IOU.

[0065] Exemplarily, the training of the second target detection model adopts a label assignment strategy including but not limited to a center-based label assignment strategy, simOTA, ATSS, and a dual-label assignment strategy; the center-based label assignment strategy can preferentially select anchors close to the target center as positive samples, thereby reducing the interference of background noise.

[0066] S105: Repeated execution: annotating visual cues on the new flotation foam image through the second target detection model, updating the annotated flotation foam image to the incompletely annotated target detection dataset, and repeating the steps of training the second target detection model according to the updated incompletely annotated target detection dataset until a preset number of executions is reached, and finally obtaining an optimized second target detection model and an updated incompletely annotated target detection dataset.

[0067] In this embodiment, the specific method is shown in Figure 7 First, 500 new unlabeled preprocessed flotation foam images are used for the second detection model to identify, and 500 flotation foam images with labeled visual cues are obtained. The 500 labeled images are combined with the first labeled 200 images (incompletely labeled target detection data set), and the new second target detection model is trained again. Assume that the second target detection model detects 460 bubbles this time. Next, 500 unlabeled images are continued to be used for the second target detection model to identify, and these 500 images are combined with the previous 700 images, and the second target detection model is trained again. The number of iterations is set (such as 5-10 times). In each iteration, the updated model is used to predict the unlabeled data, generate new pseudo labels, and select high-confidence images to expand the training data set, and then the model is trained again. After the cycle reaches the number of iterations, the data set expansion and the second target detection model training are no longer performed, and the optimized second target detection model and the updated incompletely labeled target detection data set are finally obtained.

[0068] In one embodiment, updating the labeled flotation foam image to the incompletely labeled target detection dataset includes: adding the output instance with a confidence level higher than a confidence threshold as the labeling result of the flotation foam image to the original incompletely labeled target detection dataset to obtain an updated incompletely labeled target detection dataset.

[0069] In this embodiment, the unlabeled flotation foam image is predicted by the second target detection model to generate pseudo labels, and the labeled instances with confidence levels higher than a preset threshold are screened out from the generated pseudo labels. The screened flotation foam images are added to the original incompletely labeled data set to form an extended training data set, thereby further improving the detection performance and generalization ability of the second target detection model.

[0070] S106: Using a first segmentation model, the labeled instances of the flotation foam image in the incompletely labeled target detection dataset are used as visual cues to perform contour segmentation to obtain a first incompletely labeled segmented image.

[0071] In this embodiment, the first segmentation model is a large segmentation model of visual cues, see Figure 8 , which is the neural network structure of the segmentation model. The steps to obtain the incompletely labeled segmented image are as follows. Specifically, the visual prompt segmentation model contains three key components: image encoder (ImageEncoder), prompt encoder (PromptEncoder) and mask decoder (MaskDecoder). Optionally, use masked autoencoders (MAE) for self-supervised training to obtain a pre-trained vision transformer (ViT model as the image encoder. The original image Obtain visual features after passing through the image encoder , and then pass through the image embedding layer to obtain the image embedding : , .

[0072] Optionally, sparse visual cues, such as point P and rectangular box B, are obtained from the optimized target detection in step S105. The encoding is added to the learned embedding of each cue type and to the image embedding element through position embedding. The cue visual encoding maps these embeddings to obtain the cue embedding Eprompt, that is, .

[0073] Optionally, the mask decoder uses a Transformer decoder block with self-attention and cross-attention followed by a dynamic mask prediction head to update the image embedding, prompt embedding, and then maps the output token to a dynamic linear classifier using MLP by upsampling the image embedding. Finally, the mask foreground probability S is calculated at the instance position of each flotation bubble image, and the instance under the visual prompt is contour segmented according to the mask foreground probability S. ,For complex targets that are difficult for the model to recognize, the missing or misdetected cases are manually corrected, and an incompletely labeled segmented image dataset is obtained.

[0074] S107: Acquire a second incompletely labeled segmented image that is obtained by manually supplementing and correcting the labeled instances in the first incompletely labeled segmented image, and construct an incompletely labeled segmented image dataset based on the second incompletely labeled segmented image.

[0075] In this embodiment, the rectangular box output by the second target detection model is used as a visual cue and input into the visual cue segmentation model (i.e., the first segmentation model). The first segmentation model automatically converts the rectangular box into an edge contour. However, since there are instances where automatic contour segmentation cannot be performed, manual contour annotation is required. For example, assuming that there are 500 rectangular boxes in the visual cue of the flotation foam image, but the first segmentation model only obtains the edge contours of 400 bubbles, and then the output result is manually supplemented. Assuming that the contours of 50 bubble edges are supplemented, a (450 / 500) flotation foam image is obtained, see Fig. 9 ,Finally, an incompletely labeled segmented image dataset is formed based on ,multiple labeled flotation foam images.

[0076] S108: Training a second segmentation model according to the incompletely labeled segmented image dataset to obtain a second segmentation model.

[0077] In this embodiment, see Fig.10 , which is a schematic diagram of the network structure of the second segmentation model. The second segmentation detection network includes three parts: backbone network, feature fusion layer and output head. Each part is designed around small sample and dense target instance segmentation detection.

[0078] See also Fig.11 In one embodiment, the S108 includes S1081-S1085:

[0079] S1081: constructing a backbone network of the second segmentation model, and obtaining multiple underlying feature maps of different scales through the backbone network, wherein the backbone network includes a high-resolution network structure and a multi-scale receptive field.

[0080] In this embodiment, the backbone of the segmentation network adopts a high-resolution network structure and multi-scale receptive field expansion, and adopts a hybrid architecture including but not limited to CNN, Transformer and CNN-Transformer. After the input image passes through the backbone network, feature maps C2, C3, C4, and C5 of different scales are output.

[0081] S1082: Construct a feature fusion layer of the second segmentation model, input a plurality of feature maps of different scales into the feature fusion layer, and obtain a plurality of deep feature maps of different scales.

[0082] In this embodiment, a context-aware module is introduced to enhance the expression of target features by aggregating global and local context information. The feature maps C2, C3, C4, and C5 obtained in step S1061 are input to the feature fusion layer to obtain feature maps P2, P3, P4, P5, P6, and P7; feature fusion methods include but are not limited to FPN, BiFPN, AFPN, PAFPN, NASFPN, CARAFE, ASFF, and DRFPN.

[0083] S1083: Construct a decoder of the second segmentation model, and perform feature fusion on the underlying feature map and the deep feature map through the decoder to obtain an output mask.

[0084] In this embodiment, a decoder is used to fuse the underlying features C2, C3, C4, C5 and the deep features P2, P3, P4, P5 to obtain an output mask B, wherein the decoder includes but is not limited to the decoder of the DeepLabV3 series, and decoders of other segmentation networks are also applicable.

[0085] S1084: Construct an output head of the second segmentation model, the deep feature map passes through the output head to obtain a suggested target box, the suggested target box passes through a layer of convolution to obtain an attention score, and finally passes through an activation function to obtain a mask probability map.

[0086] In this embodiment, P3, P4, P5, P6, and P7 of the feature pyramid are passed through the output head of the second segmentation model to obtain the proposed target box P. P is passed through a layer of convolution to obtain the attention score A. A is subjected to RoIAlign and reshape operations, and then bilinear interpolation is performed. Finally, softmax is used to obtain the mask probability map. , specifically, through the formula , , , .

[0087] S1085: Obtain a suggested mask according to the suggested target box and the output mask, perform a pooling operation on the suggested mask, and obtain a segmentation result according to a result of the pooling operation and the mask probability map.

[0088] In this embodiment, according to the proposed target frame Update the output mask B from the corresponding region to the proposed mask, and then perform RoI pooling on the proposed mask to obtain ,Right now ,in is the proposed target box in P; the mask probability map Multiply it with the corresponding RoI pooled suggestion mask, and then add it by channel to output the final segmentation result ,Right now .

[0089] Optionally, the loss function used for training the segmentation network includes but is not limited to a distribution-based loss function (such as cross entropy loss and focal loss), a region-based loss function (such as Dice loss and Tversky loss), a boundary-based loss function (shape aware loss), and a composite-based loss function (such as combo loss, which is a weighted sum of Dice loss and cross entropy loss).

[0090] S109: Repeat execution: perform contour segmentation on the new flotation foam image through the second segmentation model, update the flotation foam image with completed contour segmentation to the incompletely labeled segmentation image dataset, and repeat training the second segmentation model according to the updated incompletely labeled segmentation image dataset until the preset number of executions is reached, and finally obtain the optimized second segmentation model and the updated incompletely labeled segmentation image dataset.

[0091] In this embodiment, a semi-supervised training process is carried out based on the second segmentation model. For a new flotation foam image, the model predicts the unlabeled data and outputs the predicted label. The predicted label is added as a pseudo label of the unlabeled data to the incompletely labeled segmented image data to complete the update, and the updated data set is repeatedly trained on the second segmentation model. According to the above process, it is iterated 5-10 times to continuously optimize the model performance, and the second segmentation model and the updated incompletely labeled segmented image data set are obtained, thereby completing the semi-automatic labeling of the flotation foam image contour. Further, the segmentation mask is output according to the optimized second segmentation model, and the flotation foam image size distribution is calculated to guide the control and optimization of production process parameters.

[0092] The semi-automatic flotation foam image labeling method based on prompt learning provided in the present embodiment collects a flotation foam image, pre-processes the flotation foam image, and obtains a pre-processed flotation foam image; constructs a first target detection model, obtains a first flotation foam image in which a single instance in the pre-processed flotation foam image is manually labeled with an initial visual prompt, and labels the instance of the first flotation foam image through the first target detection model and the initial visual prompt to obtain a second flotation foam image; obtains a third flotation foam image in which the second flotation foam image is manually supplemented with annotated, and constructs an incompletely labeled target detection dataset according to the third flotation foam image; trains a second target detection model according to the incompletely labeled target detection dataset to obtain a second target detection model; and repeatedly executes the steps of labeling a new flotation foam image instance through the second target detection model, updating the labeled flotation foam image to the incompletely labeled target detection dataset, and repeatedly training the second target detection model according to the updated incompletely labeled target detection dataset. Steps are performed until the preset number of executions is reached, and finally the optimized second target detection model and the updated incompletely labeled target detection data set are obtained; the labeled instances of the flotation foam image in the incompletely labeled target detection data set are used as visual cues for contour segmentation by the first segmentation model to obtain the first incompletely labeled segmented image; a second incompletely labeled segmented image is obtained to supplement and correct the labeled instances in the first incompletely labeled segmented image manually, and an incompletely labeled segmented image data set is constructed according to the second incompletely labeled segmented image; a second segmentation model is trained according to the incompletely labeled segmented image data set to obtain a second segmentation model; and the following steps are repeated: contour segmentation is performed on the new flotation foam image by the second segmentation model, the flotation foam image with the completed contour segmentation is updated to the incompletely labeled segmented image data set, and the second segmentation model is repeatedly trained according to the updated incompletely labeled segmented image data set until the preset number of executions is reached, and finally the optimized second segmentation model and the updated incompletely labeled segmented image data set are obtained. By obtaining the manually labeled visual cues and optimizing the detection model and the segmentation model by the semi-supervised method, the contour labeling efficiency of the instances in the flotation foam image is improved.

[0093] Example 2

[0094] In addition, an embodiment of the present application provides a semi-automatic annotation device for flotation foam images based on prompt learning, which is applied to electronic devices.

[0095] like Fig.12 As shown, the flotation foam image semi-automatic annotation device 1200 based on prompt learning includes:

[0096] The acquisition module 1201 is used to acquire a flotation foam image and pre-process the flotation foam image to obtain a pre-processed flotation foam image;

[0097] The labeling module 1202 is used to build a first target detection model, obtain a first flotation foam image in which an initial visual cue is manually labeled for a single instance in the preprocessed flotation foam image, and label the instance of the first flotation foam image with a visual cue by using the first target detection model and the initial visual cue to obtain a second flotation foam image;

[0098] The first acquisition module 1203 is used to acquire a third flotation foam image that is manually supplemented with visual cues for annotating the second flotation foam image, and to construct an incompletely annotated target detection data set according to the third flotation foam image;

[0099] A first training module 1204 is used to train a second target detection model according to the incompletely labeled target detection data set to obtain a second target detection model;

[0100] The first loop module 1205 is used to repeatedly perform the steps of annotating a new flotation foam image with visual cues by the second object detection model, updating the annotated flotation foam image to the incompletely annotated object detection data set, and repeatedly training the second object detection model according to the updated incompletely annotated object detection data set until a preset number of executions is reached, and finally obtaining an optimized second object detection model and an updated incompletely annotated object detection data set;

[0101] A segmentation module 1206 is configured to perform contour segmentation using the labeled instances of the flotation foam image in the incompletely labeled object detection dataset as visual cues through a first segmentation model to obtain a first incompletely labeled segmented image;

[0102] A second acquisition module 1207 is used to acquire a second incompletely labeled segmented image that is obtained by manually supplementing and correcting the labeled instances in the first incompletely labeled segmented image, and to construct an incompletely labeled segmented image dataset based on the second incompletely labeled segmented image;

[0103] A second training module 1208 is used to train a second segmentation model according to the incompletely labeled segmented image dataset to obtain a second segmentation model;

[0104] The second loop module 1209 is used to repeatedly execute: contour segmentation of the new flotation foam image through the second segmentation model, updating the flotation foam image with completed contour segmentation to the incompletely labeled segmented image dataset, and repeatedly training the second segmentation model according to the updated incompletely labeled segmented image dataset until a preset number of executions is reached, and finally obtaining the optimized second segmentation model and the updated incompletely labeled segmented image dataset.

[0105] Optionally, the acquisition module 1201 shown is also used to calculate the blurriness of each collected flotation foam image based on a blurriness evaluation function, and to screen the flotation foam images according to the blurriness to obtain a clear flotation foam image; and to perform adaptive brightness control on the clear flotation foam image and then perform contrast enhancement to obtain the preprocessed flotation foam image.

[0106] Optionally, the acquisition module 1201 is also used to divide the input clear flotation foam image into non-overlapping sub-blocks of equal size; calculate the histogram and clipping threshold of each sub-block; reallocate the excess pixels in each sub-block that are greater than the clipping threshold to complete adaptive brightness; perform histogram equalization on each sub-block after adaptive brightness, and use bilinear interpolation to reconstruct the grayscale values ​​of the pixels of each sub-block after histogram equalization to complete contrast enhancement.

[0107] Optionally, the labeling module 1202 is further used to label instances in the first flotation foam image that are similar to the labeled instances as corresponding rectangular boxes, points or masks based on the rectangular boxes, points or masks of the labeled instances in the first flotation foam image through the first target detection model.

[0108] Optionally, the first training module 1204 shown is also used to construct the backbone network structure, feature fusion structure and output head of the second target detection model based on the incompletely labeled target detection dataset; input the incompletely labeled target detection dataset into the backbone network structure of the second target detection model to obtain multiple feature maps; input the multiple feature maps into the feature fusion structure to obtain a feature fusion map; input the feature fusion map into the output head of the second target detection model to output visual cues of the instance.

[0109] Optionally, the first loop module 1205 is further used to add the output instances with confidence levels higher than the confidence level threshold as the annotation results of the flotation foam images to the original incompletely annotated object detection dataset to obtain an updated incompletely annotated object detection dataset.

[0110] Optionally, the segmentation module 1206 is also used to construct a backbone network of the second segmentation model, through which multiple underlying feature maps of different scales are obtained, and the backbone network includes a high-resolution network structure and a multi-scale receptive field; construct a feature fusion layer of the second segmentation model, input multiple feature maps of different scales into the feature fusion layer, and obtain multiple deep feature maps of different scales; construct a decoder of the second segmentation model, and through the decoder, perform feature fusion on the underlying feature map and the deep feature map to obtain an output mask; construct an output head of the second segmentation model, the deep feature map passes through the output head to obtain a recommended target box, the recommended target box passes through a layer of convolution to obtain an attention score, and finally passes through an activation function to obtain a mask probability map; obtain a recommended mask based on the recommended target box and the output mask, perform a pooling operation on the recommended mask, and obtain a segmentation result based on the result of the pooling operation and the mask probability map.

[0111] The semi-automatic flotation foam image labeling device 1200 based on prompt learning provided in this embodiment can implement the semi-automatic flotation foam image labeling method based on prompt learning provided in Example 1, and will not be described again here to avoid repetition.

[0112] The semi-automatic flotation foam image labeling device based on prompt learning provided in the present embodiment collects a flotation foam image, pre-processes the flotation foam image to obtain a pre-processed flotation foam image; constructs a first target detection model, obtains a first flotation foam image in which an initial visual prompt is manually labeled for a single instance in the pre-processed flotation foam image, and labels the instance of the first flotation foam image with a visual prompt by using the first target detection model and the initial visual prompt to obtain a second flotation foam image; obtains a third flotation foam image in which visual prompts are manually supplemented for the second flotation foam image, and constructs an incompletely labeled target detection dataset according to the third flotation foam image; trains a second target detection model according to the incompletely labeled target detection dataset to obtain a second target detection model; and repeatedly performs: labeling visual prompts for a new flotation foam image by using the second target detection model, updating the labeled flotation foam image to the incompletely labeled target detection dataset, and repeatedly training the second target detection model according to the updated incompletely labeled target detection dataset. The steps of detecting the model are performed until the preset number of executions is reached, and finally the optimized second target detection model and the updated incompletely labeled target detection data set are obtained; the labeled instances of the flotation foam image in the incompletely labeled target detection data set are used as visual cues for contour segmentation through the first segmentation model to obtain the first incompletely labeled segmented image; the second incompletely labeled segmented image is obtained to supplement and correct the labeled instances in the first incompletely labeled segmented image manually, and the incompletely labeled segmented image data set is constructed according to the second incompletely labeled segmented image; the second segmentation model is trained according to the incompletely labeled segmented image data set to obtain the second segmentation model; and the following steps are repeated: contour segmentation of the new flotation foam image is performed through the second segmentation model, the flotation foam image with the completed contour segmentation is updated to the incompletely labeled segmented image data set, and the second segmentation model is repeatedly trained according to the updated incompletely labeled segmented image data set until the preset number of executions is reached, and finally the optimized second segmentation model and the updated incompletely labeled segmented image data set are obtained. By obtaining the manually labeled visual cues and optimizing the detection model and the segmentation model using the semi-supervised method, the contour labeling efficiency of the instances in the flotation foam image is improved.

[0113] Example 3

[0114] In addition, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program runs on the processor, the semi-automatic annotation method for flotation foam images based on prompt learning provided in Example 1 is executed.

[0115] The electronic device provided in the embodiment of the present invention can execute the steps of the semi-automatic flotation foam image annotation method based on prompt learning provided in the above method embodiment 1, which will not be described again to avoid repetition.

[0116] Example 4

[0117] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the semi-automatic annotation method for flotation foam images based on prompt learning provided in Example 1 is implemented.

[0118] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0119] The computer-readable storage medium provided in this embodiment can implement the semi-automatic flotation foam image labeling method based on prompt learning provided in Example 1, and will not be described again here to avoid repetition.

[0120] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0122] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A semi-automatic annotation method for flotation foam images based on prompt learning, characterized in that: The method comprises: collecting a flotation foam image, and preprocessing the flotation foam image to obtain a preprocessed flotation foam image; Constructing a first target detection model, obtaining a first flotation foam image in which an initial visual cue is manually annotated on a single instance in the preprocessed flotation foam image, and annotating visual cues on instances of the first flotation foam image by using the first target detection model and the initial visual cue to obtain a second flotation foam image; Acquire a third flotation foam image that is manually supplemented with visual cues for annotating the second flotation foam image, and construct an incompletely annotated target detection data set based on the third flotation foam image; Training a second target detection model according to the incompletely labeled target detection data set to obtain a second target detection model; Repeatedly perform: annotating visual cues on a new flotation foam image through the second target detection model, updating the annotated flotation foam image to the incompletely annotated target detection dataset, and repeatedly training the second target detection model according to the updated incompletely annotated target detection dataset until a preset number of executions is reached, and finally obtaining an optimized second target detection model and an updated incompletely annotated target detection dataset; Using a first segmentation model to perform contour segmentation on the annotated instance of the flotation foam image in the incompletely annotated object detection dataset as a visual cue, thereby obtaining a first incompletely annotated segmented image; Acquire a second incompletely labeled segmented image that is obtained by manually supplementing and correcting the labeled instances in the first incompletely labeled segmented image, and construct an incompletely labeled segmented image dataset based on the second incompletely labeled segmented image; Training a second segmentation model according to the incompletely labeled segmented image dataset to obtain a second segmentation model; Repeated execution: performing contour segmentation on the new flotation foam image by using the second segmentation model, updating the flotation foam image after contour segmentation to the incompletely labeled segmented image dataset, and repeatedly training the second segmentation model according to the updated incompletely labeled segmented image dataset until a preset number of executions is reached, and finally obtaining an optimized second segmentation model and an updated incompletely labeled segmented image dataset; The step of training a second segmentation model according to the incompletely labeled segmented image dataset to obtain a second segmentation model comprises: Constructing a backbone network of the second segmentation model, and obtaining a plurality of underlying feature maps of different scales through the backbone network, wherein the backbone network includes a high-resolution network structure and a multi-scale receptive field; Constructing a feature fusion layer of the second segmentation model, inputting a plurality of feature maps of different scales into the feature fusion layer, and obtaining a plurality of deep feature maps of different scales; Constructing a decoder of the second segmentation model, and performing feature fusion on the bottom-layer feature map and the deep-layer feature map through the decoder to obtain an output mask; Constructing an output head of the second segmentation model, the deep feature map passes through the output head to obtain a suggested target box, the suggested target box passes through a layer of convolution to obtain an attention score, and finally passes through an activation function to obtain a mask probability map; A proposed mask is obtained according to the proposed target box and the output mask, a pooling operation is performed on the proposed mask, and a segmentation result is obtained according to a result of the pooling operation and the mask probability map.

2. The method according to claim 1, characterized in that The preprocessing of the flotation foam image to obtain the preprocessed flotation foam image includes: Calculating the fuzziness of each collected flotation foam image based on a fuzziness evaluation function, screening the flotation foam images according to the fuzziness, and obtaining a clear flotation foam image; After adaptive brightness is performed on the clear flotation foam image, contrast enhancement is performed to obtain the pre-processed flotation foam image.

3. The method according to claim 2, characterized in that After the clear flotation foam image is adaptively brightened, contrast enhancement is performed, including: Dividing the input clear flotation froth image into non-overlapping sub-blocks of equal size; Calculate the histogram and clipping threshold for each sub-block; Redistributing the redundant pixels in each sub-block that are larger than the clipping threshold to complete adaptive brightness; Each word block after adaptive brightness is histogram equalized, and the grayscale value of each sub-block after histogram equalization is reconstructed by bilinear interpolation to achieve contrast enhancement.

4. The method according to claim 1, characterized in that: The visual cue includes a rectangular box, a point or a mask, and the visual cue is annotated on the instance of the first flotation foam image by using the first object detection model and the initial visual cue to obtain a second flotation foam image, including: The first object detection model marks instances in the first flotation foam image that are similar to the marked instances as corresponding rectangular boxes, points or masks based on the rectangular boxes, points or masks of the marked instances in the first flotation foam image.

5. The method according to claim 1, characterized in that The training of a second target detection model according to the incompletely labeled target detection dataset includes: Constructing the backbone network structure, feature fusion structure and output head of the second target detection model according to the incompletely labeled target detection data set; Inputting the incompletely labeled object detection data set into the backbone network structure of the second object detection model to obtain a plurality of feature maps; Inputting the plurality of feature maps into the feature fusion structure to obtain a feature fusion map; The feature fusion map is input into the output head of the second object detection model to output the visual hint of the instance.

6. The method according to claim 1, characterized in that The updating of the annotated flotation foam image to the incompletely annotated target detection data set includes: The output instances with confidence levels higher than the confidence threshold are added as annotation results of the flotation foam images to the original incompletely annotated object detection dataset to obtain an updated incompletely annotated object detection dataset.

7. A semi-automatic flotation foam image annotation device based on prompt learning, characterized in that: The device comprises: An acquisition module, used for acquiring a flotation foam image, and preprocessing the flotation foam image to obtain a preprocessed flotation foam image; a labeling module, for constructing a first target detection model, obtaining a first flotation foam image in which an initial visual cue is manually labeled for a single instance in the preprocessed flotation foam image, and labeling the instance of the first flotation foam image with a visual cue by using the first target detection model and the initial visual cue to obtain a second flotation foam image; A first acquisition module is used to acquire a third flotation foam image that is manually supplemented with annotated visual cues for the second flotation foam image, and to construct an incompletely annotated target detection data set based on the third flotation foam image; A first training module is used to train a second target detection model according to the incompletely labeled target detection data set to obtain a second target detection model; A first loop module is used to repeatedly execute the steps of annotating visual cues on a new flotation foam image by the second target detection model, updating the annotated flotation foam image to the incompletely annotated target detection data set, and repeatedly training the second target detection model according to the updated incompletely annotated target detection data set until a preset number of executions is reached, and finally obtaining an optimized second target detection model and an updated incompletely annotated target detection data set; a segmentation module, configured to perform contour segmentation using the annotated instances of the flotation foam image in the incompletely annotated object detection dataset as visual cues through a first segmentation model to obtain a first incompletely annotated segmented image; A second acquisition module is used to acquire a second incompletely labeled segmented image that is obtained by manually supplementing and correcting the labeled instances in the first incompletely labeled segmented image, and to construct an incompletely labeled segmented image dataset based on the second incompletely labeled segmented image; A second training module is used to train a second segmentation model according to the incompletely labeled segmented image data set to obtain a second segmentation model; A second loop module is used to repeatedly execute: contour segmentation of a new flotation foam image by the second segmentation model, updating the flotation foam image after contour segmentation to the incompletely labeled segmented image dataset, and repeatedly training the second segmentation model according to the updated incompletely labeled segmented image dataset until a preset number of executions is reached, and finally obtaining an optimized second segmentation model and an updated incompletely labeled segmented image dataset; The step of training a second segmentation model according to the incompletely labeled segmented image dataset to obtain a second segmentation model comprises: Constructing a backbone network of the second segmentation model, and obtaining a plurality of underlying feature maps of different scales through the backbone network, wherein the backbone network includes a high-resolution network structure and a multi-scale receptive field; Constructing a feature fusion layer of the second segmentation model, inputting a plurality of feature maps of different scales into the feature fusion layer, and obtaining a plurality of deep feature maps of different scales; Constructing a decoder of the second segmentation model, and performing feature fusion on the bottom-layer feature map and the deep-layer feature map through the decoder to obtain an output mask; Constructing an output head of the second segmentation model, the deep feature map passes through the output head to obtain a suggested target box, the suggested target box passes through a layer of convolution to obtain an attention score, and finally passes through an activation function to obtain a mask probability map; A proposed mask is obtained according to the proposed target box and the output mask, a pooling operation is performed on the proposed mask, and a segmentation result is obtained according to a result of the pooling operation and the mask probability map.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed on the processor, the semi-automatic flotation foam image labeling method based on prompt learning according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium, characterized in that: The computer program is stored therein, and when the computer program is run on a processor, the method for semi-automatic annotation of flotation foam images based on prompt learning according to any one of claims 1 to 6 is executed.

Images