A text-guided weakly labeled flotation foam segmentation and counting method and system

Abstract

The present invention provides a text-guided weakly labeled flotation foam segmentation and counting method and system, which relates to the field of artificial intelligence and computer vision technology. The method includes: collecting images of foam in a flotation pool, performing preprocessing and point-level annotation, collecting text descriptions of flotation foam and performing preprocessing; inputting the preprocessed image into a point heat map generation module, identifying key points in the image, and forming a bounding box of a preset size with the key points as the center; inputting the key points and the bounding box into a hierarchical mask generation module based on weak annotation to guide it to generate hierarchical masks of the whole, part and sub-part; inputting the text description and a few-sample example image into a visual-language feature mapping module, mapping the two into the same embedding space for feature splicing, and obtaining classification guidance features; using the hierarchical mask to extract regional features, and aligning the similarity with the classification guidance features, thereby achieving accurate segmentation and counting of specific categories of foam in the image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a text-guided weakly labeled flotation foam segmentation and counting method and system. Background Art

[0002] Mineral resources need to be processed to be converted into usable industrial raw materials. Among them, mineral processing is an essential key step that can effectively improve the utilization efficiency of ore. In industry, flotation is a common mineral sorting method. Air is injected into the slurry to generate bubbles, and a frother is used to stabilize these bubbles to form foam. The flotation method uses the difference in surface hydrophilicity and hydrophobicity of mineral particles to make hydrophobic mineral particles adhere to the foam, thereby achieving the separation of different minerals. At this time, the characteristics of the foam become particularly important, which directly determines the efficiency and sorting accuracy of the flotation process. The stability, size distribution and color of the foam can reflect the flotation effect. Stable foam can effectively carry mineral particles, and smaller bubbles can provide a larger surface area, thereby improving the attachment efficiency of mineral particles. The color change of the foam is often related to the type of mineral, surface properties or the use of flotation reagents. Therefore, the characteristics of the foam are crucial to optimize the flotation process and improve the flotation efficiency.

[0003] Traditional flotation methods mainly rely on manual observation of foam characteristics for subsequent operations, which requires a wealth of experience and is difficult to ensure the consistency and accuracy of decision-making. Although deep learning-based visual methods can use segmentation technology to represent the size of foam, the generalization ability of deep learning methods is poor when the foam morphology is complex or the lighting conditions change significantly. It is difficult to simultaneously complete the segmentation and counting tasks under the specified foam morphology under complex and changeable working conditions.

[0004] In general, the existing technologies still have the following problems: 1) Traditional methods rely on manual operation and subjective judgment, resulting in low efficiency and being easily affected by personal experience; existing deep learning models are difficult to perform diverse counting statistics on segmentation results under the limitations of foam forms such as color and shape, and cannot achieve efficient coordination of segmentation and counting in the same model. 2) The high density of foam greatly increases the complexity and difficulty of instance-level annotation, which is mainly manifested in the overlap between objects, the irregularity of foam shapes, the diversity of sizes, and their dynamic changes in images, all of which bring challenges to instance-level annotation; since instance-level annotation requires accurate pixel-level annotation of each object, this is difficult to achieve in complex situations and actual operations. 3) Flotation foam image segmentation methods based on deep learning are usually trained in a single scene, which limits their generalization ability in new scenes; due to the lack of diversity in training data and the narrow coverage of scene features, these models often find it difficult to effectively adapt to new environments, resulting in a significant decrease in segmentation and counting performance.

[0005] Therefore, the current industry urgently needs an automated method that can improve segmentation and counting accuracy and maintain good performance in diverse and dynamically changing environments. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a text-guided weakly labeled flotation foam segmentation and counting method and system to solve the problems of instance-level labeling difficulties, inability to effectively unify segmentation and counting tasks, and insufficient model generalization ability in the prior art under high foam density conditions.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] On the one hand, a text-guided weakly labeled flotation foam segmentation and counting method is provided, the method comprising the following steps:

[0009] S1, collect images of foam in the flotation tank and collect text descriptions of the flotation foam;

[0010] S2, preprocessing and point-level labeling of the collected images, and preprocessing the collected text descriptions; integrating the preprocessed text descriptions with the images to form a multimodal dataset;

[0011] S3, inputting the preprocessed image into a point heat map generation module, identifying key points in the image, and forming a bounding box of a preset size with the key points as the center;

[0012] S4, inputting the key points and bounding boxes into a weakly labeled hierarchical mask generation module to guide it to generate hierarchical masks of the whole, part and sub-part;

[0013] S5, inputting the preprocessed text description and the few-sample example images into a visual-language feature mapping module, mapping the two into the same embedding space for feature concatenation, and obtaining classification guidance features for the foam mask; the few-sample example images are sample example images within ten;

[0014] S6. Utilize the hierarchical mask to extract regional features, align similarity with the classification guidance features after processing, identify the foam mask corresponding to the text description-few sample example image, and finally segment and count the foam of a specific category in the image.

[0015] Optionally, in step S1, an image acquisition device is installed above the flotation tank to acquire images of the foam in the flotation tank; and at the same time, a text description of the color, size, density and shape of the flotation foam by the operator is collected.

[0016] Optionally, in step S2, preprocessing the acquired image includes:

[0017] Enhance the contrast of the image through histogram equalization; use Gaussian filtering to remove noise in the image; crop and scale the image, unify the image size, and perform normalization; apply the HED edge detection algorithm to identify the contour of the foam in the image, and combine the Laplacian operator to enhance the contour features;

[0018] Preprocessing of the collected text descriptions includes:

[0019] Perform natural language processing on the collected text descriptions, including word segmentation, stop word removal, named entity recognition, and removal of invalid and redundant words.

[0020] Optionally, in step S2, performing point-level labeling includes:

[0021] The given image is processed to generate masks of all objects in the image; based on the generated masks, the center point of each mask is calculated, and the properties of the generated masks are used for filtering to remove unnecessary masks and retain the masks containing the target objects; the center point is extracted based on the filtered mask and corrected at the same time. The final center point represents the position of each object in the image.

[0022] Optionally, in step S3, the processing process of the point heat map generation module includes:

[0023] For a given input image, feature extraction is performed through an image encoder, and then mask features are generated through a mask decoder; the mask features are fused with the shallow features and deep features extracted by the image encoder through a convolution operation, and the mask features are processed by a multi-layer perceptron, and a spatial point-by-point product operation is performed with the fused features to generate a target heat map;

[0024] In the inference stage, a 3x3 maximum pooling operation is applied to the target heat map to divide it into multiple rectangular areas, and the maximum value is output for each rectangular area to extract the key points in the target heat map, and a 16×16 bounding box is formed with each key point as the center; the top K key points with scores higher than the threshold are selected to guide the generation of the mask of the corresponding image.

[0025] Optionally, in step S4, the processing of the weakly labeled hierarchical mask generation module includes:

[0026] The key points and bounding boxes are feature concatenated through a cue encoder and a learnable mask to generate a visual reference cue embedding; the input image is passed through an image encoder and an image embedding is obtained through a Transformer-based feature extractor; the image embedding and the visual reference cue are embedded in the input image-cue bidirectional attention mechanism to obtain enhanced image features and fused features;

[0027] The enhanced image features are input into the convolutional layer for processing, and the enhanced fused features are input into the multi-layer perceptron for processing; the processed image features and fused features are combined through dot product operations to generate hierarchical masks of the whole, parts and sub-parts.

[0028] Optionally, in step S5, the visual-language feature mapping module includes a language-image pre-training model, and the language-image pre-training model includes a language pre-training part and an image pre-training part;

[0029] The processing of the vision-language feature mapping module includes:

[0030] Input the text description into the language pre-training part, and input the few sample example images annotated with example boxes into the image pre-training part after scale alignment; map the outputs of the language pre-training part and the image pre-training part into the same embedding space for feature splicing to obtain classification guidance features;

[0031] Wherein, a predetermined number of images are selected as the few-sample example images according to the difficulty of segmenting the object to be segmented and the difficulty of acquiring the image.

[0032] Optionally, the step S6 specifically includes:

[0033] Extract regional features from the generated hierarchical mask and process them through target region alignment and two-layer multi-layer perceptron, and align the processed regional features with the concatenated classification guidance features by similarity;

[0034] For the foam mask obtained from the same point, the classification weight is constructed from the image embedding of the corresponding image area, and the classification weight is optimized according to the prediction result; according to the optimized classification weight and the similarity of the regional features, it is determined whether each image area belongs to the target object, thereby identifying the category of the object;

[0035] Non-maximum suppression is applied to all masks to remove duplicate masks and ensure that each object is counted only once. The number of detected objects is calculated by thresholding the score, that is, the number of masks with scores higher than the preset threshold. The counting result is output together with the category and location coordinates of the object to complete the segmentation and counting tasks of specific categories of foam in the image.

[0036] Optionally, the method further comprises:

[0037] S7, dividing the multimodal dataset into a training set, a test set, and a validation set in proportion, and training and optimizing the model in combination with a loss function;

[0038] S8. Use the average precision AP and the average precision AP50 when the IoU threshold is 0.5 as evaluation indicators to evaluate and verify the performance of the model segmentation task;

[0039] The mean absolute error (MAE) and root mean square error (RMSE) are used as evaluation indicators to evaluate and verify the model counting task performance.

[0040] On the other hand, a text-guided weakly labeled flotation foam segmentation and counting system is provided for implementing any of the above methods, the system comprising:

[0041] A data acquisition module for collecting images of the froth in the flotation tank and a text description of the flotation froth;

[0042] The data processing module is used to preprocess and annotate the collected images at the point level, and to preprocess the collected text descriptions; the preprocessed text descriptions are integrated with the images to form a multimodal data set;

[0043] A point heat map generation module is used to process the input pre-processed image, identify key points in the image, and form a bounding box of a preset size with the key points as the center;

[0044] A weakly annotated hierarchical mask generation module for generating whole, part, and sub-part hierarchical masks based on input keypoints and bounding boxes as guidance;

[0045] The visual-language feature mapping module processes the input pre-processed text description and few-sample example images, maps them into the same embedding space for feature concatenation, and obtains the classification guidance features for the foam mask;

[0046] The classification and counting module is used to extract regional features using the hierarchical mask, align the similarity with the classification guidance features after processing, identify the foam mask corresponding to the text description-few sample example image, and thus segment and count the specific category of foam in the image.

[0047] In another aspect, an electronic device is provided, the electronic device comprising:

[0048] processor;

[0049] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the weakly labeled flotation foam segmentation and counting method as guided by the above text are implemented.

[0050] On the other hand, a computer-readable storage medium is provided, in which program codes are stored. The program codes can be called by a processor to execute the steps of the above-mentioned text-guided weakly labeled flotation foam segmentation and counting method.

[0051] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0052] (1) The present invention effectively integrates the feature information of the visual basic model and the language text model, realizes the coordinated processing of foam segmentation and counting tasks under limited foam morphology, and improves the task accuracy of the model in different scenarios.

[0053] (2) The present invention designs a hierarchical mask generation module based on weak annotation, which only requires semi-automatic point-level annotation, without pixel-level instance-level annotation for each bubble. This point-level annotation method can identify the boundary and morphological features of the target object with only a small amount of sample annotation information, thus achieving efficient and accurate image segmentation.

[0054] (3) The present invention uses a visual-linguistic feature mapping module to effectively fuse the few-sample example image information and text information with class-independent segmentation features, thereby achieving pixel-level category prediction and accurate foam counting. This method can adapt to different foam forms and environmental backgrounds without a large amount of labeled data, effectively improving the generalization ability of the model in new environments or different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a flow chart of a text-guided weakly labeled flotation foam segmentation and counting method provided by an embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the principle of a text-guided weakly labeled flotation foam segmentation and counting method provided by an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of a point heat map generation module provided in an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of a hierarchical mask generation module based on weak annotation provided by an embodiment of the present invention;

[0060] Figure 5 It is a structural schematic diagram of a text-guided weakly labeled flotation foam segmentation and counting system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the word "exemplary" is used to present concepts in a specific way.

[0063] The embodiment of the present invention provides a text-guided weakly labeled flotation foam segmentation and counting method, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 and Figure 2 As shown, the processing flow of the method may include the following steps:

[0064] S1. Collect images of foam in a flotation tank and simultaneously collect text descriptions of the flotation foam.

[0065] In this step, an image acquisition device is installed above the flotation cell to collect images of the foam in the flotation cell to ensure that the detailed features of the foam, including the color, size, density and shape of the foam, can be clearly captured for subsequent identification. At the same time, the operator's text description of the color, size, density and shape of the flotation foam, or language description, is collected.

[0066] S2. Preprocess and point-level annotate the collected images, and preprocess the collected text descriptions; integrate the preprocessed text descriptions with the images to form a multimodal dataset.

[0067] In this step, preprocessing the collected images includes:

[0068] The contrast of the image is enhanced by histogram equalization to more clearly distinguish the foam and the background. Gaussian filtering is used to remove noise from the image to reduce the impact on subsequent processing. The image is cropped and scaled to unify the image size and normalized to improve the stability of model training. The HED edge detection algorithm is applied to identify the contour of the foam in the image, and the Laplace operator is used to enhance the contour features to make the contour more obvious.

[0069] These steps help improve image quality, reduce noise, and extract useful features, thus providing a better basis for subsequent image analysis and processing.

[0070] Preprocessing of the collected text descriptions includes:

[0071] Perform natural language processing on the collected text descriptions, including word segmentation, stop word removal, named entity recognition, etc., to remove invalid and redundant words and extract key information.

[0072] In the embodiment of the present invention, semi-automatic point-level annotation is adopted for a given image, specifically including:

[0073] The given image is processed to generate masks of all objects in the image; based on the generated masks, the center point of each mask is calculated, and the properties of the generated masks are used for filtering, removing unnecessary masks and retaining the masks that are most likely to contain the target object. The center point is extracted based on the filtered mask, and the center point is corrected at the same time, including correcting and supplementing the wrong or missing points. The final center point represents the position of each object (foam) in the image.

[0074] S3. Input the preprocessed image into a point heat map generation module, identify key points in the image, and form a bounding box of a preset size with the key points as the center.

[0075] In this step, if Figure 3 As shown in FIG. 1 , the processing process of the point heat map generation module includes:

[0076] For a given input image , feature extraction is performed through the image encoder, and then mask features are generated through the mask decoder; the mask features are fused with the shallow features and deep features extracted by the image encoder through convolution operations, and the mask features are processed by a multi-layer perceptron and spatially dot-by-dot operations are performed with the fused features to generate a target heat map , where s is the output stride, H and W are the height and width of the heatmap, respectively.

[0077] The corrected mask center point is used as the supervision signal of the target heat map to help the point decoder learn how to more accurately predict the target heat map from the image features, thereby improving the prediction accuracy of the object position. At the same time, the point-by-point mean squared error (MSE) is used as the loss function. Train to improve the detection performance of small objects.

[0078] In the inference phase, a 3x3 max pooling operation is applied to the target heat map, dividing it into multiple rectangular regions, and outputting the maximum value for each rectangular region. Specifically, a fixed 3×3 window is used to slide on the feature map, and the maximum value in each window is selected as the output, thereby extracting the key points in the target heat map and forming a 16×16 bounding box centered on each key point.

[0079] Furthermore, the first K key points with scores higher than the threshold are selected to guide the generation of the mask of the corresponding image. Through the above steps, key information can be effectively extracted from the image, thereby accurately guiding the segmentation and counting of the target object.

[0080] S4. Input the key points and bounding boxes into a weakly labeled hierarchical mask generation module to guide it to generate hierarchical masks of the whole, part and sub-part.

[0081] In this step, a high-quality hierarchical object mask is generated based on the given keypoint and bounding box cues, including each cue for overall, partial, and sub-part segmentation. The point cues come from the K keypoints obtained in step S3. Since the model usually cannot segment small objects in crowded scenes, an additional 16×16 bounding box is chosen here around each keypoint to improve the segmentation of small objects.

[0082] like Figure 4 As shown in FIG. 1 , the processing process of the weakly labeled hierarchical mask generation module includes:

[0083] The key points and bounding boxes are feature concatenated through a cue encoder and a learnable mask to generate a visual reference cue embedding; the input image is passed through an image encoder and an image embedding is obtained through a Transformer-based feature extractor; the image embedding and the visual reference cue are embedded in the input image-cue bidirectional attention mechanism to obtain enhanced image features and fused features;

[0084] The enhanced image features are input into the convolutional layer for processing to refine and enhance the expressiveness of the features; the enhanced fused features are input into the multi-layer perceptron for processing to help the model learn the complex relationship between features; the processed image features and fused features are combined through dot product operations to generate hierarchical masks of the whole, part, and sub-part. These masks are then used for classification and counting to identify and count the bubbles in the image.

[0085] S5. Input the preprocessed text description and the few-sample example image into the visual-language feature mapping module, map them into the same embedding space for feature concatenation, and obtain the classification guidance features for the foam mask.

[0086] The few-sample example images refer to sample example images within ten. The visual-language feature mapping module includes a language-image pre-training model, which includes a language pre-training part and an image pre-training part. By learning large-scale image-text pairs, the visual features can be aligned with the text description for object detection and classification.

[0087] In this step, the processing of the visual-linguistic feature mapping module includes:

[0088] Input the text description into the language pre-training part to obtain features , C is the number of representations of the text query, and D is the dimension of the embedding. Among them, the text description is a necessary part for subsequent classification and counting.

[0089] The few sample example images annotated with example boxes are scale-aligned and input into the image pre-training part to obtain , B is the number of images selected. A predetermined number of images are selected as the few sample example images according to the difficulty of segmenting the object to be segmented and the difficulty of acquiring the image.

[0090] The outputs of the language pre-training part and the image pre-training part are mapped into the same embedding space for feature concatenation to obtain classification guidance features. .

[0091] The classification weights are constructed based on the input text description (category name, zero sample) or few-sample example image (example bounding box, few sample). In the case of zero sample, the model relies on text description to identify new categories; in the case of few samples, the model recognizes the key attributes of the target category by understanding the semantic information in the text description, thereby achieving effective learning in limited samples and improving classification accuracy. This method of combining language information and visual information greatly enhances the model's ability to recognize new categories and improves the model's generalization ability.

[0092] S6. Utilize the hierarchical mask to extract regional features, align similarity with the classification guidance features after processing, identify the foam mask corresponding to the text description-few sample example image, and thus segment and count the foam of a specific category in the image.

[0093] In this step, the image / text embedding of the language-image pre-trained model is used as a classifier to classify the generated hierarchical masks and distinguish the required objects. Specifically, it includes:

[0094] The regional features are extracted from the generated hierarchical mask and processed by Region of Interest Align (ROI Align) and two-layer Multi-Layer Perceptron (MLP). The processed regional features are similarity aligned with the spliced ​​classification guidance features.

[0095] For the foam masks obtained from the same point, classification weights are constructed from the image embeddings of the corresponding image regions, and the classification weights are optimized according to the prediction results; based on the optimized classification weights and the similarity of regional features, it is determined whether each image region belongs to the target object, thereby identifying the category of the object.

[0096] Non-Maximum Suppression (NMS) is applied to all masks to remove duplicate masks and ensure that each object is counted only once. The IoU threshold of NMS is set to 0.5. The number of detected objects is calculated by thresholding the score, that is, the number of masks with scores higher than the preset threshold. The counting result is output together with the category and position coordinates of the object to complete the segmentation and counting of specific categories of foam in the image, that is, the foam segmentation and counting task under limited foam morphology.

[0097] Furthermore, the method further comprises:

[0098] S7. Divide the multimodal dataset into a training set, a test set, and a validation set in proportion, and train and optimize the model in combination with a loss function.

[0099] During the model training process, the network parameters are optimized by back propagation by combining the point hint loss and the classification loss. Through the combination of these two loss functions, the model can balance the positioning accuracy, classification ability and adaptability to small targets during the training process, thereby improving the overall performance. During the training process, the semi-automatic point-level labeled supervised point decoder generates the target heat map, which is then screened according to the set K value (number of key points) and the heat map threshold to obtain the final key points. During the training process, 256 proposals and 16 pairs of proposals for each sample are randomly selected for classifier training to ensure 25% positive sample proposals. Use the Adaptive Moment Estimation (Adam) optimizer, set a fixed learning rate and weight decay, and perform multiple iterations of training.

[0100] S8. Use Average Precision (AP) and Average Precision at IoU=0.5 (AP50) as evaluation indicators to evaluate and verify the performance of the model segmentation task. AP measures the average precision of the model at different recall levels and is used to evaluate the detection ability of the model; AP50 is the average precision at a recall rate of 50%, which is used to measure the performance of the model under higher recall requirements.

[0101] The mean absolute error (MAE) and root mean square error (RMSE) are used as evaluation indicators to evaluate and verify the performance of the model counting task.

[0102] In the embodiment of the present invention, the feature information of the visual basic large model and the language text large model are combined to achieve the unification of segmentation and counting tasks and the improvement of accuracy; only semi-automatic point-level annotation is required, and the target heat map of all foreground objects is generated by the point heat map generation module, and converted into key point coordinates and bounding boxes centered on the key points. At the same time, a weakly annotated hierarchical mask generation module based on points and boxes is designed to automatically infer the complete area of ​​all foreground objects, thereby achieving efficient and accurate image segmentation; the features of the few sample example images and text descriptions are spliced ​​and aligned with the regional features of the segmentation mask by using the visual-language feature mapping module, so as to classify the foreground mask and obtain the number and category of foams, effectively achieving the unification of segmentation and counting under the limitation of diverse foam morphological descriptions, enhancing the generalization ability of the model in new environments or different scenarios, and further improving the accuracy of image segmentation and counting tasks in flotation processes.

[0103] Accordingly, an embodiment of the present invention also provides a text-guided weakly labeled flotation foam segmentation and counting system, such as Figure 5 As shown, the system includes:

[0104] A data acquisition module for collecting images of the froth in the flotation tank and a text description of the flotation froth;

[0105] The data processing module is used to preprocess and annotate the collected images at the point level, and to preprocess the collected text descriptions; the preprocessed text descriptions are integrated with the images to form a multimodal data set;

[0106] A point heat map generation module is used to process the input pre-processed image, identify key points in the image, and form a bounding box of a preset size with the key points as the center;

[0107] A weakly annotated hierarchical mask generation module for generating whole, part, and sub-part hierarchical masks based on input keypoints and bounding boxes as guidance;

[0108] The visual-language feature mapping module processes the input pre-processed text description and few-sample example images, maps them into the same embedding space for feature concatenation, and obtains the classification guidance features for the foam mask;

[0109] The classification and counting module is used to extract regional features using the hierarchical mask, align the similarity with the classification guidance features after processing, identify the foam mask corresponding to the text description-few sample example image, and thus accurately segment and count the specific category of foam in the image.

[0110] For ease of explanation, Figure 5 Only the main components of the system are shown. The system of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0111] The present invention aims at the problem of insufficient coordination between segmentation and counting tasks and difficulty in instance-level annotation under high-density foam conditions. It can integrate the features of large visual and language models to achieve unified processing of segmentation and counting. Based on the annotation of foam key points, it can automatically infer the complete area of ​​the foreground object, synchronize different scene features, and combine the visual-language feature mapping module to classify the foreground mask to obtain the number and category of foam. The present invention can significantly improve the generalization ability of the model in a diversified environment and improve the processing accuracy and automation efficiency of the flotation process.

[0112] In an exemplary embodiment, the present invention further provides an electronic device, the electronic device comprising:

[0113] processor;

[0114] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the weakly labeled flotation foam segmentation and counting method as guided by the above text are implemented.

[0115] In an exemplary embodiment, the present invention further provides a computer-readable storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement the steps of the weakly labeled flotation froth segmentation and counting method as guided by the above text. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0116] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0117] References in the specification to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments", etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0118] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0119] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0120] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0121] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0124] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0125] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text-guided weakly labeled flotation foam segmentation and counting method, characterized in that: The following steps are involved: S1, collect images of foam in the flotation tank and collect text descriptions of the flotation foam; S2, preprocessing and point-level labeling of the collected images, and preprocessing the collected text descriptions; integrating the preprocessed text descriptions with the images to form a multimodal dataset; S3, inputting the preprocessed image into a point heat map generation module, identifying key points in the image, and forming a bounding box of a preset size with the key points as the center; S4, inputting the key points and bounding boxes into a weakly labeled hierarchical mask generation module to guide it to generate hierarchical masks of the whole, part and sub-part; S5, inputting the preprocessed text description and the few-sample example images into a visual-language feature mapping module, mapping the two into the same embedding space for feature concatenation, and obtaining classification guidance features for the foam mask; the few-sample example images are sample example images within ten; S6. Utilize the hierarchical mask to extract regional features, align similarity with the classification guidance features after processing, identify the foam mask corresponding to the text description-few sample example image, and thus segment and count the foam of a specific category in the image.

2. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: In step S1, an image acquisition device is installed above the flotation tank to acquire images of the froth in the flotation tank; and at the same time, a text description of the color, size, density and shape of the flotation froth by an operator is collected.

3. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: In step S2, preprocessing the collected image includes: Enhance the contrast of the image through histogram equalization; use Gaussian filtering to remove noise in the image; crop and scale the image, unify the image size, and perform normalization; apply the HED edge detection algorithm to identify the contour of the foam in the image, and combine the Laplacian operator to enhance the contour features; Preprocessing of the collected text descriptions includes: Perform natural language processing on the collected text descriptions, including word segmentation, stop word removal, named entity recognition, and removal of invalid and redundant words.

4. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: In step S2, performing point-level labeling includes: The given image is processed to generate masks of all objects in the image; based on the generated masks, the center point of each mask is calculated, and the properties of the generated masks are used for filtering to remove unnecessary masks and retain the masks containing the target objects; the center point is extracted based on the filtered mask and corrected at the same time. The final center point represents the position of each object in the image.

5. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: In step S3, the processing process of the point heat map generation module includes: For a given input image, feature extraction is performed through an image encoder, and then mask features are generated through a mask decoder; the mask features are fused with the shallow features and deep features extracted by the image encoder through a convolution operation, and the mask features are processed by a multi-layer perceptron, and a spatial point-by-point product operation is performed with the fused features to generate a target heat map; In the inference stage, a 3x3 maximum pooling operation is applied to the target heat map to divide it into multiple rectangular areas, and the maximum value is output for each rectangular area to extract the key points in the target heat map, and a 16×16 bounding box is formed with each key point as the center; the top K key points with scores higher than the threshold are selected to guide the generation of the mask of the corresponding image.

6. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: In step S4, the processing process of the weakly labeled hierarchical mask generation module includes: The key points and bounding boxes are feature concatenated through a cue encoder and a learnable mask to generate a visual reference cue embedding; the input image is passed through an image encoder and an image embedding is obtained through a Transformer-based feature extractor; the image embedding and the visual reference cue are embedded in the input image-cue bidirectional attention mechanism to obtain enhanced image features and fused features; The enhanced image features are input into the convolutional layer for processing, and the enhanced fused features are input into the multi-layer perceptron for processing; the processed image features and fused features are combined through dot product operations to generate hierarchical masks of the whole, parts and sub-parts.

7. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: In step S5, the visual-language feature mapping module includes a language-image pre-training model, and the language-image pre-training model includes a language pre-training part and an image pre-training part; The processing of the vision-language feature mapping module includes: Input the text description into the language pre-training part, and input the few sample example images annotated with example boxes into the image pre-training part after scale alignment; map the outputs of the language pre-training part and the image pre-training part into the same embedding space for feature splicing to obtain classification guidance features; Wherein, a predetermined number of images are selected as the few-sample example images according to the difficulty of segmenting the object to be segmented and the difficulty of acquiring the image.

8. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: The step S6 specifically includes: Extract regional features from the generated hierarchical mask and process them through target region alignment and two-layer multi-layer perceptron, and align the processed regional features with the concatenated classification guidance features by similarity; For the foam mask obtained from the same point, the classification weight is constructed from the image embedding of the corresponding image area, and the classification weight is optimized according to the prediction result; according to the optimized classification weight and the similarity of the regional features, it is determined whether each image area belongs to the target object, thereby identifying the category of the object; Non-maximum suppression is applied to all masks to remove duplicate masks and ensure that each object is counted only once. The number of detected objects is calculated by thresholding the score, that is, the number of masks with scores higher than the preset threshold. The counting result is output together with the category and location coordinates of the object to complete the segmentation and counting tasks of specific categories of foam in the image.

9. The text-guided weakly labeled flotation foam segmentation and counting method according to claim 1, characterized in that: The method further comprises: S7, dividing the multimodal dataset into a training set, a test set, and a validation set in proportion, and training and optimizing the model in combination with a loss function; S8. Use the average precision AP and the average precision AP50 when the IoU threshold is 0.5 as evaluation indicators to evaluate and verify the performance of the model segmentation task; The mean absolute error (MAE) and root mean square error (RMSE) are used as evaluation indicators to evaluate and verify the model counting task performance.

10. A text-guided weakly labeled flotation froth segmentation and counting system, the system being used to implement the method according to any one of claims 1 to 9, characterized in that: The system comprises: A data acquisition module for collecting images of the froth in the flotation tank and a text description of the flotation froth; The data processing module is used to preprocess and annotate the collected images at the point level, and to preprocess the collected text descriptions; the preprocessed text descriptions are integrated with the images to form a multimodal data set; A point heat map generation module is used to process the input pre-processed image, identify key points in the image, and form a bounding box of a preset size with the key points as the center; A weakly annotated hierarchical mask generation module for generating whole, part, and sub-part hierarchical masks based on input keypoints and bounding boxes as guidance; The visual-language feature mapping module processes the input pre-processed text description and few-sample example images, maps them into the same embedding space for feature concatenation, and obtains the classification guidance features for the foam mask; The classification and counting module is used to extract regional features using the hierarchical mask, align the similarity with the classification guidance features after processing, identify the foam mask corresponding to the text description-few sample example image, and thus segment and count the specific category of foam in the image.

Images