Multi-stage deep learning training method based on multi-modal data

Through the multi-stage deep learning training method and a three-stage data engine, combined with SAM model and multi-modal data processing technology, the problem of insufficient efficiency and accuracy of multi-modal data processing in the existing technology is solved, and efficient and highly accurate image segmentation effect is achieved.

CN119646506APending Publication Date: 2025-03-18HANGZHOU DIANZI UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411672627.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-21
Publication Date
2025-03-18

AI Technical Summary

Technical Problem

The prior art has problems of insufficient efficiency and accuracy in multimodal data processing and precise segmentation, which is difficult to meet the needs of modern deep learning models for processing complex data.

Method used

A multi-stage deep learning training method is designed to improve data quality and processing efficiency through a three-stage data engine and SAM model, combining multi-modal data fusion, feature extraction, effective segmentation mask generation and multi-candidate mask processing.

Benefits of technology

It significantly improves the accuracy and efficiency of image segmentation, enhances the model's processing ability for complex scenarios, ensures high quality and high accuracy of masks, and is suitable for a variety of application scenarios such as autonomous driving and medical image analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119646506A_ABST
    Figure CN119646506A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-modal data-based multi-stage deep learning training method, which is used for improving the effect of a model in multi-modal data training by designing a three-stage data engine and combining an SAM model. The method is particularly suitable for various image segmentation tasks, especially zero sample migration is carried out on new data distribution, and the design of the three-stage data engine enables reasonability to be kept when input prompt is fuzzy or various explanations exist through interactive segmentation tasks in different stages, so that the method is more efficient and more reliable. And the model is excellent in the aspect of diversified image segmentation tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a multi-stage deep learning training method based on multimodal data, and belongs to the technical field of deep learning. Background Art

[0002] With the development of deep learning technology, data processing and model training methods continue to emerge. The core of deep learning is to simulate the processing methods of the human brain through algorithms to achieve the ability to learn and make judgments from large amounts of data. With the improvement of hardware performance and the optimization of algorithms, modern deep learning models are able to handle increasingly complex tasks, such as image and video analysis, natural language processing and other applications. However, with the increasing complexity of application scenarios, the development and processing of multimodal data has gradually been valued by people. At the same time, the traditional single data source and single task training methods can no longer meet modern needs.

[0003] Multimodal data fusion is an emerging solution that integrates information from different data sources (such as text, images, sounds, etc.) to more comprehensively understand and predict patterns in data. This method not only improves the generalization ability of the model, but also greatly enhances the adaptability and interpretability of the model to complex situations in the real world. How to apply this technology to model training has become a challenge.

[0004] The present invention aims to solve the main technical problems existing in the prior art, namely the main technical problems of deep learning models in multimodal data processing and accurate segmentation. By designing a three-stage data engine and combining the SAM model, different modal data are effectively utilized to systematically improve data quality and processing efficiency. The use of advanced segmentation algorithms and multi-candidate mask mechanisms enhances the model's ability to handle complex scenes. Combined with IoU prediction and non-maximum suppression technology, the accuracy and quality of the mask are significantly improved. This multi-stage and multi-modal training method provides a new solution to the limitations of existing technologies, which can effectively improve segmentation effects and training efficiency. Summary of the invention

[0005] In the current field of artificial intelligence and computer vision, the capacity and modality of data are constantly increasing, and the requirements for the accuracy and efficiency of model training are also constantly increasing. In order to overcome the shortcomings of existing training methods that are mostly efficient training of single modality data or inefficient training of multi-modality, which cannot adapt well to the requirements of industry development, the present invention provides a multi-stage deep learning training method based on multi-modal data, aiming to improve the accuracy and efficiency of image segmentation.

[0006] A multi-stage deep learning training method based on multimodal data, comprising the following steps:

[0007] Step 1: Data acquisition and preprocessing: Collect multimodal data from the Internet, databases, and laboratories, and optimize them through cleaning, standardization, and formatting processes;

[0008] In the preprocessing stage, multimodal data is collected from a variety of sources such as the Internet, databases, and laboratories, and optimized through a series of cleaning, standardization, and formatting processes. This includes removing background noise from images and videos, unifying data formats and adjusting resolutions, and performing grammatical correction and structural processing on text data. These steps ensure the consistency and quality of the data, providing high-quality input for model training, thereby enhancing the accuracy and robustness of the model, enabling it to be effective.

[0009] Step 2: Multimodal data fusion and feature extraction: The preprocessed multimodal data is integrated using a specific fusion algorithm, the image data is transformed in the frequency domain using fast Fourier transform technology, and feature extraction is performed in combination with text information; this step not only enhances the expressiveness of the data, but also provides rich input features for the SAM model.

[0010] Step 3: Generate effective segmentation masks and process multiple candidate masks: Use the SAM model to process the input multimodal data and automatically generate effective segmentation masks; these masks are evaluated by the mask quality prediction module built into the model to ensure that their overlap with the real object meets the preset standard. At the same time, the model outputs multiple candidate masks to deal with possible ambiguity, and filters out duplicate and low-confidence masks through techniques such as non-maximum suppression (NMS) to retain high-quality segmentation results.

[0011] The model is trained through three data engines. The data engine is a key component in the SAM model (SegmentAnything Model) training process. It is responsible for collecting and generating large-scale segmentation mask datasets for training models. The work of the data engine is divided into three stages: assisted manual annotation stage, semi-automatic stage and fully automatic stage.

[0012] Step 4: Data engine-assisted manual annotation stage: Use the browser-based interactive segmentation tool to annotate the target objects by explicitly clicking on the foreground and background points in the image. The SAM model provides real-time feedback and suggestions during the annotation process.

[0013] In the assisted manual annotation stage, professional annotators use a browser-based interactive segmentation tool to annotate the target objects by explicitly clicking on the foreground and background points in the image. To improve the quality of the annotation, the annotator can make adjustments using fine pixel-level "brush" and "eraser" tools. In addition, the system provides real-time feedback and suggestions during the annotation process through a model called SAM.

[0014] In the initial stage, the SAM model is trained based on a public segmentation dataset to ensure that the model has basic image recognition and segmentation capabilities. As data accumulates in actual applications, the SAM model will be retrained using only newly annotated mask data to adapt to specific annotation tasks and optimize performance. This continuous learning and adaptation process significantly improves the efficiency of annotation: the average annotation time per mask is reduced from 34 seconds to 14 seconds, and the average number of masks per image increases from 20 to 44. This improvement not only improves the efficiency of the workflow, but also improves the quality of the dataset and the final effect of model training by reducing annotation errors.

[0015] Step 5: Data engine semi-automatic stage: The system automatically identifies high-confidence masks, which are automatically generated by the bounding box detector. The mask image will be shown to the annotator, who will check and correct the automatically generated masks and manually annotate the remaining objects that were not detected by the system. The collected masks effectively expand the dataset.

[0016] The main purpose of the semi-automatic stage is to greatly improve the model's ability to segment various objects by increasing the diversity of masks. The workflow of this stage is carefully designed to ensure that training data is generated both efficiently and accurately.

[0017] First, the system automatically identifies high-confidence masks. These masks are automatically generated using an advanced bounding box detector that is trained on all first-stage masks using a generic “object” category to ensure coverage of a wide range of object types. This approach allows the model to identify the correct mask with high probability in the initial stage.

[0018] These pre-populated mask images are then shown to human annotators. In this step, the task of the human annotator is to review and correct these automatically generated masks and manually annotate those objects that were not detected by the system. This process not only improves the quality of the masks but also ensures the comprehensiveness and diversity of the data.

[0019] The 5.9 million masks collected during this phase effectively expanded the dataset, providing rich and diverse input for the training of deep learning models. Although the average annotation time per mask rose back to 34 seconds, this time investment ensured the high quality and comprehensiveness of the data.

[0020] Step 6: Data engine fully automatic stage: The model is significantly improved by accumulating a large number of diverse masks in the previous steps. The model uses a regular 32x32 grid as a prompt, and each grid point is used to predict the mask that may correspond to a valid object. The model also contains an IoU prediction module for screening out high-confidence masks;

[0021] The fully automatic stage of the data engine marks the apex of the training method, relying on key improvements in the first two steps to enable it to efficiently generate high-quality masks without human intervention.

[0022] First, the model is significantly improved by accumulating a large number of diverse masks in previous stages. This large dataset provides the model with rich learning materials, enabling it to adapt to various complex image conditions and object types. And a large amount of multimodal data develops a new blur perception model. This model is designed to handle blurry areas of images and can predict effective masks when visual information is incomplete.

[0023] To further optimize the accuracy and efficiency of mask generation, the model uses a regular 32x32 grid as a hint, and each grid point is used to predict a set of masks that may correspond to valid objects. In addition, the model contains an IoU (Intersection over Union) prediction module to filter out masks with high confidence. This module can also identify stable masks, which are determined by similar results obtained by processing probability maps under specific thresholds (0.5-δ to 0.5+δ).

[0024] To handle potential overlaps between masks and improve the quality of small-sized masks, the model applies non-maximum suppression (NMS) technology, which effectively filters out duplicate or low-quality masks, ensuring that each mask has high quality and accuracy.

[0025] This fully automatic stage successfully processed a large-scale dataset of 11 million images and generated a total of about 1.1 billion high-quality masks. This stage not only demonstrated the powerful capabilities of deep learning models in practical applications, but also proved the efficiency and reliability of multi-stage training methods when processing large-scale data.

[0026] Step 7: Quality Control and Model Optimization: Introduce strict quality control methods, such as IoU prediction and stability assessment, to ensure that the generated mask dataset meets the training requirements, and through iterative optimization, achieve efficient deep learning training for multimodal data.

[0027] The optimization through cleaning, standardization and formatting process in step 1 specifically includes: removing background noise in images and videos, unifying data formats to a unified standard, adjusting data resolution, and performing grammatical correction and structural processing on text data.

[0028] The step three specifically includes:

[0029] The SAM model is used to process the input multimodal data and automatically generate effective segmentation masks. The built-in mask quality prediction module evaluates the generated masks to ensure that their overlap with the real objects meets the preset standards. The model outputs multiple candidate masks at the same time and uses non-maximum suppression technology to screen the candidate masks, filtering out duplicate and low-confidence masks and retaining only high-quality segmentation results.

[0030] The step 4 specifically includes:

[0031] Using the browser-based interactive segmentation tool, you can annotate the target objects by explicitly clicking on the foreground and background points in the image. You can make adjustments using fine pixel-level "brush" and "eraser" tools. The SAM model provides real-time feedback and suggestions during the annotation process.

[0032] The step five specifically includes:

[0033] First, high-confidence masks are automatically identified and automatically generated by an advanced bounding box detector. The detector is trained on all first-stage masks and uses a general "object" category to ensure coverage of a wide range of object types. The pre-filled mask images are shown to annotators to inspect and correct these automatically generated masks and manually annotate objects that the system failed to detect.

[0034] The step six specifically includes:

[0035] A blur perception model was developed to handle blurry areas in images. A regular 32x32 grid was introduced as a hint. Each grid point was used to predict a set of masks that may correspond to valid objects. An IoU prediction module was added to filter out high-confidence masks and identify stable masks. The model applied non-maximum suppression technology to effectively filter out duplicate or low-quality masks, ensuring that each mask has high quality and accuracy.

[0036] The core of the present invention is a three-stage data engine, which is assisted manual annotation, semi-automatic annotation and fully automatic annotation. In combination with the SAM model, in the assisted manual annotation stage, professional annotators use interactive tools to annotate images. These tools are driven by real-time feedback from the model and can be dynamically adjusted according to the accuracy of the model's predictions, thereby optimizing the quality and efficiency of annotation. Then, in the semi-automatic stage, the model automatically predicts high-confidence masks and presents these masks to the annotators, allowing them to focus more on difficult-to-identify image areas, thereby increasing the diversity and complexity of the data set. Finally, in the fully automatic stage, the model fully automatically generates masks, uses a trained blur perception model to process the blurred parts of the image, and optimizes the quality of the mask through advanced algorithms such as non-maximum suppression.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention greatly improves the efficiency and accuracy of deep learning training by introducing the SAM (Segment Anything Model) large model and its multi-stage data engine. By assisting manual annotation, iterative optimization of semi-automatic and fully automatic stages, SAM not only improves the adaptability of the model, but also realizes high-precision segmentation mask generation on large-scale data sets. Secondly, by using fuzzy perception model and non-maximum suppression (NMS), the present invention significantly improves the quality and stability of the mask in the fully automatic stage, ensuring that high-confidence output can be generated under various conditions. In addition, the multi-candidate mask and mask quality prediction functions of the present invention allow the model to effectively manage and resolve segmentation ambiguity when facing complex scenes. In this way, the present invention not only optimizes the data processing process and enhances the generalization ability of the model, but is also particularly suitable for processing large-scale multimodal data, and is suitable for a variety of application scenarios that require efficient and high-precision segmentation, such as autonomous driving, medical image analysis, etc. In summary, the present invention provides an innovative training framework that significantly improves the performance of the model and the breadth of application by combining multimodal data and advanced deep learning technology.

[0039] The multi-stage deep learning training method based on multimodal data proposed in the present invention can effectively solve the problem of insufficient efficiency of multimodal training or the inability of university training to meet the number of data modalities. Experimental results show that compared with traditional processing methods, the present invention can significantly improve data quality and processing efficiency. Therefore, the present invention not only improves the technical level of deep learning model training, but also improves the direction of deep learning for multimodal data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a specific structural diagram of the model training of the multi-stage deep learning training method based on multimodal data of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Multimodal Data: refers to the simultaneous fusion of data from different sources or forms. By integrating these heterogeneous data resources, richer and more comprehensive information can be obtained than a single modality. The sources of multimodal data can be different types such as text, images, videos, and sounds. They are often used in deep learning and artificial intelligence fields, such as natural language processing, computer vision, and machine hearing. For example, in autonomous driving technology, vehicles need to process multimodal data from cameras (visual data), radars (distance data), and voice inputs (sound data) to achieve accurate perception and decision-making of the surrounding environment. In this way, multimodal data processing can enhance the system's understanding ability and improve the accuracy and efficiency of decision-making.

[0044] SAM (Segment Anything Model) large model: a basic model for image segmentation in the field of computer vision. The core goal of SAM is to build a model through a task called "cue segmentability", which enables the model to perform zero-shot generalization on various downstream segmentation problems without seeing specific data. The model's architecture design includes a powerful image encoder, a flexible cue encoder, and a fast mask decoder, which work together to respond to external input segmentation cues in real time and output the corresponding segmentation mask. By combining with existing components such as object detectors, SAM can cope with a variety of visual understanding tasks without additional training.

[0045] Data Engine: A technical architecture for data management and analysis, which aims to efficiently process and analyze large amounts of data. The data engine integrates various data processing technologies such as data acquisition, data storage, data processing, and data analysis to form an efficient and dynamic data processing process. In practical applications, the data engine can support high-performance data query, large-scale data storage, and complex data analysis tasks. For example, in the field of financial analysis, the data engine can quickly process and analyze massive amounts of transaction data, helping analysts to discover market trends and investment opportunities in a timely manner.

[0046] The core of the present invention is to design a three-stage data engine. This design uses interactive segmentation tasks at different stages to maintain rationality even when the input prompt is vague or has multiple interpretations, so that the model performs well in diverse image segmentation tasks. The specific implementation process of the embodiment is as follows:

[0047] A multi-stage deep learning training method based on multimodal data, comprising the following steps:

[0048] Step 1: Data acquisition and preprocessing: In the preprocessing stage, multimodal data is collected from sources including the Internet, databases, and laboratories, and the data volume is large. First, a series of preprocessing is performed on the data, including cleaning, standardization, and formatting. Specific operations include removing background noise in images and videos, unifying data formats to a unified standard, adjusting data resolution, and performing grammatical correction and structural processing on text data. Through these steps, the consistency and quality of the data are ensured, and the input data for model training is optimized, thereby effectively enhancing the accuracy and robustness of the model, improving data processing efficiency and the breadth of data application;

[0049] Step 2: Multimodal data fusion and feature extraction: The pre-processed multimodal data is integrated using a specific fusion algorithm. The image data is transformed in the frequency domain using the Fast Fourier Transform (FFT) technique, and feature extraction is performed in combination with text information. This process not only improves the expressiveness of the data, but also provides a rich set of input features for the SAM model, thereby enhancing the model's input efficiency and processing accuracy. The key to this step is to improve the availability of data and the comprehensive performance of the model, ensuring the efficiency and accuracy of the data processing process.

[0050] Step 3: Valid segmentation mask generation and multiple candidate mask processing: The SAM model is used to process the input multimodal data and automatically generate a series of valid segmentation masks. The built-in mask quality prediction module evaluates the generated mask to ensure that its overlap with the real object meets the preset standard. In order to deal with possible ambiguity in the input, the model outputs multiple candidate masks at the same time. The non-maximum suppression (NMS) technology is used to screen these candidate masks, filter out duplicate and low-confidence masks, and retain only high-quality segmentation results. This step significantly improves the practicality and efficiency of the model and ensures the accuracy and reliability of the segmentation results.

[0051] Step 4: Data engine assisted manual annotation stage: In this stage, professional annotators use a browser-based interactive segmentation tool to annotate the target objects by explicitly clicking on the foreground and background points in the image. To improve the quality of the annotation, the annotator can use fine pixel-level "brush" and "eraser" tools to make adjustments. The SAM model provides real-time feedback and suggestions during the annotation process. In the initial stage, the SAM model is trained based on a public segmentation dataset to ensure that the model has basic image recognition and segmentation capabilities. As data accumulates in actual applications, the SAM model will be retrained using only newly annotated mask data to adapt to specific annotation tasks and optimize performance. This continuous learning and adaptation process significantly improves the efficiency of annotation: the average annotation time per mask is reduced from 34 seconds to 14 seconds, and the average number of masks per image increases from 20 to 44.

[0052] Step 5: Data Engine Semi-Automatic Phase: The main goal of this phase is to enhance the diversity of masks, thereby significantly improving the model's ability to segment a variety of objects. The workflow has been carefully designed to ensure the efficiency and accuracy of data generation. The system first automatically identifies high-confidence masks, which are automatically generated by advanced bounding box detectors. The detector is trained on all first-stage masks, using a general "object" category to ensure that a wide range of object types can be covered. This method allows the model to identify correct masks with high probability in the preliminary stage. These pre-filled mask images are then presented to annotators, whose task is to check and correct these automatically generated masks and manually annotate those objects that the system has failed to detect. This process not only improves the quality of the masks, but also ensures the comprehensiveness and diversity of the data. The 5.9 million masks collected in this phase effectively expand the dataset and provide rich and diverse inputs for deep learning models. Although the average annotation time per mask has risen back to 34 seconds, this time investment ensures the high quality and comprehensiveness of the data.

[0053] Step 6: Data Engine Fully Automatic Stage: Model Enhancement Stage: In the previous stages, the model has been significantly improved by accumulating a large number of diverse masks. These diverse masks constitute a large dataset, providing the model with rich learning materials, enabling the model to adapt to various complex image conditions and object types. First, a new blur perception model was developed, which is specially designed to handle blurry areas in images and can effectively predict masks even when visual information is incomplete. To further improve the accuracy and efficiency of mask generation, the model introduces a regular 32x32 grid as a hint. Each grid point is used to predict a set of masks that may correspond to a valid object. In addition, an IoU (intersection over union) prediction module is added to filter out high-confidence masks and identify stable masks. In order to deal with potential overlaps between masks and improve the quality of small-sized masks, the model applies non-maximum suppression (NMS) technology. This technology can effectively filter out duplicate or low-quality masks, ensuring that each mask is of high quality and high accuracy. This stage successfully processed a large-scale dataset containing 11 million images and generated a total of about 1.1 billion high-quality masks. This phase not only demonstrated the powerful capabilities of deep learning models in practical applications, but also proved the efficiency and reliability of multi-stage training methods in processing large-scale data;

[0054] Step 7: Quality Control and Model Optimization: Finally, a series of strict quality control measures are introduced, including Intersection over Union (IoU) prediction and stability assessment, to ensure that the generated mask dataset strictly meets the training specifications. Through continuous iterative optimization procedures, the present invention significantly improves the performance of the model, and ultimately enables efficient deep learning training of multimodal data. This process not only improves the accuracy of model training, but also optimizes the resource utilization efficiency during training, thereby achieving superior learning results in multimodal data processing.

[0055] The core technology of the present invention is to introduce a three-stage data engine to support the training of the model, which includes three stages: assisted manual annotation, semi-automatic annotation, and fully automatic annotation. This structure not only improves the diversity and quality of data, but also greatly improves the annotation efficiency. This technology effectively enables the model to perform well when processing diverse and ambiguous inputs, while also ensuring flexibility and efficiency in practical applications. Compared with the prior art, this method uses data engine technology to help better process multimodal image data, thereby improving model performance.

[0056] The invention is significantly enhanced by the accumulation of a large number of diverse masks during the training process, and then a fuzzy perception model is developed to handle incomplete vision, and the mask generation accuracy and efficiency are optimized through a 32x32 grid. In addition, the introduction of the IoU prediction module and non-maximum suppression technology effectively improves the mask quality and accuracy. Finally, this stage independently processed a large-scale dataset containing 11 million images and generated about 1.1 billion high-quality masks, proving its efficiency and reliability in practical applications, and significantly improving the applicability and accuracy of the model in various scenarios.

[0057] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A multi-stage deep learning training method based on multimodal data, characterized in that: The following steps are involved: Step 1: Data acquisition and preprocessing: Collect multimodal data from the Internet, databases, and laboratories, and optimize them through cleaning, standardization, and formatting processes; Step 2: Multimodal data fusion and feature extraction: The pre-processed multimodal data is integrated using a specific fusion algorithm, the image data is transformed in the frequency domain using the fast Fourier transform technology, and feature extraction is performed in combination with text information; Step 3: Generate effective segmentation masks and process multiple candidate masks: Use the SAM model to process the input multimodal data and automatically generate effective segmentation masks; Step 4: Data engine-assisted manual annotation stage: Use the browser-based interactive segmentation tool to annotate the target objects by explicitly clicking on the foreground and background points in the image. The SAM model provides real-time feedback and suggestions during the annotation process. Step 5: Data engine semi-automatic stage: The system automatically identifies high-confidence masks, which are automatically generated by the bounding box detector. The mask image will be shown to the annotator, who will check and correct the automatically generated masks and manually annotate the remaining objects that were not detected by the system. The collected masks effectively expand the dataset. Step 6: Data engine fully automatic stage: The model is significantly improved through the accumulation of a large number of diverse masks in the previous step. The model uses a regular 32x32 grid as a prompt. Each grid point is used to predict the mask corresponding to a valid object. The model contains an IoU prediction module for screening out high-confidence masks. Step 7: Quality control and model optimization: Introduce strict quality control methods to ensure that the generated mask dataset meets the training requirements, and achieve efficient deep learning training for multimodal data through iterative optimization.

2. The multi-stage deep learning training method based on multimodal data according to claim 1, characterized in that: The optimization through cleaning, standardization and formatting process in step 1 specifically includes: removing background noise in images and videos, unifying data formats to a unified standard, adjusting data resolution, and performing grammatical correction and structural processing on text data.

3. The multi-stage deep learning training method based on multimodal data according to claim 1, characterized in that: The step three specifically includes: The SAM model is used to process the input multimodal data and automatically generate effective segmentation masks. The built-in mask quality prediction module evaluates the generated masks to ensure that their overlap with the real objects meets the preset standards. The model outputs multiple candidate masks at the same time and uses non-maximum suppression technology to screen the candidate masks, filtering out duplicate and low-confidence masks and retaining only high-quality segmentation results.

4. The multi-stage deep learning training method based on multimodal data according to claim 1, characterized in that: The step 4 specifically includes: Use the browser-based interactive segmentation tool to annotate target objects by explicitly clicking on foreground and background points in the image. Adjustments can be made using fine pixel-level "brush" and "eraser" tools, and the SAM model provides real-time feedback and suggestions during the annotation process.

5. The multi-stage deep learning training method based on multimodal data according to claim 1, characterized in that: The step five specifically includes: First, high-confidence masks are automatically identified and automatically generated by an advanced bounding box detector. The detector is trained on the first-stage masks, using a generic "object" category to ensure coverage of a wide range of object types. The pre-filled mask images are shown to annotators to review and correct these automatically generated masks, and manually annotate objects that the system failed to detect.

6. The multi-stage deep learning training method based on multimodal data according to claim 1, characterized in that: The step six specifically includes: A blur perception model is developed to handle blurry areas in images, and a regular 32x32 grid is introduced as a prompt. Each grid point is used to predict a set of masks corresponding to valid objects. An IoU prediction module is then added to screen out high-confidence masks. It can identify stable masks. The model uses non-maximum suppression technology to filter out duplicate or low-quality masks to ensure that each mask is of high quality and accuracy.

Citation Information

Patent Citations

  • Multi-modal medical image fusion method based on deep learning

    CN116630762A

  • Training and labeling method based on industrial AI vision general segmentation large model

    CN117671342A

  • Text-visual anaphora segmentation method based on three-stage training strategy

    CN118038043A

  • Medical diagnosis method and system based on multi-modal AIGC model

    CN118098570A

  • Panoptic segmentation with multi-dataset training and part-whole awareness

    WO2024233927A1