Pre-training data preparation method based on RAM, SAM, internVL and OMchat

Abstract

The invention discloses a pre-training data preparation method based on an RAM, an SAM, an internVL and an OMchat, and the method comprises the following steps: 1, fixing data sources, setting a specific proportion according to each source, and finally combining one million of data; step 2, pre-labeling the original picture by using the everything recognition RAM; step 3, on the basis of the picture pre-labeled in the step 2, using internVL for pre-labeling; step 4, carrying out manual checking and calibration on the picture pre-marked by the internVL in the step 3; step 5, obtaining target edge mask information from the picture which is manually checked and calibrated in the step 4 through everything segmentation SAM; step 6, using the OMchat multi-modal large model to train the features of the target and mask information to perform multiple rounds of training processing; and step 7, finally taking the processed data as pre-training data of the OMdet multi-modal large model. According to the pre-training data preparation method based on the RAM, the SAM, the internVL and the OMchat, the pre-training data is efficiently generated, so that the algorithm generalization performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence deep learning, and more specifically to a method for preparing pre-training data based on RAM, SAM, internVL and OMchat. Background Art

[0002] In the era of deep learning, the number of parameters of neural networks is increasing, and the cost of training from scratch is also increasing. Fortunately, in the main fields of artificial intelligence applications such as computer vision and natural language processing, people can use the pre-training-fine-tuning paradigm of transfer learning to effectively reduce training costs. Based on the above background, how to make excellent pre-training has become a difficulty. The difficulty of making pre-training data lies mainly in how to effectively collect large-scale and diverse data and ensure the quality and accuracy of data annotation. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for preparing pre-training data based on RAM, SAM, internVL and OMchat. By producing pre-training data through this method, the labeling efficiency can be effectively improved, and data sets can be produced on a large scale, thereby reducing the cost and difficulty of producing pre-training data.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for preparing pre-training data based on RAM, SAM, internVL and OMchat, characterized in that it comprises the following steps:

[0005] Step 1: Fix the data sources, set a specific ratio for each source, and finally combine them into 1 million data, which will enter the iterative process in multiple rounds;

[0006] Step 2: Use the Everything Recognition RAM to pre-label the original image and obtain the target bbox information and label information;

[0007] Step 3: Use internVL to pre-annotate the image pre-annotated in step 2;

[0008] Step 4: manually check and calibrate the image pre-annotated by internVL in step 3;

[0009] Step 5: The image manually checked and calibrated in step 4 is subjected to the object segmentation SAM to obtain the target edge mask information;

[0010] Step 6: Use the features and mask information of the OMchat multimodal large model training target to perform multiple rounds of training to obtain OMchat model pre-labeled data;

[0011] Step 7: Finally, the processed data is used as pre-training data for the OMdet multimodal large model.

[0012] As a further improvement of the present invention, the specific steps of using internVL for pre-labeling in step three are: use internVL to pre-label the targets after the all-things recognition RAM, then debug different prompts to obtain different features of the targets such as: attributes, relationships, locations, real and non-existent information, and solidify the prompts, and after testing that the OMchat effect is better than the internVL effect, replace the model for pre-labeling.

[0013] As a further improvement of the present invention, before internVL pre-labels the target after the all-things recognition RAM, a bbox frame is drawn on the original image after using the all-things recognition RAM in step 2. One image displays one target, and the thickness of the target frame border needs to be adjusted according to the pixel size of the image.

[0014] As a further improvement of the present invention, the specific method of manual verification and calibration in step 4 is to adjust the label, bbox and target features obtained after pre-labeling of the target, and ensure that the target information contained in different features of the target is not less than 2 contents.

[0015] As a further improvement of the present invention, the specific method of obtaining the target edge mask information through the object segmentation SAM in step 5 is: inputting the bbox information of the target in SAM to obtain the mask information of each target.

[0016] As a further improvement of the present invention, the specific method of the multiple rounds of training in step six is: use the OMchat multimodal large model to train the target features and mask information, perform multiple rounds of training with different data amounts, and use the same prompt of internVL for testing, focusing on the label accuracy, the total amount of target features and the correct proportion of target features.

[0017] As a further improvement of the present invention, the data in step seven is the label accuracy of the OMchat model. After the correct proportion of target features reaches 70%, it can be used as pre-training data for a fixed data source. The specific method of processing the data is: splitting the content of the fixed prompt answer of OMchat, processing each target feature information into a single label and corresponding to the bbox information of the original target.

[0018] Beneficial effects of the present invention: Compared with the traditional deep learning target detection pre-training data production:

[0019] Using the large model of all things recognition, the data covers a wide range. Using mask information to train a multimodal large model has a higher accuracy rate than traditional bbox information. Using other multimodal large models in the initial stage can save more manpower labeling costs. While making pre-training data, a multimodal large model with excellent results is also obtained. Iterative optimization can be performed on the data source. When a data source reaches the corresponding accuracy, pre-training data can be continuously and automatically output for the source without manual labeling. The pre-training data produced using this method has a stronger semantic generalization ability for the target features. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic flow chart of the method for preparing pre-training data based on RAM, SAM, internVL and OMchat of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0022] Reference Figure 1 As shown, a method for preparing pre-training data based on RAM, SAM, internVL and OMchat in this embodiment specifically includes the following steps:

[0023] 1. Data preparation: fix the data sources, set a specific ratio for each source, and finally combine them into 1 million data, and enter the iterative process in multiple rounds

[0024] 2. Everything Recognition RAM obtains the target detection frame: Use Everything Recognition RAM to pre-label the original image and obtain the target bbox frame information and label information

[0025] 3. InternVL pre-labeling: After all objects are recognized, a bbox is drawn on the original image. One object is drawn in one image. The thickness of the object box needs to be adjusted according to the pixel size of the image. Use internVL to pre-label the objects after all objects are recognized in RAM. Different prompts need to be debugged to obtain different features of the object, such as:

[0026] Attributes, relationships, locations, real and non-existent information, etc., and solidify the prompt. After testing that OMchat is better than internVL, the model needs to be replaced for pre-labeling.

[0027] 4. Manual verification and calibration: After internVL pre-labeling, the target label, bbox and target features obtained after pre-labeling need to be manually adjusted in the early stage, and ensure that the target information contained in different features of the target is not less than 2 contents.

[0028] 5. Segmentation of all objects SAM obtains target edge mask information: After manual verification and calibration, you need to enter the target bbox information in SAM to obtain the mask information of each target. Note: Mask information: a collection of target edge points

[0029] 6.OMchat model training: Use the OMchat multimodal large model to train the target features and mask information, perform multiple rounds of training with different data amounts, and use the same prompt of internVL for testing, focusing on label accuracy, total number of target features, and correct proportion of target features.

[0030] 7. After data processing, it can be used as OMdet pre-training data: After the label accuracy and target feature correctness ratio of the OMchat model reach 70%, it can be used as pre-training data for fixed data sources. The main process of data processing is: split the content of the fixed prompt answer of OMchat, process each target feature information into a single label and correspond to the bbox information of the original target.

[0031] To sum up, the pre-training data preparation method based on RAM, SAM, internVL and OMchat of this embodiment can effectively improve the labeling efficiency by producing pre-training data, and can produce data sets on a large scale, thereby reducing the cost and difficulty of producing pre-training data.

[0032] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for preparing pre-training data based on RAM, SAM, internVL and OMchat, characterized in that: The steps include: Step 1: Fix the data sources, set a specific ratio for each source, and finally combine them into 1 million data, which will enter the iterative process in multiple rounds; Step 2: Use the Everything Recognition RAM to pre-label the original image and obtain the target bbox information and label information; Step 3: Use internVL to pre-annotate the image pre-annotated in step 2; Step 4: manually check and calibrate the image pre-annotated by internVL in step 3; Step 5: The image manually checked and calibrated in step 4 is subjected to the object segmentation SAM to obtain the target edge mask information; Step 6: Use the features and mask information of the OMchat multimodal large model training target to perform multiple rounds of training to obtain OMchat model pre-labeled data; Step 7: Finally, the processed OMchat model pre-labeled data is used as the pre-training data for the OMdet multimodal large model.

2. The method for preparing pre-training data based on RAM, SAM, internVL and OMchat according to claim 1, characterized in that: The specific steps of using internVL for pre-labeling in step three are: use internVL to pre-label the targets after the all-things recognition RAM, then debug different prompts to obtain different features of the targets such as: attributes, relationships, locations, real and non-existent information, and solidify the prompts. After testing that the OMchat effect is better than the internVL effect, replace the model for pre-labeling.

3. The method for preparing pre-training data based on RAM, SAM, internVL and OMchat according to claim 2, characterized in that: Before internVL pre-labels the targets after the Everything Recognition RAM, a bbox is drawn on the original image after using the Everything Recognition RAM in step 2. One target is drawn per image, and the thickness of the target box border needs to be adjusted according to the pixel size of the image.

4. The method for preparing pre-training data based on RAM, SAM, internVL and OMchat according to claim 1, 2 or 3, characterized in that: The specific method of manual verification and calibration in step 4 is to adjust the label, bbox and target features obtained after pre-labeling of the target, and ensure that the target information contained in different features of the target is not less than 2 contents.

5. The method for preparing pre-training data based on RAM, SAM, internVL and OMchat according to claim 3, characterized in that: The specific method of obtaining the target edge mask information through the all-things segmentation SAM in step 5 is: input the bbox information of the target in SAM to obtain the mask information of each target.

6. The method for preparing pre-training data based on RAM, SAM, internVL and OMchat according to claim 1, 2 or 3, characterized in that: The specific method of the multiple rounds of training in step six is: use the OMchat multimodal large model to train the target features and mask information, perform multiple rounds of training with different data amounts, and use the same prompt of internVL for testing, focusing on the label accuracy, the total number of target features and the correct proportion of target features.

7. The method for preparing pre-training data based on RAM, SAM, internVL and OMchat according to claim 1, 2 or 3, characterized in that: The data in step seven is the label accuracy of the OMchat model. After the correct ratio of target features reaches 70%, it can be used as pre-training data for a fixed data source. The specific method of processing the data is: split the content of the fixed prompt answer of OMchat, process each target feature information into a single label and correspond to the bbox information of the original target.

Images