Automatic line labeling method based on AI multi-modal large model

Abstract

The invention relates to the technical field of AI multi-modal large models, and discloses an automatic line labeling method based on an AI multi-modal large model, and the method comprises the steps: data set and data processing: firstly, searching two-dimensional and three-dimensional drawing data sets in a targeted manner; converting the three-dimensional structure diagram of the drawing into a two-dimensional three-view diagram; carrying out linear, radius, diameter and angle automatic labeling and manual checking on a two-dimensional three-view graph and a two-dimensional drawing which are based on the three-dimensional structure graph characteristics; performing structured text processing on the two-dimensional drawing, verifying data legality, and removing and repairing data with poor quality; a two-dimensional drawing scheme is segmented, the whole scheme is converted into a plurality of N blocks with fixed sizes, and then structured texts of all the blocks are compressed. The automatic line labeling method based on the AI multi-modal large model has universality and can be widely applied to the field of engineering design, engineers are helped to rapidly and accurately label drawings, the design efficiency and quality are improved, and the problems that the requirement for developers is high, and the customization cost is high are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of AI multimodal large models, and in particular to an automatic line labeling method based on AI multimodal large models. Background Art

[0002] In the field of CAD / CAM, modeling technology is the key method to convert physical objects and their properties into digital expressions inside computers. It not only contains the geometric topological information of parts, but also contains a lot of non-geometric information such as material properties, process requirements, and annotation lines, so it has rich engineering semantics;

[0003] In the field of modern machine learning, Multimodal Large Language Models (MM-LLMs) refer to large language models that combine multiple information modalities. These models can not only process text data, but also other forms of data, such as images, audio, video, models, etc. The emergence of MM-LLMs has promoted the development of multimodal machine learning, allowing machines to more comprehensively understand and process different forms of data, thereby improving the ability to understand and model the complex real world;

[0004] Multimodal pre-training technology refers to a method of pre-training on multiple data modalities, which aims to improve the model's ability to understand and process multiple data types. By performing multimodal pre-training on large-scale data sets, the model can learn the correlation and semantic information between different data modalities, thereby performing better in various upstream and downstream tasks;

[0005] Full parameter fine-tuning technology refers to fine-tuning all parameters (full parameters) of the model to adapt to specific tasks or data sets during transfer learning. In the case of multimodal pre-training, full parameter fine-tuning technology can help the model better adapt to data in specific fields or tasks and improve the performance of the model on specific tasks;

[0006] Explore the same 3D model in two modes: modeling process and geometric shape: modeling process and geometric shape can be regarded as different manifestations of the same entity, that is, the relationship between the abstract representation of the same object in the CAD modeling process and its geometric form in the real world. By studying the relationship between these two modes, we can better understand and optimize the CAD / CAM modeling process. This research method helps to explore the application of multimodal data in the engineering field and provide new ideas and methods for the development of machine learning technology in the CAD / CAM field. In this method, the use of automatic annotation models can improve the efficiency and accuracy of engineering design;

[0007] However, there are still the following disadvantages: traditional annotation often uses customized development scripts based on the developer's experience to annotate features, which places high demands on developers and has high customization costs.

[0008] Therefore, an automatic line labeling method based on AI multimodal large model is proposed. Summary of the invention

[0009] Technical issues solved

[0010] In view of the shortcomings of the prior art, the present invention provides an automatic line labeling method based on an AI multimodal large model, which has the advantages of being versatile and can be widely used in engineering design and manufacturing fields, helping engineers to quickly and accurately label drawings, and improving design efficiency and quality. It solves the problem that traditional labeling often labels features based on customized development scripts based on the developer's experience, which has high requirements for developers and high customization costs.

[0011] Technical Solution

[0012] In order to achieve the above-mentioned universality and be widely applied in the field of engineering design and manufacturing, to help engineers quickly and accurately mark drawings and improve design efficiency and quality, the present invention provides the following technical solutions:

[0013] A method for automatic line annotation based on an AI multimodal large model comprises the following steps:

[0014] Step 1: Dataset and data processing: First, collect 2D and 3D drawing data sets in a targeted manner;

[0015] Step 2: Then convert the three-dimensional structure diagram of the drawing into two-dimensional three-view drawing;

[0016] Step 3: Automatically mark and manually review the linear, radius, diameter, and angle of the 2D three-view drawings and 2D drawings based on the features of the 3D structural drawing;

[0017] Step 4: Perform structured text processing on the 2D drawings and verify the legitimacy of the data, and remove and repair poor quality data;

[0018] Step 5: Segment the 2D drawing plan, convert the entire plan into N blocks of fixed size, and then compress the structured text of all blocks;

[0019] Step 6: Process the data set and divide it into an unsupervised data set for pre-training and a supervised data set for fine-tuning. Then split the supervised data set into a training set and a test set.

[0020] Step 7: Model training: Based on the MM-LLMs model, find the hot words in the data set and expand the original vocabulary;

[0021] Step 8: Update and expand the original model parameters. Given a pre-trained large language model parameter set Θ, a language model containing text x text and label x dimension The multimodal data set D = {(x text ,x dimension )};

[0022] To continue pre-training the model and update the parameters, the following formula can be used:

[0023]

[0024] Step 9: Fine-tune all parameters of the pre-trained model, given a supervised dataset D {dimension_task} ={(x i ,y i )};

[0025] The goal of fine-tuning is to minimize the loss function on the task dataset, that is, to minimize the loss function L(Θ; D dimension_task ), in order to fine-tune all parameters and update the parameters, the following formula is used to adapt to the automatic labeling task:

[0026]

[0027] Step 10: Test set and manual evaluation of the model results.

[0028] Preferably, the specific objects of structured text processing in step 4 are lines, types, blocks, fillets, arcs, text information and two-dimensional coordinates.

[0029] Preferably, in step five, irrelevant information is removed and annotated while compressing the structured text.

[0030] Preferably, in step ten, the erroneous data is enhanced and then retrained using a reinforcement learning decision policy optimization algorithm, Decision Policy Optimization (DPO).

[0031] Preferably, in the step eight, θ (t) is the model parameter of the tth iteration, Θ (t+1) is the updated parameter, η is the learning rate, represents the gradient with respect to the parameter Θ.

[0032] Preferably, by calculating the gradient of the loss function with respect to the parameters and updating the parameters according to the gradient, the model can continue to learn on multimodal data and improve performance.

[0033] Preferably, in step nine, x iis to input unmarked drawings, y i Output annotation information.

[0034] Compared with the prior art, the present invention provides an automatic line annotation method based on an AI multimodal large model, which has the following beneficial effects:

[0035] This automatic line labeling method based on AI multimodal large models is versatile and can be widely used in engineering design and manufacturing fields, helping engineers to quickly and accurately label drawings, improve design efficiency and quality, and solve the problems of high requirements for developers and high customization costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a data processing flow chart of the automatic annotation method of the present invention;

[0037] Figure 2 This is a flow chart of model training for the automatic annotation method of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1-Figure 2 ;

[0040] A method for automatic line annotation based on an AI multimodal large model comprises the following steps:

[0041] Step 1: Dataset and data processing: First, collect 2D and 3D drawing data sets in a targeted manner;

[0042] Step 2: Then convert the three-dimensional structure diagram of the drawing into two-dimensional three-view drawing;

[0043] Step 3: Automatically mark and manually review the linear, radius, diameter, and angle of the 2D three-view drawings and 2D drawings based on the features of the 3D structural drawing;

[0044] Step 4: Perform structured text processing on the 2D drawings and verify the legitimacy of the data, remove and repair poor quality data. The specific objects of structured text processing are lines, types, blocks, fillets, arcs, text information and 2D coordinates;

[0045] Step 5: Segment the 2D drawing plan and convert the entire plan into N blocks of fixed size. Then compress the structured text of all blocks and remove and annotate irrelevant information while compressing the structured text.

[0046] Step 6: Process the data set and divide it into an unsupervised data set for pre-training and a supervised data set for fine-tuning. Then split the supervised data set into a training set and a test set.

[0047] Step 7: Model training: Based on the MM-LLMs model, find the hot words in the data set and expand the original vocabulary;

[0048] Step 8: Update and expand the original model parameters. Given a pre-trained large language model parameter set Θ, a language model containing text x text and label x dimension The multimodal data set D = {(x text ,x dimension )};

[0049] To continue pre-training the model and update the parameters, the following formula can be used:

[0050]

[0051] Among them, Θ (t) is the model parameter of the tth iteration, Θ (t+1) is the updated parameter, η is the learning rate, Represents the gradient of the parameter Θ. By calculating the gradient of the loss function with respect to the parameter and updating the parameter according to the gradient, the model can continue to learn on multimodal data and improve performance.

[0052] Step 9: Fine-tune all parameters of the pre-trained model, given a supervised dataset D {dimension_task} ={(x i ,y i )};

[0053] The goal of fine-tuning is to minimize the loss function on the task dataset, that is, to minimize the loss function L(Θ; D dimension_task ), in order to fine-tune all parameters and update the parameters, the following formula is used to adapt to the automatic labeling task:

[0054] x i is to input unmarked drawings, y i Output annotation information;

[0055] Step 10: Test the model results and conduct manual evaluation. After data enhancement, retrain the erroneous data using the reinforcement learning decision policy optimization algorithm Decision Policy Optimization (DPO).

[0056] The beneficial effects of the present invention are as follows: the automatic line marking method based on the AI ​​multimodal large model is versatile and can be widely used in the fields of engineering design and manufacturing, helping engineers to mark drawings quickly and accurately, improve design efficiency and quality, and solve the problems of high requirements for developers and high customization costs.

[0057] When in use, the historical plans are parsed, features are extracted, semantic features and data structures are reconstructed, and data sets are exported. The reconstructed semantic feature topological vocabulary is used to pre-train the unsupervised annotation large model, and then the supervised annotation data set is used to fine-tune the annotation large model, so as to realize automatic annotation of linear, radius, diameter, angle, etc. of the drawings using the multimodal large model.

[0058] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic line annotation based on AI multimodal large model, characterized in that: The following steps are involved: Step 1: Dataset and data processing: First, collect 2D and 3D drawing data sets in a targeted manner; Step 2: Then convert the three-dimensional structure diagram of the drawing into two-dimensional three-view drawing; Step 3: Automatically mark and manually review the linear, radius, diameter, and angle of the 2D three-view drawings and 2D drawings based on the features of the 3D structural drawing; Step 4: Perform structured text processing on the 2D drawings and verify the legitimacy of the data, and remove and repair poor quality data; Step 5: Segment the 2D drawing plan, convert the entire plan into N blocks of fixed size, and then compress the structured text of all blocks; Step 6: Process the data set and divide it into an unsupervised data set for pre-training and a supervised data set for fine-tuning. Then split the supervised data set into a training set and a test set. Step 7: Model training: Based on the MM-LLMs model, find the hot words in the data set and expand the original vocabulary; Step 8: Update and expand the original model parameters. Given a pre-trained large language model parameter set Θ, a language model containing text x text and label x dimension The multimodal data set D = {(x text , x dimension )}; To continue pre-training the model and update the parameters, the following formula can be used: Step 9: Fine-tune all parameters of the pre-trained model, given a supervised dataset D {dimension_task} ={(x i ,y i )}; The goal of fine-tuning is to minimize the loss function on the task dataset, that is, to minimize the loss function L(Θ; D dimension_task ), in order to fine-tune all parameters and update the parameters, the following formula is used to adapt to the automatic labeling task: Step 10: Test the model results and conduct manual evaluation.

2. The method for automatic line annotation based on AI multimodal large model according to claim 1 is characterized in that: The specific objects of structured text processing in step 4 are lines, types, blocks, fillets, arcs, text information and two-dimensional coordinates.

3. The method for automatic line annotation based on AI multimodal large model according to claim 1 is characterized in that: In the step 5, irrelevant information is removed and annotated while compressing the structured text.

4. The method for automatic line annotation based on AI multimodal large model according to claim 1 is characterized in that: In the step 10, the erroneous data is enhanced and then retrained using the reinforcement learning decision policy optimization algorithm Decision Policy Optimization (DPO).

5. The method for automatic line annotation based on AI multimodal large model according to claim 1 is characterized in that: In the step eight, θ (t) is the model parameter of the tth iteration, Θ (t+1) is the updated parameter, η is the learning rate, represents the gradient with respect to the parameter Θ.

6. The method for automatic line annotation based on AI multimodal large model according to claim 5 is characterized in that: By calculating the gradient of the loss function with respect to the parameters and updating the parameters according to the gradient, the model can continue to learn on multimodal data and improve performance.

7. The method for automatic line annotation based on AI multimodal large model according to claim 1 is characterized in that: Step nine x i is to input unmarked drawings, y i Output annotation information.

Images