Direct generation of 0 # ledger according to design drawing in end-to-end manner through multi-modal deep learning

Abstract

The invention provides a data processing method and device, and the method comprises the steps: collecting engineering industry professional data, including a text class and a picture class; carrying out graying processing on the picture; the multi-modal model is trained by using an RLHF technology based on industry professional data, so that the model has engineering industry knowledge; designing drawings and 0 # machine accounts generated according to the designing drawings are used as mark data, and a multi-modal model is trained and optimized according to the mark data; and according to the trained multi-modal model, the 0 # ledger is directly generated end to end through a design drawing, and a user does not need to have professional industry knowledge and a large number of industries.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a data processing method, device, storage medium, and electronic device. Background Art

[0002] At present, the generation of "0# ledger" in the engineering industry is mainly done manually. Specifically, technicians extract information from design drawings (such as roadbed, pavement, bridges, tunnels), and manually organize the data into structured data "0# ledger" after manual verification.

[0003] Manually generating the "0# ledger" places high demands on technical personnel, who must possess professional industry knowledge and have a wealth of industry experience. At the same time, since the generation of the "0# ledger" involves a large number of complex design drawings, the workload is huge and prone to errors, and the final "0# ledger" needs to be manually reviewed.

[0004] Therefore, there is an urgent need for a data processing method, device and storage medium that can simultaneously solve the problems that require professional industry knowledge, a large amount of industry experience, a huge workload and are prone to errors. Summary of the invention

[0005] The embodiments of the present invention provide a data processing method, device and storage medium to solve the problem that generating 0# ledger requires professional industry knowledge, a lot of industry experience, huge workload and prone to errors.

[0006] A first aspect of an embodiment of the present invention provides a data processing method, including:

[0007] Professional data collection in the engineering industry, including text data and image data;

[0008] Convert color images to grayscale;

[0009] Manually annotate the collected text data and grayscaled image data;

[0010] Use RLHF technology to train the model on the labeled data to obtain a professional model;

[0011] Split the design drawings according to "sub-projects", and split the 0# ledger according to "sub-projects";

[0012] Using the drawings of the sub-projects and their partial 0# ledger data as annotation data, the RLHF technology is used to train and optimize the model, and a model that can generate partial "0# ledgers" through partial construction drawings is obtained;

[0013] Using design drawings as input and sub-project data as output labeled data, the RLHF technology is used to train and optimize the model to obtain a model that can split the design drawings according to sub-projects;

[0014] Input the construction drawing data into the "split model", and the model will automatically input the split data into the "0# production model" to generate the local 0# ledger data for each sub-project;

[0015] Integrate all local 0# ledger data to obtain the final 0# ledger. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A detailed flow chart of a first implementation of a data processing method;

[0017] Figure 2 A flowchart of a first implementation of a data processing method;

[0018] Figure 3 A diagram of a model training architecture for a first implementation of a data processing method;

[0019] Figure 4 A flow chart for generating a local 0# account model training for the first implementation of the data processing method;

[0020] Figure 5 A file splitting model training flow chart for a first implementation of the data processing method;

[0021] Figure 6 The present invention is a flowchart of the use of the first embodiment of the data processing device. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0024] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0025] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0026] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0027] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0028] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0029] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0030] The present invention provides a data processing method, such as Figure 1 The flow chart shown includes:

[0031] ①. Data collection, collecting pictures and text-based knowledge information of the engineering industry;

[0032] ②. Grayscale the collected images to simplify the model input, reduce computing costs, and improve generalization capabilities;

[0033] ③. Annotation: manually annotating the collected text and image information;

[0034] ④. Using the labeled data, a dedicated model is trained through RLHF (the RL algorithm here uses PPO) technology with CLIP and LLaMA-2-70B as the base model.

[0035] The multimodal model trained in the present invention adopts the DeepSpeed-VisualChat framework, such as Figure 2 As shown, including:

[0036] Use a combination of images and text to train the model so that it can understand different types of input data;

[0037] The present invention uses a combination of multiple models to generate the "0# ledger" to solve the "forgetting problem" of large models, including:

[0038] CZeroModel (classification model) is used to process text and image classification. It does not need to remember a lot of contextual knowledge. It only divides the current design drawing into different sub-projects according to its content.

[0039] HZeroModel (local 0# ledger data generation model) is used to generate corresponding sub-project 0# ledger data based on the sub-project data. HZeroModel only needs to remember the context content of the sub-project, which greatly reduces the input content of the model.

[0040] Use a trained model such as Figure 5 As shown, including:

[0041] ①. Upload design drawing information, usually in PDF or Word document;

[0042] ②. Use code to split the entire design drawing information into single pages;

[0043] ③. Use CZeroModel to classify single-page design drawings according to different sub-projects;

[0044] ④. The classified design drawing information is grouped according to the sub-projects and then transferred to HZeroModel, and the "0# account" of each sub-project is obtained through HZeroModel;

[0045] ⑤. Combine the “0# ledger” of each sub-project to obtain the final “0# ledger”.

Claims

1. A data processing method, characterized in that: include: Professional data collection in the engineering industry, including text data and image data; Convert color images to grayscale; Manually annotate the collected text data and grayscaled image data; Use RLHF technology to train the model on the labeled data to obtain a professional model; Split the design drawings according to "sub-projects", and split the 0# ledger according to "sub-projects"; Using the drawings of the sub-projects and their partial 0# ledger data as annotation data, the RLHF technology is used to train and optimize the model, and a model that can generate partial "0# ledger" from partial construction drawings is obtained; Using design drawings as input and sub-project data as output labeled data, the RLHF technology is used to train and optimize the model to obtain a model that can split the design drawings according to sub-projects; Input the construction drawing data into the "split model", and the model will automatically input the split data into the "0# production model" to generate the local 0# ledger data for each sub-project; Integrate all local 0# ledger data to obtain the final 0# ledger.

2. The data processing method according to claim 1, characterized in that: The professional data collection of the engineering industry includes: The national standards and requirements data of the engineering industry, industry standard requirements data, owner standard requirements data, construction enterprise and project department standard requirements data, and various construction drawings are used as data sets; the data sets include text and image data; the data sets are used for subsequent training of professional models.

3. The data processing method according to claim 1, characterized in that: The manual labeling of the collected text data and the grayscaled image data includes: Combine text and image data in the form of "questions" and "answers" as manually annotated data.

4. The data processing method according to claim 1, characterized in that: The method of using the RLHF technology to train the model on the labeled data to obtain a professional model includes: Using the labeled data, a dedicated model is trained through the RLHF (the RL algorithm here uses PPO) technology with CLIP and LLaMA-2-70B as the base model; for the convenience of subsequent description, the model trained here is recorded as ZeroModel.

5. The RLHF technique according to claim 4, wherein the RLHF comprises: A method that combines reinforcement learning and human feedback; in traditional RL, the agent usually learns directly from the interaction with the environment, while in RLHF, the agent's learning process uses feedback information provided by humans. This feedback can be reward signals, preferences, guidance, or other forms of evaluation. RLHF enables the agent to learn complex tasks faster, especially in scenarios where it is difficult to define a clear reward function. By combining human intuition and experience, RLHF helps improve the learning efficiency and performance of the agent.

6. The PPO algorithm according to claim 4, wherein the PPO comprises: A policy gradient optimization algorithm for policy search in reinforcement learning (RL); PPO aims to solve the stability and efficiency problems in the policy gradient method; it limits the step size of policy updates to ensure that the difference between the new policy and the old policy is not too large, thereby avoiding drastic fluctuations during training. The PPO algorithm adjusts the policy by using truncated probability ratios to make the policy update smoother while maintaining a high exploration efficiency. PPO performs well in continuous control tasks and complex environments and is a very popular algorithm in the current reinforcement learning field.

7. The LLaMA-2-70B model according to claim 4, wherein the LLaMA-2-70B comprises: The Large Language Model (LLM) open sourced by Meta (Facebook), which is the second generation of the LLaMA family of models and has approximately 70 billion parameters.

8. The CLIP according to claim 4, comprising: The abbreviation of Contrastive Language–Image Pre-training is a pre-trained machine learning model that is trained through a large number of image and text pairs to learn the association between images and texts.

9. The base model according to claim 4, characterized in that: The base model includes: Use the LLaMA-2-70B model as the language decoder and CLIP as the visual encoder; use the DeepSpeed-VisualChat architecture as a dedicated model training framework.

10. The data processing method according to claim 1, characterized in that: The drawings of the sub-projects and their partial 0# account data are used as annotation data, and the RLHF technology is used to train and optimize the model to obtain a model that can generate partial "0# account" through partial construction drawings, including: The ZeroModel model is specially trained again through RLHF (the RL algorithm here adopts PPO) technology to obtain a proprietary model that can generate part of the "0# ledger" through part of the construction drawings. For the convenience of subsequent description, it is recorded here as HZeroModel.

11. The data processing method according to claim 1, characterized in that: The method uses the design drawings as input and the sub-project data as output labeled data, uses the RLHF technology to train and optimize the model, and obtains a model that can split the design drawings according to the sub-projects, including: Use RLHF combined with labeled data to train the ZeroModel model to obtain a classification model, which is denoted as CZeroModel here for the convenience of subsequent description.

12. The data processing method according to claim 1, characterized in that: The construction drawing design data is input into the "split model", and the model automatically inputs the split data into the "0# ledger generation model" to generate local 0# ledger data for each sub-project, including: The "split model" is the CZeroModel described above, and the "0# ledger generation model" is the HZeroModel described above. The two models, CZeroModel and HZeroModel, are combined to generate scattered local "0# ledger" data based on the original design drawings.

13. The data processing method according to claim 1, characterized in that: The above mentioned integration of all local 0# account data to obtain the final 0# account includes: According to claim 10, all scattered local "0# ledger" data have been obtained, and the scattered local "0# ledger" data can be combined through Java, Python and other code programs to obtain the final "0# ledger".

14. A method for automatically generating 0# accounts from end to end, characterized in that You only need to input the design drawings into the model, and the 0# account will be automatically generated through the model: Input the design drawings into the system and use the CZeroModel+HZeroModel+combination code to get the 0# ledger.

Images