A data processing method and system based on a multi-modal large model

Abstract

The application provides a data processing method and system based on a multi-modal large model, relates to the technical field of data processing, and comprises the following steps: analyzing and identifying to-be-inspected data, and determining corresponding engineering types and inspection strategies; sending a query request to a pre-trained knowledge graph and a pre-trained vector knowledge base based on the engineering types and the inspection strategies, obtaining path retrieval results associated with structured knowledge and semantic retrieval results associated with unstructured knowledge, and fusing the path retrieval results and the semantic retrieval results to obtain fused retrieval results; the pre-trained knowledge graph comprises entities, attributes and the association relationships between the entities; the pre-trained vector knowledge base comprises a plurality of professional material text coding blocks; the to-be-inspected data is input into a pre-set inspection model for preprocessing and fine-grained element recognition, global macroscopic description, key path calculation and key information extraction are completed, reasoning is performed in combination with the fused retrieval results, and a structured inspection result is generated.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method and system based on a multimodal large model. Background Technology

[0002] With the deepening of the global energy internet strategy, smart grid construction has entered a phase of rapid acceleration. As a key infrastructure for building a modern energy system, the design quality of distribution network projects directly determines the reliability, economy, and intelligence level of grid operation. In recent years, distribution network projects have shown significant characteristics of exponential growth in construction scale and continuous increase in system complexity. Design deliverables encompass multimodal technical data such as electrical wiring diagrams, geographic information data, and equipment parameter tables, involving multiple interdisciplinary fields such as power system analysis, structural engineering, and geographic information systems, and including complex design stages such as load forecasting, grid optimization, equipment selection, and route planning. Traditional manual review methods, when dealing with large-scale, multi-dimensional design deliverables, reveal an inherent contradiction between efficiency and quality. Manual review is limited by individual differences in expert experience, the lag in knowledge updates, and subjective oversights caused by high-intensity work, making it difficult to meet the stringent requirements of smart grid construction for standardized and timely design reviews.

[0003] In recent years, the emergence of multimodal large models has provided new possibilities for data review. These models have demonstrated strong cross-modal correlation capabilities in natural image understanding, but their application in highly specialized power distribution network engineering scenarios is still in the exploratory stage. The models struggle to effectively integrate domain knowledge, leading to discrepancies between interpretation results and professional understanding. Furthermore, existing technical systems generally lack closed-loop review mechanisms, relying heavily on fixed thresholds to assess the confidence level of recognition results. They cannot dynamically adjust standards, nor can they automatically locate error types and provide correction paths, thus affecting the accuracy and reliability of power distribution networks. Summary of the Invention

[0004] In view of this, the present invention proposes a data processing method and system based on a multimodal large model.

[0005] The technical solution of this invention is implemented as follows: The first aspect of this invention provides a data processing method based on a multimodal large model, comprising: The data to be reviewed is analyzed and identified to determine the corresponding project type and review strategy; the data to be reviewed includes design specifications, design drawings, equipment list, and budget estimate; the review strategy includes the business boundaries of the review, key points of the review, and implementation standards. Based on the project type and the review strategy, query requests are sent to the pre-trained knowledge graph and the pre-trained vector knowledge base to obtain path retrieval results associated with structured knowledge and semantic retrieval results associated with unstructured knowledge. The path retrieval results and the semantic retrieval results are then fused to obtain fused retrieval results. The pre-trained knowledge graph includes entities, attributes, and relationships between entities. The pre-trained vector knowledge base includes multiple professional document text encoding blocks. The data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, completing global macro-description, critical path calculation, and key information extraction. The results are combined with the fusion retrieval results for reasoning to generate structured review results. The preset review model includes a multimodal sub-module, a professional vision sub-module, and a large language sub-module. The structured review results include problem description, original text location, standard basis, and modification suggestions.

[0006] Based on the above technical solutions, preferably, the step of parsing and identifying the data to be reviewed to determine the corresponding project type and review strategy includes: The data to be reviewed is parsed and standardized to transform the unstructured data into structured data carrying metadata. The Qwen3 large language model is used to identify the project type of the structured data, determine the corresponding project type, and match the corresponding special review strategy based on the project type.

[0007] Based on the above technical solutions, preferably, before sending a query request to the pre-trained knowledge graph and the pre-trained vector knowledge base based on the project type and the review strategy, the method further includes: The Qwen3 language model is used to perform semantic parsing and knowledge extraction on preprocessed professional materials, extracting entities, attributes and relationships between entities to obtain a pre-trained knowledge graph. The BGE-M3 model is used to slice and divide the preprocessed professional data into blocks. Each block of text is encoded into a semantic vector, and the generated semantic vectors are stored in a vector database according to the indexing rules to obtain a pre-trained vector knowledge base.

[0008] Based on the above technical solutions, preferably, the multimodal submodule is the qwen-vl model, and the professional vision submodule is the YOLOv8 model; the step of inputting the data to be reviewed into the preset review model for preprocessing and fine-grained component recognition to complete global macro-description, critical path calculation, and key information extraction includes: The YOLOv8 model is used to preprocess the electrical components of the power distribution network in the design drawings and perform end-to-end fine-grained component identification. The identified electrical components are used as the node set and the actual connection relationship between the components is used as the edge set to construct an undirected graph model describing the local electrical connection relationship of the design drawings. The overall structure and design intent of the undirected graph model are analyzed using the qwen-vl model to generate a global macro description, which is then used as a review criterion.

[0009] Based on the above technical solutions, preferably, the step of inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component identification to complete global macro-description, critical path calculation, and key information extraction includes: A macroscopic graph model is constructed based on the global macroscopic description; Mapping the nodes in the undirected graph model to the nodes in the macrograph model yields the corresponding full graph model; After performing connectivity and closure checks on the full-map model, the electrical distance information from the power source to the load point is calculated based on the Dijkstra algorithm, and the electrical distance information is used as the review criterion.

[0010] Based on the above technical solutions, preferably, the large language submodule is the Qwen3 large language model; the step of inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, to complete global macro-description, critical path calculation, and key information extraction, includes: The Qwen3 model is used to extract key information from the unstructured text and tables in the data to be reviewed, and the extracted key information is associated with the corresponding nodes in the full-map model to obtain an information-enhanced engineering model, which is then used as the review criterion. The key information includes equipment model, parameter specifications, installation location, and design standards.

[0011] Based on the above technical solutions, preferably, the step of inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, completing global modeling, critical path calculation, and key information extraction, and combining the fusion retrieval results for reasoning to generate structured review results includes: The global macro description, the electrical distance information, and the key information are summarized to obtain summarized information, and the fusion retrieval results obtained from the pre-trained knowledge graph and the pre-trained vector knowledge base are called. The mind chain prompting engineering technique guides the large language model to perform step-by-step reasoning on the summarized information and the fused retrieval results, generating structured review results.

[0012] Furthermore, a second aspect of the present invention provides a data processing system based on a multimodal large model, comprising: a parsing and identification module, a query fusion module, and an inference and generation module; wherein, The parsing and identification module is configured to parse and identify the data to be reviewed, and determine the corresponding project type and review strategy; the data to be reviewed includes design specifications, design drawings, equipment list and budget; the review strategy includes the business boundaries of the review, key points of the review and the execution standards. The query fusion module is configured to send query requests to a pre-trained knowledge graph and a pre-trained vector knowledge base based on the project type and the review strategy, obtain path retrieval results associated with structured knowledge and semantic retrieval results associated with unstructured knowledge, and fuse the path retrieval results and the semantic retrieval results to obtain a fused retrieval result; the pre-trained knowledge graph includes entities, attributes, and relationships between entities; the pre-trained vector knowledge base includes multiple professional data text encoding blocks; The reasoning generation module is configured to input the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, complete global macro-description, critical path calculation and key information extraction, and perform reasoning based on the fusion retrieval results to generate structured review results. The preset review model includes a multimodal sub-module, a professional vision sub-module and a large language sub-module, and the structured review results include problem description, original text location, standard basis and modification suggestions.

[0013] More preferably, a third aspect of the present invention provides an electronic device, including a processor and a memory; the memory has a computer program stored thereon, wherein the computer program, when executed by the processor, implements the data processing method based on a multimodal large model as described in the first aspect.

[0014] More preferably, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data processing method based on a multimodal large model as described in the first aspect.

[0015] The data processing method and system based on a multimodal large model of the present invention have the following advantages over the prior art: 1. By sending query requests to a pre-trained knowledge graph and a pre-trained vector knowledge base based on project type and review strategy, path retrieval results of structured knowledge associations and semantic retrieval results of unstructured knowledge associations are obtained. This approach acquires and utilizes domain knowledge from different perspectives, effectively solving the problem of models struggling to effectively integrate domain knowledge, thereby reducing the deviation between interpretation results and professional understanding. The data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, completing global macro-description, critical path calculation, and key information extraction. Combined with the fused retrieval results, reasoning is performed to generate structured review results containing problem descriptions, original text location, regulatory basis, and modification suggestions, forming a complete closed-loop review mechanism and improving the accuracy and reliability of review and subsequent power distribution network operations.

[0016] 2. During the query process, targeted queries are conducted based on the determined project type and review strategy to ensure that the acquired knowledge is highly relevant to the current review task. This improves the accuracy and effectiveness of domain knowledge fusion and helps the model generate review results that better meet professional requirements. Building on this, a multimodal submodule is used to process various data types, a professional vision submodule is used to provide professional interpretation of visual information such as design drawings, and a large language submodule is used to process text information for semantic understanding and analysis. Through the collaborative work of multiple modules, a global macro-level description, critical path calculation, and key information extraction are completed, providing comprehensive and accurate basic information for subsequent reviews.

[0017] 3. The generated review results not only pinpoint the problems but also clearly indicate their location in the original text, the applicable standards and norms, and specific modification suggestions, providing rich evidence for dynamically assessing the confidence level of the identification results. Furthermore, because the review results contain multifaceted information, the evaluation of the identification results no longer relies solely on fixed thresholds. Clear modification suggestions provide a correction path for automatically locating error types, improving the intelligence and practicality of the review process. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a data processing method based on a multimodal large model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data processing system based on a multimodal large model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] In some embodiments, such as Figure 1 As shown, Figure 1 The following is a flowchart illustrating a data processing method based on a multimodal large model provided by an embodiment of the present invention; the data processing method based on a multimodal large model provided by the present invention includes: S110 involves analyzing and identifying the data to be reviewed to determine the corresponding project type and review strategy. The data to be reviewed includes design specifications, design drawings, equipment lists, and cost estimates. The review strategy includes the business boundaries of the review, key points of the review, and implementation standards.

[0022] S120: Based on the project type and review strategy, a query request is sent to the pre-trained knowledge graph and the pre-trained vector knowledge base to obtain the path retrieval results associated with structured knowledge and the semantic retrieval results associated with unstructured knowledge. The path retrieval results and the semantic retrieval results are then fused to obtain the fused retrieval results. The pre-trained knowledge graph includes entities, attributes, and the relationships between entities. The pre-trained vector knowledge base includes multiple professional data text encoding blocks.

[0023] S130: Input the data to be reviewed into the preset review model for preprocessing and fine-grained component identification, complete the global macro description, critical path calculation and key information extraction, and perform reasoning based on the fusion retrieval results to generate structured review results; The preset review model includes a multimodal sub-module, a professional vision sub-module and a large language sub-module, and the structured review results include problem description, original text location, standard basis and modification suggestions.

[0024] In this embodiment, different types of data to be reviewed are parsed and identified to gain a comprehensive understanding of engineering-related information. A pre-trained knowledge graph stores entities, attributes, and relationships between entities in a structured manner, enabling rapid retrieval of structured knowledge paths related to the engineering project. For example, when reviewing electrical engineering, the knowledge graph can quickly locate connections and technical parameters between relevant equipment. A pre-trained vector knowledge base encodes multiple professional documents into vector form and uses semantic similarity for retrieval, enabling the acquisition of unstructured knowledge semantically related to the data to be reviewed. For instance, for some ambiguous statements in design specifications, similar professional explanations can be found from the vector knowledge base. Integrating path retrieval results and semantic retrieval results combines the accuracy of structured knowledge with the richness of unstructured knowledge, providing more comprehensive and accurate knowledge support for subsequent reviews.

[0025] The multimodal submodule in the pre-defined review model can handle various data types, such as text and images, preprocessing the data to be reviewed to make it more suitable for subsequent analysis. The professional vision submodule performs fine-grained component recognition on image data such as design drawings, accurately identifying various components in the drawings along with their positions and parameters. The large language submodule possesses powerful language understanding and generation capabilities, enabling tasks such as global macro-level description and key information extraction. Inference is performed using fused search results because these results provide rich professional knowledge and regulatory basis. Based on this knowledge, the review model can conduct in-depth analysis of the data to be reviewed, determine whether problems exist, and generate structured review results containing problem descriptions, original text locations, regulatory basis, and modification suggestions.

[0026] For example, natural language processing (NLP) technology is used to parse textual data such as design specifications and cost estimates to extract key information, such as project name, scale, and design requirements. For design drawings, image recognition technology is used to identify graphics, symbols, and text information. Equipment lists can be parsed through data format conversion and field matching. Based on the parsed key information, it is matched with predefined project type characteristics to determine the project type corresponding to the data to be reviewed. Then, based on the project type, an appropriate review strategy is selected from a pre-defined review strategy library, and a query statement is constructed.

[0027] For example, to review whether the performance parameters of a piece of equipment in an electrical engineering project comply with specifications, the query statement can include the equipment name, performance parameter keywords, etc. The query statement is sent to a pre-trained knowledge graph and a pre-trained vector knowledge base. Upon receiving the query request, the pre-trained knowledge graph performs a path search based on entities, attributes, and relationships between entities, finding structured knowledge relevant to the query and returning the path retrieval results. The pre-trained vector knowledge base converts the query statement into vector form, calculates semantic similarity with the encoded text blocks of professional materials in the library, finds the most semantically relevant unstructured knowledge, and returns the semantic retrieval results. The two are then fused to obtain the fused detection result. Based on the fused retrieval result, inference analysis is performed on the data to be reviewed to determine if there are any violations of the review strategy. If a problem exists, a problem description is generated, clarifying the specific content of the problem and locating its position in the original text to facilitate reviewers' retrieval. Additionally, regulatory basis is provided, explaining which standards and specifications the problem violates, and modification suggestions are given to guide reviewers in rectification. Finally, a structured review result containing a problem description, original text location, regulatory basis, and modification suggestions is generated.

[0028] In some embodiments, the data to be reviewed is parsed and identified to determine the corresponding project type and review strategy, including: The data to be reviewed is parsed and standardized to transform unstructured data into structured data carrying metadata. The Qwen3 large language model is used to identify the project type of structured data, determine the corresponding project type, and match the corresponding special review strategy based on the project type.

[0029] In this embodiment, a multimodal data source containing design specifications, design drawings, equipment lists, and cost estimates is obtained from the engineering management system. The file formats can include Word, PDF, Excel, JPG, PNG images, and CAD drawings. Unstructured data is parsed and converted, including using the PyMuPDF library to parse PDF documents and combining it with OCR (Optical Character Recognition) technology to recognize text and images, and calling the ezdxf library to read CAD vector data and convert it into image format. Metadata tags containing source file, data type, and location information are added to all parsed data blocks to achieve standardized data access. The Qwen3 large language model identifies the engineering type of the structured data and extracts key features related to the engineering type. Here, engineering types include cable line engineering, switch station engineering, and transformer substation renovation engineering, etc. Based on the identified engineering type, the corresponding review configuration is loaded from the review strategy library. The review configuration includes a completeness review checklist, professional technical review focus, and knowledge base query path corresponding to different engineering types.

[0030] In some embodiments, before sending a query request to the pre-trained knowledge graph and the pre-trained vector knowledge base based on the project type and review strategy, the method further includes: The Qwen3 language model is used to perform semantic parsing and knowledge extraction on preprocessed professional materials, extracting entities, attributes and relationships between entities to obtain a pre-trained knowledge graph. The BGE-M3 model is used to slice and divide the preprocessed professional data into blocks. Each block of text is encoded into a semantic vector, and the generated semantic vectors are stored in a vector database according to the indexing rules to obtain a pre-trained vector knowledge base.

[0031] In this embodiment, the Qwen3 large language model is used to perform semantic parsing and knowledge extraction on preprocessed professional materials, extracting entities, attributes, and relationships between entities. The extracted relationships are then stored in a graph database according to a standardized format, constructing a structured knowledge graph that supports relational reasoning and multi-hop queries, enabling structured organization and efficient relational retrieval of professional knowledge. The graph database can be Neo4j, which supports complex relational path queries and reasoning. Furthermore, a pre-trained semantic vector model is used to slice the full text of the preprocessed professional materials, encoding each text block into a semantic vector to achieve vectorized representation of text information. The generated semantic vectors are stored in a vector database according to indexing rules, constructing a vector knowledge base that supports semantic fuzzy retrieval and similarity matching, enabling efficient semantic retrieval of unstructured professional knowledge. The pre-trained semantic vector model can be the BGE-M3 model, with text slicing using a sliding window mechanism; the window size and step size can be adaptively adjusted according to the text length. The vector database uses Milvus, supporting fast retrieval and similarity ranking of large-scale vectors.

[0032] In the process of fusing path retrieval results and semantic retrieval results, firstly, in the graph database, the query request is converted into a Cypher statement for exact matching and related path queries; secondly, in the vector database, the FAISS vector retrieval tool is used to perform nearest neighbor search based on cosine similarity; after merging the two retrieval results, a relevance-based reordering algorithm is used for fusion, and the fused retrieval result is output. The fusion formula is as follows: , and These are the matching scores for graph databases and vector databases, respectively. , Select 0.5 and 0.5 respectively.

[0033] In some embodiments, the multimodal submodule is the qwen-vl model, and the professional vision submodule is the YOLOv8 model; the data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, completing global macro-description, critical path calculation, and key information extraction, including: The YOLOv8 model is used to preprocess the electrical components of the distribution network in the design drawings and identify fine-grained components end-to-end. The identified electrical components are used as the node set and the actual connection relationship between the components is used as the edge set to construct an undirected graph model that describes the local electrical connection relationship in the design drawings. The qwen-vl model is used to analyze the overall structure and design intent of the undirected graph model, generate a global macro description, and use the global macro description as a review criterion.

[0034] In this embodiment, the input design drawing image undergoes preprocessing to remove blur, noise, and enhance contrast, eliminating issues such as blurriness, broken lines, and background interference in the scanned engineering drawings. Specifically, this includes:

[0035] The Adaptive Histogram Equalization (CLAHE) algorithm is used to enhance the contrast of lines in the drawing. Its core calculation formula is: ; in, For the drawings before preprocessing Pixel value at coordinates and These are the minimum and maximum values ​​of the drawing, respectively. This represents the pixel value at that coordinate after preprocessing.

[0036] Combining morphological opening operations to remove salt-and-pepper noise and background interference, the morphological opening operation consists of erosion followed by dilation operations. The erosion operation formula is as follows: ; The expansion operation formula is: ; in, The image of the drawing after being processed by CLAHE. This is a pre-defined 3×3 rectangular structural element, i.e., a set of adjacent elements.

[0037] Preprocessed drawing images Input a YOLOv8 model finely adjusted from power distribution network engineering drawings, perform end-to-end target detection and classification of electrical components in the drawings, and output the category label for each component. Bounding box coordinates and confidence level ,in, The coordinates of the top left corner of the bounding box. The coordinates are the lower right corner of the bounding box, and the confidence threshold is set to 0.8. Recognition results below the threshold are considered invalid. The categories of components to be recognized include transformers, circuit breakers, disconnect switches, cables, poles, various graphic symbols and text labels, etc.

[0038] Electrical components identified by the YOLOv8 model are used as node sets. The actual connection relationships between the extracted components are used as edge sets. Construct an undirected graphical model to describe the local electrical connection relationships in the drawings. ,in, For the number of components, nodes The attribute contains its category label. Boundary frame and confidence level The graph model uses an adjacency matrix. storage.

[0039] Preprocessed drawing images The input is a qwen-vl model fine-tuned with domain data. Through instruction-based fine-tuning, the model executes a global image understanding task, outputting a global macroscopic description of the drawing. Key topological information is clearly annotated in a structured format, making it an undirected graph model. The overall analysis provides contextual constraints and guidance. Specific content includes: engineering scenario types. This includes main wiring diagrams, route diagrams, single-line diagrams, and transformer substation diagrams, clearly defining the engineering scenarios corresponding to the drawings and providing a basis for selecting rules in topology analysis; backbone network... ,in, This is the node sequence corresponding to the trunk line. The node sequence is arranged sequentially according to the trunk line's direction, clearly defining the node composition and connection order of the trunk line, providing a macro-level basis for verifying connection relationships; branch line set. ,in, This is the node sequence corresponding to this branch line. For the connection nodes between branch lines and trunk lines, the node sequence comes from the corresponding trunk line. This clarifies the node composition of the branch lines and their connection points with the main line, and is used to verify the rationality of the branch connections.

[0040] In some embodiments, the data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, completing global macro-level description, critical path calculation, and key information extraction, including: Construct a macroscopic graph model based on a global macroscopic description; Mapping nodes in an undirected graph model to nodes in a macrograph model yields the corresponding full graph model; After performing connectivity and closure checks on the full-map model, the electrical distance information from the power source point to the load point is calculated based on the Dijkstra algorithm, and the electrical distance information is used as the review criterion.

[0041] In this embodiment, a macroscopic graph model is constructed based on the output of the qwen-vl model. ,in, It is a set of key nodes, including key nodes in the main line node sequence, branch line node sequence, branch connection nodes, etc. It is a set of macroscopic topological relationships, including the connection relationships of the trunk line node sequence, the connection relationships between the branch line node sequence and the trunk line connection nodes, etc.

[0042] Undirected graph model Nodes in With macroscopic graph models Nodes in Mapping is performed to form a unified full-graph model. ,in, Using the fused full-map model Based on this, a graph algorithm is used to automatically verify the correctness of the electrical topology, providing quantified topology feature input for compliance review. A depth-first search (DFS) algorithm is employed to verify whether the drawings meet basic connectivity requirements. The recursive formula for the DFS algorithm is: ,in, For nodes The algorithm uses a set of adjacent nodes to quickly determine whether there are connected paths between nodes, records corresponding missing paths, and identifies problems such as topological breaks. It also employs the Tarjan algorithm to detect unexpected closed loops, recording the discovery time of nodes through depth-first search. with low value When a node is discovered The low value is equal to its discovery time, i.e. When a loop is detected, the loop detection result is crucial for determining the rationality of the protection configuration and avoiding short-circuit risks. For unexpected loops detected, the set of nodes contained in the loop is output. And the connections and relationships, providing a clear basis for review.

[0043] Dijkstra's algorithm, a graph-based shortest path algorithm, calculates the electrical distance from the power source to the load. Its core formula is: ,in, For the edge The weights are calculated, and the results are used to assist in verifying whether the selected conductors meet the voltage drop requirements, ensuring the safety and rationality of the engineering design. If the calculated electrical distance exceeds the allowable range of the specification, it is marked as an unreasonable conductor selection problem and associated with the corresponding technical specification.

[0044] In some embodiments, the large language submodule is the Qwen3 large language model; the data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, completing global macro-description, critical path calculation, and key information extraction, including: The Qwen3 model is used to extract key information from unstructured text and tables in the data to be reviewed. The extracted key information is then associated with the corresponding nodes in the full-map model to obtain an information-enhanced engineering model, which is used as the review criterion. The key information includes equipment model, parameter specifications, installation location, and design standards.

[0045] In this embodiment, the Qwen3 model is used to extract key information from the unstructured text and tables in the design specifications and equipment list. The extracted key information includes equipment model, parameter specifications, installation location, design standards, etc., and this information is associated with the full-map model. The corresponding nodes in the model form an information-enhanced engineering model. ,in, This provides a set of node attributes, supplementing key information such as device parameters, and providing data support for subsequent deep inference.

[0046] In some embodiments, the data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, completing global modeling, critical path calculation, and key information extraction. Inference is then performed using the fused retrieval results to generate structured review results, including: The global macro description, electrical distance information and key information are summarized to obtain the summarized information, and the fusion retrieval results obtained from the pre-trained knowledge graph and the pre-trained vector knowledge base are called. The mind chain prompting engineering technique guides the large language model to perform step-by-step reasoning on the summarized information and the fused retrieval results, generating structured review results.

[0047] In this embodiment, the macro-level description, calculation results, and extracted key information are summarized, and the knowledge base query module is called to match relevant technical specifications. The thinking chain prompting engineering technology is used to guide the large language model to perform step-by-step reasoning, and generate a structured review result containing problem description, original text location, standard basis, and modification suggestions.

[0048] In an optional embodiment, the LoRA method is used to fine-tune the large language submodule and the multimodal submodule. The core formula is: ,in, This is the original weight matrix of the pre-trained model. To pass through a low-rank matrix and The incremental weights are achieved through the product of these factors, thus maintaining model performance while significantly reducing the number of training parameters; the loss function during training is an adaptive weighted loss. ,in, To describe the generation loss, For visual question answering loss, For image-text matching loss, Select values ​​of 0.4, 0.3, and 0.3 respectively.

[0049] For the professional vision submodule, its training data comes from a large number of finely annotated power distribution network engineering design drawings, with annotation categories covering transformers, circuit breakers, disconnect switches, cables, poles, various graphic symbols, and text annotations; its training loss function is an adaptive weighted loss. ,in, For Varifocal Loss, For CIoU Loss, Select 0.5 and 0.5 respectively.

[0050] In some embodiments, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a data processing system based on a multimodal large model provided in an embodiment of the present invention. The present invention provides a data processing system 200 based on a multimodal large model, including: a parsing and identification module 210, a query fusion module 220, and an inference and generation module 230; wherein,

[0051] The parsing and identification module 210 is configured to parse and identify the data to be reviewed, and determine the corresponding project type and review strategy. The data to be reviewed includes design specifications, design drawings, equipment list and budget; the review strategy includes the business boundaries of the review, the key points of the review and the implementation specifications. The query fusion module 220 is configured to send query requests to the pre-trained knowledge graph and the pre-trained vector knowledge base based on the project type and review strategy, obtain path retrieval results associated with structured knowledge and semantic retrieval results associated with unstructured knowledge, and fuse the path retrieval results and semantic retrieval results to obtain the fused retrieval results; the pre-trained knowledge graph includes entities, attributes and relationships between entities; the pre-trained vector knowledge base includes multiple professional data text encoding blocks; The reasoning generation module 230 is configured to input the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, complete global macro description, critical path calculation and key information extraction, and perform reasoning based on the fusion retrieval results to generate structured review results. The preset review model includes a multimodal submodule, a professional vision submodule and a large language submodule. The structured review results include problem description, original text location, standard basis and modification suggestions.

[0052] In some embodiments, the parsing and identification module 210 is specifically configured as follows: The data to be reviewed is parsed and standardized to transform unstructured data into structured data carrying metadata. The Qwen3 large language model is used to identify the project type of structured data, determine the corresponding project type, and match the corresponding special review strategy based on the project type.

[0053] In some embodiments, the data processing system 200 based on a multimodal large model further includes a pre-training module; the pre-training module is specifically configured as follows: The Qwen3 language model is used to perform semantic parsing and knowledge extraction on preprocessed professional materials, extracting entities, attributes and relationships between entities to obtain a pre-trained knowledge graph. The BGE-M3 model is used to slice and divide the preprocessed professional data into blocks. Each block of text is encoded into a semantic vector, and the generated semantic vectors are stored in a vector database according to the indexing rules to obtain a pre-trained vector knowledge base.

[0054] In some embodiments, the multimodal submodule is a qwen-vl model, and the professional vision submodule is a YOLOv8 model; the inference generation module 230 is specifically configured as follows: The YOLOv8 model is used to preprocess the electrical components of the distribution network in the design drawings and identify fine-grained components end-to-end. The identified electrical components are used as the node set and the actual connection relationship between the components is used as the edge set to construct an undirected graph model that describes the local electrical connection relationship in the design drawings. The qwen-vl model is used to analyze the overall structure and design intent of the undirected graph model, generate a global macro description, and use the global macro description as a review criterion.

[0055] In some embodiments, the inference generation module 230 is specifically configured as follows: Construct a macroscopic graph model based on a global macroscopic description; Mapping nodes in an undirected graph model to nodes in a macrograph model yields the corresponding full graph model; After performing connectivity and closure checks on the full-map model, the electrical distance information from the power source point to the load point is calculated based on the Dijkstra algorithm, and the electrical distance information is used as the review criterion.

[0056] In some embodiments, the large language submodule is the Qwen3 large language model; the reasoning generation module 230 is specifically configured as follows: The Qwen3 model is used to extract key information from unstructured text and tables in the data to be reviewed. The extracted key information is then associated with the corresponding nodes in the full-map model to obtain an information-enhanced engineering model, which is used as the review criterion. The key information includes equipment model, parameter specifications, installation location, and design standards.

[0057] In some embodiments, the inference generation module 230 is specifically configured as follows: The global macro description, electrical distance information and key information are summarized to obtain the summarized information, and the fusion retrieval results obtained from the pre-trained knowledge graph and the pre-trained vector knowledge base are called. The mind chain prompting engineering technique guides the large language model to perform step-by-step reasoning on the summarized information and the fused retrieval results, generating structured review results.

[0058] It should be noted that the data processing system based on a multimodal large model provided in this application embodiment and the data processing method based on a multimodal large model provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned data processing method based on a multimodal large model, and the repeated parts will not be described again.

[0059] In some embodiments, please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 300 provided in this application includes a processor 310 and a memory 320; the memory 320 stores a computer program, wherein the computer program, when executed by the processor, implements the aforementioned data processing method based on a multimodal large model.

[0060] Specifically, processor 310 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 310 may also include onboard memory for caching purposes. Processor 310 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0061] The memory 320 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, the memory 320 may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, apparatuses, or propagation media. Specific examples of the memory 320 include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and may also be random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0062] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned data processing method based on a multimodal large model. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0063] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0064] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A data processing method based on a multimodal large model, characterized in that, include: The data to be reviewed is analyzed and identified to determine the corresponding project type and review strategy. The data to be reviewed includes design specifications, design drawings, equipment lists, and cost estimates; the review strategy includes the business boundaries of the review, key points of the review, and implementation standards. Based on the project type and the review strategy, query requests are sent to the pre-trained knowledge graph and the pre-trained vector knowledge base to obtain path retrieval results associated with structured knowledge and semantic retrieval results associated with unstructured knowledge. The path retrieval results and the semantic retrieval results are then fused to obtain fused retrieval results. The pre-trained knowledge graph includes entities, attributes, and relationships between entities. The pre-trained vector knowledge base includes multiple professional document text encoding blocks. The data to be reviewed is input into a preset review model for preprocessing and fine-grained component identification, to complete global macro description, critical path calculation and key information extraction, and to generate structured review results by combining the fusion retrieval results; The preset review model includes a multimodal submodule, a professional visual submodule, and a large language submodule. The structured review results include problem description, original text location, standard basis, and modification suggestions.

2. The data processing method based on a multimodal large model as described in claim 1, characterized in that, The process of parsing and identifying the data to be reviewed, and determining the corresponding project type and review strategy, includes: The data to be reviewed is parsed and standardized to transform the unstructured data into structured data carrying metadata. The Qwen3 large language model is used to identify the project type of the structured data, determine the corresponding project type, and match the corresponding special review strategy based on the project type.

3. The data processing method based on a multimodal large model as described in claim 1, characterized in that, The method further includes, before sending a query request to the pre-trained knowledge graph and the pre-trained vector knowledge base based on the project type and the review strategy: The Qwen3 language model is used to perform semantic parsing and knowledge extraction on preprocessed professional materials, extracting entities, attributes and relationships between entities to obtain a pre-trained knowledge graph. The BGE-M3 model is used to slice and divide the preprocessed professional data into blocks. Each block of text is encoded into a semantic vector, and the generated semantic vectors are stored in a vector database according to the indexing rules to obtain a pre-trained vector knowledge base.

4. The data processing method based on a multimodal large model as described in claim 1, characterized in that, The multimodal submodule is the qwen-vl model, and the professional vision submodule is the YOLOv8 model; the process of inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component recognition, to complete global macro-description, critical path calculation, and key information extraction, includes: The YOLOv8 model is used to preprocess the electrical components of the power distribution network in the design drawings and perform end-to-end fine-grained component identification. The identified electrical components are used as the node set and the actual connection relationship between the components is used as the edge set to construct an undirected graph model describing the local electrical connection relationship of the design drawings. The overall structure and design intent of the undirected graph model are analyzed using the qwen-vl model to generate a global macro description, which is then used as a review criterion.

5. The data processing method based on a multimodal large model as described in claim 4, characterized in that, The process of inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, thereby completing global macro-level description, critical path calculation, and key information extraction, includes: A macroscopic graph model is constructed based on the global macroscopic description; Mapping the nodes in the undirected graph model to the nodes in the macrograph model yields the corresponding full graph model; After performing connectivity and closure checks on the full-map model, the electrical distance information from the power source to the load point is calculated based on the Dijkstra algorithm, and the electrical distance information is used as the review criterion.

6. The data processing method based on a multimodal large model as described in claim 5, characterized in that, The large language submodule is the Qwen3 large language model; the process of inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, completing global macro-description, critical path calculation, and key information extraction, includes: The Qwen3 model is used to extract key information from the unstructured text and tables in the data to be reviewed, and the extracted key information is associated with the corresponding nodes in the full-map model to obtain an information-enhanced engineering model, which is then used as the review criterion. The key information includes equipment model, parameter specifications, installation location, and design standards.

7. The data processing method based on a multimodal large model as described in claim 6, characterized in that, The process involves inputting the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, completing global modeling, critical path calculation, and key information extraction, and then combining the fusion retrieval results for reasoning to generate structured review results, including: The global macro description, the electrical distance information, and the key information are summarized to obtain summarized information, and the fusion retrieval results obtained from the pre-trained knowledge graph and the pre-trained vector knowledge base are called. The mind chain prompting engineering technique guides the large language model to perform step-by-step reasoning on the summarized information and the fused retrieval results, generating structured review results.

8. A data processing system based on a multimodal large model, characterized in that, include: The module comprises a parsing and recognition module, a query fusion module, and a reasoning and generation module; among which, The parsing and identification module is configured to parse and identify the data to be reviewed, and determine the corresponding project type and review strategy; the data to be reviewed includes design specifications, design drawings, equipment list and budget; the review strategy includes the business boundaries of the review, key points of the review and the execution standards. The query fusion module is configured to send query requests to a pre-trained knowledge graph and a pre-trained vector knowledge base based on the project type and the review strategy, obtain path retrieval results associated with structured knowledge and semantic retrieval results associated with unstructured knowledge, and fuse the path retrieval results and the semantic retrieval results to obtain a fused retrieval result; the pre-trained knowledge graph includes entities, attributes, and relationships between entities; the pre-trained vector knowledge base includes multiple professional data text encoding blocks; The reasoning generation module is configured to input the data to be reviewed into a preset review model for preprocessing and fine-grained component identification, complete global macro description, critical path calculation and key information extraction, and perform reasoning in combination with the fusion retrieval results to generate structured review results; The preset review model includes a multimodal submodule, a professional visual submodule, and a large language submodule. The structured review results include problem description, original text location, standard basis, and modification suggestions.

9. An electronic device comprising a processor and a memory; said memory storing a computer program, wherein, When the computer program is executed by the processor, it implements the data processing method based on a multimodal large model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the data processing method based on a multimodal large model as described in any one of claims 1 to 7.

Images