Method and related device for determining decision information based on multi-modal data

Abstract

The application provides a decision information determination method based on multi-modal data and related equipment, the method comprises the following steps: obtaining multi-modal data, and respectively preprocessing each kind of modal data to obtain each kind of preprocessed modal data; performing feature fusion on all preprocessed modal data to obtain fusion features; based on the fusion features, using a pre-constructed knowledge graph and a pre-constructed industry knowledge base, determining industry knowledge prompt information; based on the fusion features and the industry knowledge prompt information, using a pre-constructed decision model, determining decision information corresponding to the multi-modal data, solving the technical problem that the determined decision information in the prior art is not accurate, and improving the accuracy and efficiency of determining decision information.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and related equipment for determining decision information based on multimodal data. Background Technology

[0002] With the continuous advancement of the construction of new power systems, the complexity of power industry operations has increased significantly. Core scenarios such as engineering construction, equipment operation and maintenance, and dispatching have generated massive amounts of multimodal heterogeneous data, and the industry's requirements for the intelligence, accuracy, and timeliness of decision analysis are constantly increasing.

[0003] Existing technologies have limited means of processing multi-source heterogeneous data, lack a unified data representation and alignment system, and exhibit semantic barriers across modal data, resulting in significant data silos. Meanwhile, traditional decision analysis processes heavily rely on manual data processing, correlation analysis, and solution formulation. When faced with massive amounts of multimodal data, these processes are not only inefficient but also susceptible to subjective judgments, missing information, and insufficient data fusion, leading to decision-making biases and inaccurate decision information. This severely hinders the digital transformation of the power industry. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method and related equipment for determining decision information based on multimodal data, so as to overcome all or part of the shortcomings of the prior art.

[0005] To achieve the above objectives, this application provides a method for determining decision information based on multimodal data, comprising: acquiring multimodal data and preprocessing each type of modality data to obtain each type of preprocessed modality data; performing feature fusion on all preprocessed modality data to obtain fused features; determining industry knowledge prompts based on the fused features using a pre-built knowledge graph and a pre-built industry knowledge base; and determining the decision information corresponding to the multimodal data using a pre-built decision model based on the fused features and the industry knowledge prompts.

[0006] Optionally, the step of fusing features from all preprocessed modal data to obtain fused features includes: mapping all preprocessed modal data to a unified semantic vector space to obtain a semantically aligned feature vector for each type; and using a predetermined spatiotemporal attention fusion algorithm to perform weighted fusion on each semantically aligned feature vector to obtain the fused features.

[0007] Optionally, determining industry knowledge prompts based on the fusion features, using a pre-built knowledge graph and a pre-built industry knowledge base, includes: performing semantic extraction and structural transformation on the fusion features to obtain multiple types of semantic information; searching for multiple first-related information corresponding to each type of semantic information in the knowledge graph; using the multiple types of semantic information and the multiple first-related information to search the industry knowledge base to obtain multiple second-related information; and determining the industry knowledge prompts based on the multiple types of semantic information, the multiple first-related information, and the multiple second-related information.

[0008] Optionally, determining the industry knowledge prompt information based on the multiple types of semantic information, the multiple first association information, and the multiple second association information includes: extracting information from the multiple types of semantic information, the multiple first association information, and the multiple second association information using a pre-built prompt template to obtain initial industry knowledge prompt information; and logically adjusting the initial industry knowledge prompt information to obtain the industry knowledge prompt information.

[0009] Optionally, before determining industry knowledge prompts based on the fusion features using a pre-built knowledge graph and a pre-built industry knowledge base, the method includes: performing constraint verification on the fusion features using a pre-built rule engine; and generating warning information in response to determining that the fusion features have failed the constraint verification.

[0010] Optionally, after determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts using a pre-built decision model, the method includes: performing constraint verification on the decision information using a pre-built rule engine; in response to determining that there is information in the decision information that fails the constraint verification, replacing the information that fails the constraint verification using the rule engine to obtain corrected decision information; and optimizing the decision model using the fusion features, the industry knowledge prompts, and the corrected decision information.

[0011] Optionally, after determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts using a pre-built decision model, the method includes: receiving human feedback information corresponding to the decision information; and optimizing the decision model using the fusion features, the industry knowledge prompts, and the human feedback information.

[0012] Based on the same inventive concept, this application also provides a decision information determination device based on multimodal data, comprising: a processing module configured to acquire multimodal data and preprocess each type of modality data to obtain each type of preprocessed modality data; a fusion module configured to perform feature fusion on all preprocessed modality data to obtain fused features; a first determination module configured to determine industry knowledge prompt information based on the fused features, using a pre-built knowledge graph and a pre-built industry knowledge base; and a second determination module configured to determine the decision information corresponding to the multimodal data based on the fused features and the industry knowledge prompt information, using a pre-built decision model.

[0013] Based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0014] Based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to perform the method described above.

[0015] As described above, the decision information determination method and related equipment based on multimodal data provided in this application include: acquiring multimodal data and preprocessing each type of data separately to obtain each preprocessed modal data. Dual preprocessing (general and specific) is performed for each modality, adapting to various data characteristics, effectively removing redundant noise, standardizing data formats, and ensuring data quality for each modality. Feature fusion is performed on all preprocessed modal data to obtain fused features, effectively capturing complex relationships between cross-modal data, thereby obtaining fused features with strong representational capabilities and semantic consistency. Based on the fused features, industry knowledge prompts are determined using a pre-built knowledge graph and a pre-built industry knowledge base, enriching the fused features from a professional perspective, making the industry knowledge prompts more accurate and aligned with actual business needs. Based on the fused features and the industry knowledge prompts, a pre-built decision model is used to determine the decision information corresponding to the multimodal data, significantly improving the accuracy of the decision information. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the decision information determination method based on multimodal data according to an embodiment of this application.

[0018] Figure 2 This is a schematic diagram illustrating the process of determining fusion features in an embodiment of this application; Figure 3 This is a flowchart illustrating the intelligent decision-making process in an embodiment of this application. Figure 4 This is a flowchart illustrating a decision information determination method based on multimodal data according to another embodiment of this application. Figure 5 This is a schematic diagram of the structure of the decision information determination device based on multimodal data according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0020] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] As described in the background section, with the in-depth advancement of the construction of new power systems, the complexity of power industry operations has increased significantly. Core scenarios such as engineering construction, equipment operation and maintenance, and operation control have generated massive amounts of multimodal heterogeneous data, leading to increasingly higher requirements for intelligent, precise, and real-time decision analysis. Traditional decision analysis processes rely on manual work to complete tasks such as multi-source data processing, correlation analysis, and scheme evaluation. Faced with massive amounts of multimodal data, this is not only time-consuming and labor-intensive, but also prone to decision-making biases due to subjective judgment, information omissions, and insufficient data fusion. The determined decision information is inaccurate, seriously hindering the digital transformation process of the power industry.

[0022] To address this issue, the industry has gradually introduced artificial intelligence and multimodal fusion technologies, resulting in three main technological paths: 1. Single-modal intelligent decision-making systems: These systems process only single-type data such as text or images, providing decision assistance through simple rule matching or machine learning models. Their core relies on feature extraction and analysis of single-modal data. Representative technologies include text semantic analysis tools (such as Sentence-BERT), image recognition models (such as the YOLO series), and single-modal feature extraction algorithms (such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network)). These systems are widely used for preliminary screening in simple scenarios within the power industry.

[0023] Limitations: It cannot handle multimodal heterogeneous data, making it difficult to achieve correlation and fusion of different types of data. Decisions are based on a single basis, resulting in low decision accuracy. It lacks adaptation to professional knowledge in the power industry, failing to meet the decision-making needs of complex scenarios. Different modal systems are independent of each other, and data cannot be shared, forming "data silos" and leading to low decision-making efficiency.

[0024] 2. Decision-making schemes based on traditional fusion algorithms: These schemes fuse multimodal data using simple feature concatenation and weighted summation, and combine this with traditional machine learning models (such as SVM and random forest) to achieve decision analysis. The core relies on manually designed fusion rules and feature engineering. Representative technologies include multimodal feature concatenation algorithms, weighted fusion models, traditional machine learning classifiers, and simple rule engines (such as Drools).

[0025] Limitations: The fusion method is simple and cannot solve the problem of cross-modal semantic barriers, resulting in poor fusion effects; feature engineering relies on manual design, which is labor-intensive and has poor adaptability, making it difficult to quickly adjust to the diverse scenarios in the power industry; the model has weak generalization ability and limited ability to process complex data formats (such as point clouds and engineering drawings); it lacks autonomous evolution capabilities, and the decision accuracy cannot be continuously optimized through business feedback. These limitations lead to inaccurate decision information.

[0026] 3. Multimodal Decision-Making Scheme Based on General-Purpose Large Models: This scheme calls the API (Application Programming Interface) of general-purpose large-scale multimodal models to perform unified processing and fusion of multimodal data. It leverages the semantic understanding capabilities of the large-scale model to achieve decision analysis, relying primarily on the general capabilities of the external model. Representative technologies include general-purpose large-scale multimodal models (such as GPT-4V (GPT-4 Visual Version) and Qwen-VL (Tongyi Qianwen Visual Large Model)), API calling interfaces, and simple data preprocessing tools.

[0027] Limitations: There is a risk of technical black boxes, insufficient understanding of professional business rules in the power industry, and significant deviations in decision-making results; the call chain is affected by network stability, resulting in poor real-time performance and inability to adapt to on-site decision-making scenarios; data needs to be uploaded to external platforms, posing a risk of leakage of core business data in the power industry (such as engineering drawings and equipment parameters); the model structure is complex, has high computing power requirements, and cannot be customized and optimized according to industry needs.

[0028] In summary, against the backdrop of digital transformation and the construction of new power systems in the power industry, core professional scenarios such as power engineering construction and intelligent operation and maintenance of power equipment face prominent problems such as difficulties in integrating multi-source heterogeneous data, insufficient decision-making accuracy, and low business management efficiency. General artificial intelligence technologies are difficult to adapt to the special business rules and data characteristics of the power industry. Existing intelligent decision-making technologies suffer from pain points such as semantic barriers to multimodal data, poor adaptability of fusion algorithms, lack of self-evolution capabilities in decision-making models, and insufficient depth of professional knowledge integration, failing to meet the power industry's needs for precise, efficient, and autonomously controllable decision-making.

[0029] Existing technologies face the following key technical challenges in realizing intelligent decision analysis in the power industry: 1. Insufficient multimodal data fusion: Traditional technologies have a single approach to processing heterogeneous data from multiple sources, such as text, images, tables, point clouds, and sensor data. They lack a unified representation and alignment mechanism, resulting in cross-modal semantic barriers and prominent "data silos." Engineering construction drawings and on-site images cannot be effectively matched, which cannot support complete decision-making and leads to inaccurate decision-making information.

[0030] 2. Limited Decision Accuracy: Existing fusion algorithms mostly rely on general models and lack in-depth injection of professional knowledge in the power industry. They have insufficient understanding of industry-specific business rules (such as construction safety specifications and equipment operation standards) and limited ability to parse complex data formats (such as multi-level tables and engineering drawings). This results in a large deviation between the decision results and expert judgments, and cannot meet the needs of professional scenarios.

[0031] 3. Low decision-making efficiency: Existing decision-making models are mostly general-purpose, with complex structures, high computing power requirements, and high deployment costs. Moreover, the multimodal data processing links are lengthy, resulting in excessively long response times for moderately complex decision-making tasks, making it difficult to meet the needs of real-time decision-making in the power industry.

[0032] 4. Insufficient model adaptability and evolutionary capability: The decision-making needs of different professional scenarios in the power industry (such as engineering construction and equipment operation and maintenance) vary greatly. The existing model strategy configuration is fixed, making it difficult to dynamically adapt to diverse scenarios. Moreover, it lacks an autonomous evolutionary mechanism of "decision-feedback-learning" and cannot continuously optimize decision accuracy based on business feedback.

[0033] 5. Poor interpretability of decisions: Most existing intelligent decision-making models are black-box models, lacking traceability in the decision-making process and failing to clearly define the basis for the generation of decision results. This makes it difficult to meet the compliance and verifiability requirements of the power industry and hinders expert review and problem identification.

[0034] In view of this, embodiments of this application propose a method for determining decision information based on multimodal data, referring to... Figure 1 This includes the following steps: Step 101: Obtain multimodal data and preprocess each type of modal data to obtain each type of preprocessed modal data.

[0035] This step involves acquiring multimodal data, which includes various data types such as text, images, tables, point clouds, and sensor data. The multimodal data can be data related to the power industry and can be obtained from business systems such as power industry engineering construction and equipment operation and maintenance.

[0036] Each modality of data is preprocessed separately to obtain preprocessed modal data. Preprocessing includes general preprocessing and specific preprocessing. General preprocessing, such as data cleaning, redundancy removal, standardization, and format conversion, is uniformly performed on each modality to remove invalid information and standardize data specifications. Based on the data characteristics of each modality, specific preprocessing is obtained and performed for that modality. For example, using tools such as OCR (Optical Character Recognition), point cloud parsing, and sensor data standardization, non-text data such as images and point clouds are converted into an encodeable format; text data is segmented and anonymized, ensuring that all heterogeneous multimodal data are processed into a unified and standardized form. Performing both general and specific preprocessing for each modality adapts to various data characteristics, effectively removes redundant noise, standardizes data formats, and ensures the data quality of each modality.

[0037] Step 102: Perform feature fusion on all preprocessed modal data to obtain fused features.

[0038] In this step, a graph neural network-based data fusion module is used to uniformly extract features and perform deep fusion on all preprocessed modal data, effectively capturing the complex relationships between cross-modal data, thereby obtaining fusion features with strong representational ability and semantic uniformity.

[0039] Step 103: Based on the fusion features, determine industry knowledge prompts using a pre-built knowledge graph and a pre-built industry knowledge base.

[0040] In this step, existing methods for determining decision-making information suffer from insufficient professional adaptability: due to a lack of in-depth input of power industry expertise, and the failure to optimize the model in conjunction with industry business rules such as power construction specifications, equipment operation standards, and maintenance procedures, the model's ability to analyze industry-specific data such as engineering drawings and equipment point clouds is limited. This results in an inability to accurately identify key professional qualifiers in the data, ultimately leading to a significant deviation between the decision-making results and the actual judgments of industry experts. To address this, this application pre-constructs a knowledge graph and industry knowledge base related to the power industry context. For example, the knowledge graph includes engineering construction specifications, equipment operation standards, risk cases, defect types, maintenance strategies, environmental protection requirements, and safety constraint rules. The industry knowledge base includes historical cases, construction specifications, maintenance standards, risk management strategies, and expert experience records. Based on fusion features, the pre-constructed knowledge graph and industry knowledge base are used to determine industry knowledge prompts. Based on fusion features, the content of the pre-constructed knowledge graph and industry knowledge base is retrieved to mine related professional information, enriching the fusion features from a professional perspective, making the industry knowledge prompts more accurate and aligned with actual business needs.

[0041] Step 104: Based on the fusion features and the industry knowledge prompts, the decision information corresponding to the multimodal data is determined using a pre-built decision model.

[0042] In this step, fused features objectively recreate the actual on-site working conditions, while industry knowledge prompts supplement professional rules and practical experience. Both fused features and industry knowledge prompts are input into a pre-trained decision model. This model incorporates a reinforcement learning mechanism, dynamically iteratively optimizing decision information by combining real-time input data with historical decision feedback. The decision model outputs decision information corresponding to multimodal data, which may include risk level, defect type, recommended handling plan, decision confidence level, and reasoning basis. Through a knowledge graph of the power industry context, professional knowledge is deeply linked to fused features, supporting intelligent decision-making. Combined with an industry knowledge base, reliable industry knowledge support is provided to the decision model, effectively addressing the problems of insufficient professional adaptability, weak ability to analyze special power data, and inability to identify professional qualifiers in existing methods. This approach relies on feature data to realistically recreate the details of on-site working conditions while leveraging professional knowledge to supplement business rules and practical experience, achieving dual support from real-world data and industry standards. It effectively avoids the one-sidedness of a single judgment method, ensuring that decision-making aligns with actual scenarios and regulatory requirements, significantly improving the accuracy of decision information.

[0043] Traditional solutions often rely on black-box models or simple rule-driven approaches, making the decision-making process difficult to trace and failing to clearly define the basis for the results. This fails to meet the compliance requirements for decision-making in the power industry and hinders expert review and troubleshooting. This application integrates multimodal fusion features with industry knowledge prompts for decision-making analysis. It leverages knowledge graphs and industry knowledge bases to provide clear professional basis for decisions. Simultaneously, the decision-making model retains complete deductive logic, ultimately outputting decision content with accompanying reasoning. The entire decision-making chain is clear and traceable, meeting industry compliance standards and facilitating expert review and verification for precise problem identification.

[0044] This application establishes a complete knowledge enhancement chain: "Knowledge Graph Storage—RAG Retrieval—Prompt Injection—Model Reasoning." A structured industry knowledge network is built from the knowledge graph. Retrieval enhancement technology retrieves scenario-appropriate knowledge as needed. The knowledge prompting module formats the information and feeds it into the decision model. The decision model combines fused features and industry knowledge to complete joint reasoning. A knowledge graph for the power industry is built based on graph neural networks (GNNs), aggregating content such as engineering construction specifications, equipment operation and maintenance standards, and risk cases. Through knowledge slicing and association mapping, a systematic knowledge network is formed. Simultaneously, knowledge prompting and retrieval enhancement generation (RAG) technologies are used to integrate industry rules and expert experience into model computation, and heterogeneous data parsing algorithms are optimized to effectively improve the parsing capabilities of professional data such as engineering drawings and point clouds.

[0045] The decision-making model is constructed using knowledge distillation, based on mainstream multimodal large models (such as LLaMA and ChatGLM), and distilled into a lightweight, dedicated model adapted to power scenarios with low computational consumption, meeting on-site deployment requirements. Simultaneously, a computable rule engine is introduced to transform business rules such as construction safety guidelines and equipment operating thresholds into executable algorithms, integrating rule-based reasoning and data-driven reasoning modes to ensure interpretability and traceability throughout the decision-making process. The system also establishes a "decision-feedback-learning" closed loop, using expert review feedback data as optimization samples, and leveraging reinforcement learning (RL) and human feedback optimization (RLHF) techniques to iterate model parameters and fusion strategies, achieving autonomous model evolution and upgrades.

[0046] In the actual processing stage, semantic parsing is first performed on the fused feature vectors to extract key content such as equipment status, spatial location, temporal information, and risk tags. Various information, including equipment defect areas, specification clauses, sensor anomaly data, and point cloud spatial relationships, are uniformly transformed into structured semantic tags. Then, entity links and relationship mappings are completed based on the power knowledge graph. Corresponding nodes and related networks are retrieved based on entity information to generate a scenario knowledge subgraph. This subgraph is then linked and matched with related content in the industry knowledge base, such as historical fault cases, operating thresholds, maintenance guidelines, risk levels, and handling plans. Using the feature semantic vectors as the retrieval basis, vector retrieval and graph path retrieval modes are integrated. Historical cases, operation and maintenance standards, risk strategies, and expert experience are retrieved from the industry knowledge base and combined with the fused features to form an enhanced context, allowing the model's judgment to simultaneously consider measured data and industry experience and standards.

[0047] Finally, by integrating and fusing features, search content, graph association paths, and rule constraints, standardized knowledge prompts are generated according to a predetermined template. This limits the model's reasoning logic and avoids deductions that violate industry standards. The entire mechanism significantly enhances the model's business adaptability. Experimental results show that equipment defect identification efficiency is improved by over 30%, multimodal fusion accuracy exceeds 85%, decision-making bias in complex scenarios is reduced by 60%, and construction risk warning response time is controlled within one minute, effectively reducing decision-making errors caused by a lack of professional knowledge.

[0048] It should be noted that traditional solutions suffer from low decision-making efficiency, complex fusion and decision-making chains, and time-consuming multimodal data processing. Furthermore, general-purpose models consume significant computing power and have high deployment costs, resulting in slow response times for moderately complex tasks, making them unsuitable for real-time decision-making scenarios in the power industry. This solution optimizes the process from two aspects: model lightweighting and processing chains. Firstly, a lightweight decision-making model adapted to power business scenarios is trained. This lightweight model is built using knowledge distillation technology, using a large multimodal model as the teacher model and selecting power industry scenario data as distillation samples. The semantic understanding and logical reasoning capabilities of the teacher model are transferred and reused to distill a lightweight decision-making model adapted to the power industry scenario, reducing computing power requirements. Secondly, optimization techniques such as parameter pruning, low-rank decomposition, INT8 quantization (8-bit integer quantization), and LoRA fine-tuning (low-rank adapter fine-tuning) are simultaneously applied to further compress the model parameter size and reduce computing power consumption during inference. The lightweighted model can be adapted for deployment on edge servers, industrial control terminals, field workstations, and other devices, effectively meeting the real-time decision-making requirements of the power industry.

[0049] On the other hand, the business processing flow is streamlined, referring to the entire processing chain from multimodal data access to the generation of decision results. The traditional four-layer processing chain is compressed into a streamlined architecture of multimodal data input, feature fusion, model decision-making, and result output, reducing information transmission losses and significantly shortening the overall processing cycle. Combined with a parallel processing mechanism, the task concurrency capacity is significantly improved. Experiments have verified that the response time for moderately complex decision tasks can be controlled within 50 seconds after optimization, without requiring high-end GPU equipment. With multi-node load distribution and a single-node multi-threaded parallel processing mode, a single node can simultaneously handle 20 to 30 decision tasks, reducing the overall decision-making process time by 30%, fully meeting the application standards for real-time decision-making in the power field.

[0050] The above scheme acquires multimodal data and preprocesses each modality separately to obtain preprocessed modal data. Dual preprocessing (general and specific) is performed for each modality to adapt to various data characteristics, effectively remove redundant noise, standardize data formats, and ensure data quality for each modality. Feature fusion is then performed on all preprocessed modal data to obtain fused features, effectively capturing complex relationships between cross-modal data, thus obtaining fused features with strong representational capabilities and semantic consistency. Based on these fused features, industry knowledge prompts are determined using a pre-built knowledge graph and a pre-built industry knowledge base, enriching the fused features from a professional perspective and making the industry knowledge prompts more accurate and aligned with actual business needs. Based on the fused features and the industry knowledge prompts, a pre-built decision model is used to determine the decision information corresponding to the multimodal data, significantly improving the accuracy of the decision information.

[0051] In some embodiments, the step of fusing features from all preprocessed modal data to obtain fused features includes: mapping all preprocessed modal data to a unified semantic vector space to obtain a semantically aligned feature vector for each type; and using a predetermined spatiotemporal attention fusion algorithm to perform weighted fusion on each semantically aligned feature vector to obtain the fused features.

[0052] In this embodiment, when performing feature fusion on the preprocessed modal data and obtaining fused features, the various modal data are first mapped to a unified semantic vector space, semantic alignment is completed to obtain the corresponding feature vectors, and then a predetermined spatiotemporal attention fusion algorithm is used to weight and fuse the vectors to finally generate fused features.

[0053] Traditional solutions often employ simple feature concatenation or independent processing of single modalities, lacking a unified representation and semantic alignment system. This fails to eliminate cross-modal semantic barriers, resulting in significant data silos. Furthermore, fused features struggle to reflect the inherent relationships between multi-source data, leading to insufficient decision-making basis and inaccurate judgment. For example... Figure 2As shown, Figure 2 This diagram illustrates the process of determining fusion features in an embodiment of this application. For preprocessed multimodal data, this solution builds a cross-modal unified encoder based on a Transformer, introduces modality alignment technology, and relies on contrastive learning to complete model training and application. It normalizes and maps different types of data, such as text, images, point clouds, sensor time-series data, and Word and PDF format documents, achieving semantic dimension alignment and fundamentally breaking down cross-modal semantic barriers. Based on this, a predetermined spatiotemporal attention fusion algorithm is constructed. Combining spatiotemporal attributes such as engineering image time-series information and point cloud spatial location, the aligned feature vectors are weighted and fused to achieve deep spatiotemporal and semantic fusion, effectively breaking down data silos and ensuring that the fused features accurately reflect the correlation between multi-source data, thus improving the completeness of decision-making basis.

[0054] This embodiment follows a three-level processing logic of unified encoding, semantic alignment, and spatiotemporal fusion. It is compatible with multiple data formats and avoids information loss, greatly improving the fusion effect. Experiments have verified that the accuracy of multimodal fusion can reach 85% or more, and the accuracy of association and fusion of engineering drawings with on-site images and sensor data exceeds 90%, effectively ensuring the completeness and accuracy of subsequent decision-making basis.

[0055] In some embodiments, determining industry knowledge prompts based on the fusion features using a pre-built knowledge graph and a pre-built industry knowledge base includes: performing semantic extraction and structural transformation on the fusion features to obtain multiple types of semantic information; searching for multiple first association information corresponding to each type of semantic information in the knowledge graph; retrieving multiple second association information from the industry knowledge base using the multiple types of semantic information and the multiple first association information; and determining the industry knowledge prompts based on the multiple types of semantic information, the multiple first association information, and the multiple second association information.

[0056] In this embodiment, semantic extraction and structural transformation are performed on the fused features to obtain various types of semantic information. Specifically, semantic extraction is performed on the fused features output by the cross-modal fusion module to extract corresponding key business entities, equipment status, spatial location, time information, and risk tags, forming a structured decision context. Examples include: equipment defect areas identified in images, construction specification clauses in text, abnormal temperature rise data from sensors, and spatial distance relationships in point clouds. The structured decision context is then uniformly converted into structured semantic tags. Multiple first-level association information corresponding to each type of semantic information is searched in the knowledge graph, enabling rapid matching of entity associations corresponding to the semantics, building a bridge between feature information and industry knowledge, narrowing down the scope of scene-related knowledge, and reducing the subsequent retrieval range. The knowledge graph is responsible for constructing a structured industry knowledge network, and RAG (Retrieval Enhancement Generation) is responsible for dynamically retrieving knowledge related to the current scene from the industry knowledge network.

[0057] By leveraging various types of semantic information and multiple primary association information to search the industry knowledge base, multiple secondary association information is obtained. Combining semantic information with graph-based association clues, matching cases, standards, and practical experience can be accurately selected, enriching the information dimensions and overcoming the limitations of single graph information. This makes the subsequently generated knowledge prompts more comprehensive and aligned with actual business scenarios. Based on multiple types of semantic information, multiple primary association information, and multiple secondary association information, industry knowledge prompts are determined. This allows for the simultaneous integration of on-site data characteristics, entity relationships, industry standards, historical cases, and other multi-source content, avoiding the bias caused by single information sources. The final output knowledge prompts are therefore relevant to the current practical scenario and are accurate and complete.

[0058] The knowledge graph retrieval system matches semantic information with related content. Then, by combining the semantic information with the graph association information, the system further retrieves relevant information from the industry knowledge base. The system integrates these three types of information to initially organize the prompts. Finally, the system applies a preset prompt template to extract effective information and completes logical regularization, ultimately outputting standardized industry knowledge prompts.

[0059] In some embodiments, determining the industry knowledge prompt information based on the multiple types of semantic information, the multiple first association information, and the multiple second association information includes: extracting information from the multiple types of semantic information, the multiple first association information, and the multiple second association information using a pre-built prompt template to obtain initial industry knowledge prompt information; and logically adjusting the initial industry knowledge prompt information to obtain the industry knowledge prompt information.

[0060] In this embodiment, after integrating various semantic information, knowledge graph association information, and industry knowledge base retrieval information, a preset prompt template is first applied to filter and extract effective content to obtain initial industry knowledge prompt information. Then, the content logic is sorted and adjusted to finally form standardized knowledge prompt information. The prompt template helps to organize various scattered data and professional knowledge, output standardized content, and facilitate unified reading and parsing by the decision-making model.

[0061] The system integrates features, retrieved content, knowledge graph association paths, and rule constraints, extracting and organizing them into standardized knowledge prompts according to a fixed template. An example is: "The current equipment exhibits a continuous abnormal temperature rise. According to Article X of the 'Equipment Operation Specification,' a temperature exceeding Y℃ constitutes a Level 2 risk. Based on historical cases A and B, it is recommended to prioritize a cooling system inspection." By using knowledge prompt technology to organize knowledge content and input it into the decision-making model, the system can standardize the model's reasoning logic and effectively avoid conclusions that do not conform to power industry standards.

[0062] In some embodiments, before determining industry knowledge prompts based on the fusion features using a pre-built knowledge graph and a pre-built industry knowledge base, the method includes: performing constraint verification on the fusion features using a pre-built rule engine; and generating warning information in response to determining that the fusion features have failed the constraint verification.

[0063] In this embodiment, before generating industry knowledge prompts based on fusion features combined with knowledge graphs and industry knowledge bases, the fusion features are first constrained and verified by a preset rule engine. If the verification fails, an early warning message is automatically issued. Early verification can identify data anomalies as early as possible and issue risk warnings in a timely manner. It can detect abnormal situations on site at the first time, quickly prompt potential risks, and facilitate early intervention and handling by staff to avoid safety hazards.

[0064] It should be noted that this application introduces a computable rule engine, transforming power industry regulations such as construction safety standards, equipment operating thresholds, risk level assessments, and environmental protection constraints into executable algorithmic rules. It adopts a dual-path collaborative working mode of rule matching and model inference, balancing rule judgment and intelligent analysis, while ensuring the entire decision-making process is traceable. It can generate complete reports annotated with data sources, inference logic, and judgment basis, conforming to industry compliance verification standards and facilitating expert review and problem tracing. The rule engine's operation process consists of three steps: First, pre-verification of rules is conducted before inference, checking and integrating feature data for indicators such as temperature limits, safety distances, and work procedures. If a hard constraint is triggered, an early warning is immediately generated. Second, joint inference of complex scenarios is performed, incorporating the rule verification results as additional conditions into the warning information, and simultaneously sending them to a lightweight model for analysis and judgment. Finally, post-verification of the decision result is completed. If the model's output conclusion contradicts the industry's hard rules, the engine will perform rule correction operations, adjusting the final decision content according to the specifications to ensure the judgment result conforms to business standards.

[0065] In some embodiments, after determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts using a pre-built decision model, the method includes: performing constraint verification on the decision information using a pre-built rule engine; in response to determining that there is information in the decision information that fails the constraint verification, replacing the information that fails the constraint verification using the rule engine to obtain corrected decision information; and optimizing the decision model using the fusion features, the industry knowledge prompts, and the corrected decision information.

[0066] In this embodiment, traditional technologies lack a self-evolving closed loop; model parameters and operating strategies remain fixed. When dealing with differentiated business scenarios, configurations can only be manually modified, resulting in time-consuming adaptation and poor versatility. This application, by leveraging fusion features and industry knowledge hints to derive decision information through a decision model, then calls a rule engine to re-verify the decision results. If the results do not meet the constraints, the rule engine replaces and corrects the relevant content to obtain compliant decision information. The decision model is then iteratively optimized by combining fusion features, knowledge hints, and the corrected results.

[0067] It should be noted that after the model generates decision results, the output can be displayed externally through a visual interactive interface. The system establishes a self-evolving closed loop of "decision-feedback-learning," continuously adjusting model parameters and feature fusion strategies based on actual business feedback data. It also supports visual scene configuration. Experimental verification shows that for new power industry scenarios, only 3 to 5 sets of sample data and business rules need to be uploaded, and the system can automatically generate an adaptation strategy in a short time without repeated development. The scene reusability is improved by more than 80%, and it can flexibly adapt to various decision-making scenarios such as engineering construction and equipment operation and maintenance.

[0068] This application integrates multimodal perception, feature fusion, and intelligent reasoning capabilities, tailored to the business characteristics of the power industry, and establishes a four-in-one technical framework of "multimodal perception - spatiotemporal fusion - intelligent decision-making - feedback optimization." Relying on core technologies such as unified representation, spatiotemporal semantic fusion, and lightweight self-evolving decision-making models, it solves the problems of difficult data fusion, insufficient judgment accuracy, and low processing efficiency in traditional solutions, achieving intelligent upgrades in power scenario decision-making and ensuring autonomous controllability throughout the entire business process.

[0069] In some embodiments, after determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts using a pre-built decision model, the method includes: receiving human feedback information corresponding to the decision information; and optimizing the decision model using the fusion features, the industry knowledge prompts, and the human feedback information.

[0070] In this embodiment, after the decision information is derived by combining the decision model with fused features and industry knowledge prompts, the system receives corresponding human feedback. Then, based on the fused features, industry knowledge prompts, and human feedback data, the model is optimized. The decision results are synchronously transmitted back to the business system, expert review opinions are collected, and the feedback data is sent to the optimization module. Using reinforcement learning and human feedback optimization techniques, the model parameters and feature fusion strategies are continuously adjusted, forming a complete closed loop of collection, fusion, decision-making, feedback, and optimization.

[0071] It should be noted that this application incorporates a computable rule engine and records the entire reasoning process, making the decision-making process traceable and verifiable. The output report fully records the data source, reasoning logic, and judgment basis, complying with power industry compliance standards and facilitating expert review and problem identification. The system dynamically adjusts rule weights based on expert feedback, historical decision accuracy, scenario category, and data quality. Rules with stable judgment effects in operation and maintenance scenarios have their weights increased, while rules with significant deviations from expert conclusions have their confidence levels reduced. Relying on reinforcement learning and human feedback optimization mechanisms, the system iterates rule parameters, driving the rule base to autonomously adapt and update.

[0072] Finally, the decision score is jointly calculated by combining rule confidence and model output probability to enhance the stability of decision-making in complex scenarios. It also supports configuring scenario parameters via a visual interface, eliminating the need for redevelopment when adding new business scenarios. Experimental results show that the iteration cycle is controlled within 2 days. The entire mechanism achieves a multimodal fusion accuracy of no less than 85%, a decision response time of no more than 50 seconds, and a 90% or higher consistency between decision conclusions and expert judgments, effectively improving the accuracy and processing efficiency of intelligent power decision-making.

[0073] In another embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating the intelligent decision-making process in an embodiment of this application. First, a professional knowledge graph of the power industry, including engineering specifications, operation and maintenance standards, and risk cases, is constructed. The graph content is then injected into the decision model through knowledge prompts and retrieval enhancement. This decision model is a lightweight decision model obtained through knowledge distillation. This lightweight model receives the injected knowledge graph content as input and achieves autonomous evolution and iteration by combining reinforcement learning and human feedback reinforcement learning techniques. Simultaneously, a computable rule engine parses power business rules. Using the initial decision information output by the decision model and the parsed power business rules, a hybrid decision-making process combining rule reasoning and data-driven approaches is conducted, and the final decision information is output. In this step, the computable rule engine first parses the power business rules into machine-executable judgment logic, then combines it with the initial decision information output by the decision model. Through the synergistic effect of rule constraint verification and model trend analysis, a hybrid decision-making process combining rule reasoning and data-driven approaches is conducted, ultimately outputting compliant decision information that conforms to industry standards and is based on actual data. After expert review and feedback, the decision-making information is converted into human feedback content. This human feedback content is then fed back into the model optimization process. At the same time, the human feedback content is used to update the professional knowledge graph, forming a complete input-output closed loop process, which continuously improves the professionalism, accuracy, and reliability of power decision-making.

[0074] In another embodiment, such as Figure 4 As shown, Figure 4This is a flowchart illustrating another embodiment of the decision information determination method based on multimodal data in this application. This intelligent decision analysis method operates in a closed-loop process of "collection → fusion → decision → feedback → optimization → loop": First, through the multimodal data collection and preprocessing stage, text, image, table, point cloud, and sensor data are standardized. The processed data is input into the cross-modal unified representation and spatiotemporal fusion module to achieve feature alignment and spatiotemporal dimension fusion of multi-source information. The fused features are then injected into the power professional knowledge graph injection module to supplement industry norms, standards, and other professional information, generating knowledge-enhanced features. The enhanced features and power business rules parsed by the computable rule engine are jointly input into the lightweight self-evolving decision model to conduct rule-driven and data-driven hybrid decision-making and output results. After the decision output and reasoning process are displayed, the model enters the result feedback and model iteration optimization module, where model parameters and fusion strategies are adjusted based on expert feedback. The optimized model and feedback data re-enter the multimodal data collection and preprocessing stage, continuously iterating to achieve autonomous model evolution and improve decision accuracy.

[0075] In another embodiment, this application also establishes an efficiency optimization mechanism to optimize performance for power field decision-making tasks. The optimization method involves using multi-node load distribution to amortize the computational pressure, and combining it with single-node multi-threaded parallel processing to improve computing speed. Experimental verification shows that the response time for moderately complex decision-making tasks is controlled within 50 seconds, and the overall time does not exceed 1 minute, fully meeting the real-time decision-making needs of field personnel.

[0076] This application also adds a security and compliance mechanism: multimodal data and decision results are stored locally with encryption, and the core algorithm is independently developed and does not rely on third-party external platforms. At the same time, the output decision information fully retains the source of multimodal data, reasoning logic, and decision basis corresponding to each stage, supporting full-process traceability and complying with the regulatory and compliance requirements of the power industry.

[0077] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0078] A full-process optimization mechanism for cross-modal intelligent decision-making in the power industry: It achieves deep fusion of multi-source data through cross-modal unified representation and spatiotemporal fusion, enhances the professionalism and accuracy of decision-making by injecting professional knowledge, optimizes decision-making efficiency and adaptability by combining lightweight and self-evolutionary design, and ensures decision compliance and data security through interpretability and security compliance design. It forms an integrated intelligent decision-making closed loop covering data collection, fusion, decision-making, feedback and optimization, and adapts to the diverse professional scenario needs of the power industry.

[0079] It should be noted that some embodiments of this application have been described above. In some cases, the actions or steps described in the above embodiments can be performed in a different order than that shown in the above embodiments and the desired result can still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a decision information determination device based on multimodal data.

[0081] refer to Figure 5 The decision information determination device based on multimodal data includes: The processing module 10 is configured to acquire multimodal data and preprocess each type of modal data to obtain each type of preprocessed modal data.

[0082] The fusion module 20 is configured to perform feature fusion on all preprocessed modal data to obtain fused features.

[0083] The first determining module 30 is configured to determine industry knowledge prompt information based on the fusion features, using a pre-built knowledge graph and a pre-built industry knowledge base.

[0084] The second determining module 40 is configured to determine the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts, using a pre-built decision model.

[0085] The aforementioned apparatus acquires multimodal data, and each modality is preprocessed separately to obtain preprocessed modal data. Dual preprocessing—general and specific—is performed modally to adapt to various data characteristics, effectively removing redundant noise, standardizing data formats, and ensuring data quality across modalities. Feature fusion is then performed on all preprocessed modal data to obtain fused features, effectively capturing complex relationships between cross-modal data, thus obtaining fused features with strong representational capabilities and semantic consistency. Based on these fused features, industry knowledge prompts are determined using a pre-built knowledge graph and a pre-built industry knowledge base, enriching the fused features from a professional perspective and making the industry knowledge prompts more accurate and aligned with actual business needs. Based on the fused features and the industry knowledge prompts, a pre-built decision model is used to determine the decision information corresponding to the multimodal data, significantly improving the accuracy of the decision information.

[0086] In some embodiments, the fusion module 20 is further configured to map all preprocessed modal data to a unified semantic vector space to obtain each semantically aligned feature vector; and to use a predetermined spatiotemporal attention fusion algorithm to perform weighted fusion on each semantically aligned feature vector to obtain the fused feature.

[0087] In some embodiments, the first determining module 30 is further configured to perform semantic extraction and structural transformation on the fused features to obtain multiple types of semantic information; search for multiple first association information corresponding to each type of semantic information in the knowledge graph; use the multiple types of semantic information and the multiple first association information to search in the industry knowledge base to obtain multiple second association information; and determine the industry knowledge prompt information based on the multiple types of semantic information, the multiple first association information and the multiple second association information.

[0088] In some embodiments, the first determining module 30 is further configured to extract information from the multiple types of semantic information, the multiple first association information and the multiple second association information using a pre-built prompt template to obtain initial industry knowledge prompt information; and to logically adjust the initial industry knowledge prompt information to obtain the industry knowledge prompt information.

[0089] In some embodiments, a generation module is further included, which is configured to perform constraint verification on the fusion features using a pre-built knowledge graph and a pre-built industry knowledge base before determining industry knowledge prompt information based on the fusion features; and to generate warning information in response to determining that the fusion features have failed the constraint verification.

[0090] In some embodiments, the system further includes a first optimization module, which is configured to: after determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts using a pre-built decision model; perform constraint verification on the decision information using a pre-built rule engine; in response to determining that there is information in the decision information that fails the constraint verification, replace the information that fails the constraint verification using the rule engine to obtain corrected decision information; and optimize the decision model using the fusion features, the industry knowledge prompts, and the corrected decision information.

[0091] In some embodiments, a second optimization module is further included. The second optimization module is configured to, after determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompt information using a pre-built decision model, receive human feedback information corresponding to the decision information; and optimize the decision model using the fusion features, the industry knowledge prompt information, and the human feedback information.

[0092] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0093] The apparatus of the above embodiments is used to implement the corresponding decision information determination method based on multimodal data in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0094] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the decision information determination method based on multimodal data as described in any of the above embodiments.

[0095] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0096] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0097] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0098] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0099] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0100] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0101] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0102] The electronic devices described above are used to implement the corresponding decision information determination method based on multimodal data in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0103] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the decision information determination method based on multimodal data as described in any of the above embodiments.

[0104] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0105] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the decision information determination method based on multimodal data as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0106] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to execute the decision information determination method based on multimodal data as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0107] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0108] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0109] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0110] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0111] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0112] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0113] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0114] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for determining decision information based on multimodal data, characterized in that, include: Acquire multimodal data and preprocess each type of data separately to obtain each type of preprocessed modal data; Feature fusion is performed on all preprocessed modal data to obtain fused features; Based on the fusion features, industry knowledge prompts are determined using a pre-built knowledge graph and a pre-built industry knowledge base. Based on the fusion features and the industry knowledge prompts, the decision information corresponding to the multimodal data is determined using a pre-built decision model.

2. The method according to claim 1, characterized in that, The feature fusion of all preprocessed modal data to obtain fused features includes: All preprocessed modal data are mapped to a unified semantic vector space to obtain semantically aligned feature vectors for each type. Using a predetermined spatiotemporal attention fusion algorithm, each semantically aligned feature vector is weighted and fused to obtain the fused feature.

3. The method according to claim 1, characterized in that, Based on the fusion features, the industry knowledge prompt information is determined using a pre-built knowledge graph and a pre-built industry knowledge base, including: Semantic extraction and structural transformation are performed on the fused features to obtain various types of semantic information; Search the knowledge graph for multiple first-related information corresponding to each type of semantic information; By utilizing the various types of semantic information and the multiple first association information, a retrieval is performed in the industry knowledge base to obtain multiple second association information; Based on the various types of semantic information, the multiple first association information, and the multiple second association information, the industry knowledge prompt information is determined.

4. The method according to claim 3, characterized in that, The determination of the industry knowledge prompt information based on the multiple types of semantic information, the multiple first association information, and the multiple second association information includes: By using a pre-built prompt template, information is extracted from the various types of semantic information, the multiple first association information and the multiple second association information to obtain initial industry knowledge prompt information; The initial industry knowledge prompts are logically adjusted to obtain the new industry knowledge prompts.

5. The method according to claim 1, characterized in that, Before determining industry knowledge hints based on the fusion features and utilizing a pre-built knowledge graph and a pre-built industry knowledge base, the method includes: The fused features are constrained and validated using a pre-built rule engine. In response to determining that the fusion feature has failed the constraint verification, an early warning message is generated.

6. The method according to claim 1, characterized in that, After determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge hints, using a pre-built decision model, the method includes: The decision information is constrained and validated using a pre-built rule engine. In response to determining that there is information in the decision information that fails the constraint verification, the rule engine is used to replace the information that fails the constraint verification to obtain the corrected decision information; The decision-making model is optimized using the fusion features, the industry knowledge prompts, and the corrected decision information.

7. The method according to claim 1, characterized in that, After determining the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge hints, using a pre-built decision model, the method includes: Receive human feedback information corresponding to the decision information; The decision-making model is optimized using the fusion features, the industry knowledge prompts, and the human feedback.

8. A decision information determination device based on multimodal data, characterized in that, include: The processing module is configured to acquire multimodal data and preprocess each type of modal data separately to obtain each type of preprocessed modal data. The fusion module is configured to perform feature fusion on all preprocessed modal data to obtain fused features; The first determining module is configured to determine industry knowledge prompt information based on the fusion features, using a pre-built knowledge graph and a pre-built industry knowledge base; The second determining module is configured to determine the decision information corresponding to the multimodal data based on the fusion features and the industry knowledge prompts, using a pre-built decision model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method described in any one of claims 1 to 7.

Images