A grain storage question and answer system and method based on atlas and multi-modal fusion

Abstract

The application discloses a kind of based on atlas and multimodal fusion's grain storage question and answer system and method, belong to grain storage intelligent management technical field.System includes input layer, knowledge system module, multimodal fusion module and knowledge enhancement question and answer generation module.Knowledge system module is constituted by static field knowledge atlas, sensor time series database and hierarchical regional event state atlas, realizes grain storage knowledge structured representation and grain condition event hierarchical tracking;Multimodal fusion module is through the double-path design of image coding path and sensor area coding path and cross-modal bidirectional attention mechanism, image information and sensor information are deeply fused, and attention weight is guided by prior bias of cause-effect correlation rule entity coding;Question and answer generation module generates conditional normalization parameter modulation large language model dynamically with joint activation subgraph graph coding, output structured answer and evaluate confidence.The application can effectively improve the situation awareness capability and question and answer interaction efficiency of grain storage management.

Description

Technical Field

[0001] This invention relates to the field of intelligent management technology for grain storage, and in particular to a grain storage graphic question-and-answer system and method based on graph and multimodal fusion. Background Technology

[0002] Grain storage safety is a crucial aspect of ensuring national food security. During the storage process, changes in environmental parameters such as grain pile temperature, humidity, and carbon dioxide concentration directly affect the quality of stored grain, while biological hazards such as pest infestation and mold growth can cause severe grain losses. Traditional grain storage monitoring methods mainly rely on manual inspections and single-point sensor data collection, which have the following shortcomings:

[0003] First, the problem of data silos is serious. Existing grain storage management systems typically store image data, sensor data, disposal records, and other information in different databases, lacking a unified data organization and association mechanism, making it difficult to effectively utilize the semantic relationships between multidimensional data.

[0004] Second, there is a lack of hierarchical modeling for grain situation events. Existing technologies mostly use a flat event recording method, which cannot reflect the hierarchical transmission relationship and causal evolution chain of events under the three-level spatial structure of grain depots, warehouses, and regions. When regional local anomalies escalate into warehouse-level or even grain depot-level risks, the system struggles to perform cross-level situational awareness and root cause tracing.

[0005] Third, the multimodal data fusion methods are limited. Existing systems process images (such as grain surface condition and pest trace images) and sensor time-series data (temperature, humidity, etc.) independently, lacking an effective cross-modal semantic fusion mechanism, and are unable to comprehensively utilize visual information and numerical sensing information for joint analysis.

[0006] Fourth, the question-and-answer interaction capabilities are insufficient. Existing grain storage management systems typically only provide a fixed data query interface, preventing users from asking questions flexibly in natural language, and the system is also unable to dynamically retrieve relevant knowledge and generate structured answers based on the question context. Summary of the Invention

[0007] The technical problem solved by this invention: This invention provides a grain storage graphic question-and-answer system and method based on graph and multimodal fusion, aiming to solve the technical problems in existing grain storage management systems such as serious data silos, lack of hierarchical modeling of grain situation events, single multimodal data fusion methods, and insufficient question-and-answer interaction capabilities.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a grain storage graphic question-and-answer system based on graph and multimodal fusion, the system including an input layer, a knowledge system module, a multimodal fusion module and a knowledge-enhanced question-and-answer generation module;

[0009] The input layer is used to determine the current question-and-answer mode, which includes text-and-image mode, plain text mode, plain image mode, and plain sensor mode.

[0010] The knowledge system module includes a static domain knowledge graph, a sensor time-series database, and a hierarchical regional event state graph submodule. The static domain knowledge graph is stored in a graph database and includes grain entities, pest entities, mold entities, treatment procedure entities, semantic relationships between entities, and causal association rule entities between the surface and interior of the grain pile. The sensor time-series database is used to store the time-series data of all sensors in the entire warehouse. The hierarchical regional event state graph submodule organizes grain condition events in a three-level node structure of grain depot, warehouse, and region. Each event node of the grain condition event is dynamically created and forms a cross-level causal event chain through directed relationships, which is driven by the bottom-up aggregation of sensor data organized according to the structure.

[0011] The multimodal fusion module includes an image encoding path and a sensor region encoding path. The image encoding path switches between real image input, learnable global query vector, and learnable region query vector according to the question-and-answer mode, and outputs an image feature sequence. The sensor region encoding path uses real sensor readings as the basic unit and obtains a sensor region feature sequence through lightweight self-attention aggregation within the region. The multimodal fusion module also performs semantic fusion of image features and sensor region features through a cross-modal bidirectional attention mechanism. The attention weights are guided by the prior bias matrix of the activated surface and internal causal association rule entity encoding, and output a unified grain condition representation vector.

[0012] The knowledge-enhanced question-answering generation module constructs an input sequence using a unified grain situation representation vector and user questions. It dynamically modulates a large language model with conditional normalization parameters generated by graph encoding the joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule. This drives the large language model to generate structured answers, and the generated answers are output after confidence evaluation. The joint activation subgraph includes the currently active event node in the hierarchical regional event state graph and the entities in the static domain knowledge graph that the currently active event node hits.

[0013] Furthermore, in the hierarchical regional event state graph submodule, regional nodes divide each warehouse into multiple regions according to a grid. Each region corresponds to a regional node, carrying a corresponding regional event node. The regional event node includes regional aggregated temperature, regional aggregated humidity, regional carbon dioxide concentration, and regional grain condition status label attributes. Warehouse nodes correspond to each independent grain warehouse, carrying a corresponding warehouse event node, which is obtained by aggregating all regional event nodes under the warehouse. Grain depot nodes correspond to the entire grain storage park, obtained by aggregating all warehouse event nodes under the grain depot. When the grain condition changes, event nodes are automatically created and attached to the corresponding structural organization level node, carrying event type, event level, occurrence timestamp, associated level node, triggering numerical snapshot, and event duration attributes. Event nodes form a cross-level grain condition causal event chain through directed relationships.

[0014] Furthermore, the generated conditional normalization parameters include: performing graph encoding on the joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule, outputting a node embedding matrix, and then performing global attention pooling to obtain a graph-level representation; inputting the graph-level representation into the conditional normalization parameter generation network to generate conditional normalization parameters, which include a conditional scaling vector and a conditional offset vector; the graph encoding is a relational graph convolutional network encoding; the attention weights of the global attention pooling are obtained by normalizing the inner product of the learnable query vector and the embeddings of each node; and the conditional normalization parameter generation network is a multilayer perceptron.

[0015] Furthermore, in the sensor region encoding path, the individual sensor features are formed by linearly projecting the concatenated three parts: reading value features, the relative position encoding of the sensor within its region, and the sensor type encoding. All individual sensor features within the same region are input to the self-attention layer for aggregation, and the output is pooled to obtain a fixed-length feature for that region. The features of all regions constitute a region feature sequence, and each region feature is appended with a spatial coordinate position encoding of the region's center coordinates. The self-attention layer is a single-layer self-attention layer. The pooling is global average pooling. The reading value features are obtained by linearly projecting the sensor reading values ​​after high-dimensional expansion using sine and cosine functions. The relative position encoding is the position encoding after the sensor coordinates are normalized to the coordinate range of its region.

[0016] Furthermore, in the cross-modal bidirectional attention mechanism, image features and sensor features are enhanced by bidirectional cross-attention to obtain image enhancement features and sensor enhancement features, and the image enhancement features and sensor enhancement features are spliced ​​together and linearly projected to obtain a unified grain condition representation vector.

[0017] Furthermore, in image-text mode, the image encoding path takes a real image as input, and the knowledge graph query scope is limited to the corresponding region nodes of the image; in plain text mode, the image encoding path outputs a learnable global query vector, and the knowledge graph query is aggregated to the warehouse node layer or grain depot node layer; in pure image mode, the image encoding path takes a real image as input, the sensor encoding path outputs a zero vector with an additional missing label, and the knowledge graph query only uses the static domain knowledge graph submodule; in pure sensor mode, the image encoding path outputs the learnable region query vector corresponding to the specified region, and the knowledge graph query scope is limited to the specified region nodes.

[0018] Furthermore, the confidence level is jointly determined by the model output confidence benchmark value, the signal consistency factor, and the data integrity factor; the signal consistency factor is calculated based on the bidirectional attention weight matrix after the completion of cross-modal bidirectional attention, and is determined according to the number of attention asymmetries exceeding a preset threshold; the data integrity factor is determined based on the proportion of currently missing sensor data.

[0019] This invention also provides a grain storage image-text question-answering method based on graph and multimodal fusion, characterized in that it is applied to the grain storage image-text question-answering system based on graph and multimodal fusion as described above after training using the training steps, and the method includes the following steps:

[0020] The knowledge system construction steps are as follows: A static domain knowledge graph submodule, a sensor time-series database submodule, and a hierarchical regional event state graph submodule are constructed. The static domain knowledge graph submodule stores grain entities, pest entities, mold entities, treatment procedure entities, and surface and internal causal relationship rule entities. The hierarchical regional event state graph submodule organizes grain condition events using a three-level node structure of grain depots, warehouses, and regions. Each event node of a grain condition event is dynamically created and forms a cross-level causal event chain through directed relationships, driven by bottom-up aggregation of sensor data.

[0021] The hierarchical dynamic update steps are as follows: continuously monitor sensor data, perform regional state updates and event detection, warehouse-level state aggregation, and grain depot-level state aggregation from bottom to top according to the preset collection cycle, activate relevant entities in the static domain knowledge graph sub-module based on newly created event nodes, and perform graph encoding on the joint activation sub-graph of the hierarchical regional event state graph sub-module and the static domain knowledge graph sub-module to generate conditional normalization parameters.

[0022] Multimodal fusion steps: Switching is performed on the input side of the image encoding path according to the current question-answering mode. The sensor region encoding path obtains the region feature sequence by self-attention aggregation within the region, using the actual sensor reading as the basic unit. The image block features and sensor region features are positionally encoded in the same spatial coordinate encoding system. Semantic fusion is performed through a cross-modal bidirectional attention mechanism. The attention weight of the cross-modal bidirectional attention is guided by the prior bias matrix of the activated surface and internal causal association rule entity encoding, and a unified grain condition representation vector is output.

[0023] Question-answer generation steps: Construct the input sequence of the large language model using the unified grain condition representation vector, modulate the layer normalization of the large language model using the conditional normalization parameter, drive the large language model to generate structured answers, and output the generated answers after confidence evaluation.

[0024] Furthermore, in the hierarchical dynamic update step:

[0025] Regional-level status updates and event detection include querying the latest sensor readings for each regional node, performing regional-level aggregation calculations and updating status attributes, creating regional-level event nodes and establishing associations with historical events when threshold conditions are met;

[0026] Warehouse-level status aggregation includes calculating the current status of all regional nodes under each warehouse, updating the comprehensive status attributes of warehouse nodes, creating warehouse-level concurrent early warning event nodes and establishing aggregation triggering relationships with subordinate regional event nodes when warehouse-level event triggering conditions are met;

[0027] Grain depot-level status aggregation includes statistical analysis of the current status of all warehouse nodes, updating the comprehensive attributes of grain depot nodes, and creating grain depot-level event nodes when the conditions for triggering grain depot-level events are met.

[0028] Furthermore, the training steps include:

[0029] Phase 1: The image encoder performs pre-training on the grain storage scene image dataset for surface anomaly classification task, the sensor region encoder performs pre-training on the historical sensor dataset for regional grain condition classification task, and the graph encoder performs pre-training on the grain storage domain knowledge graph triple dataset for entity embedding.

[0030] Second stage: Freeze all pre-trained encoder parameters from the first stage, and train a cross-modal bidirectional attention fusion layer, a conditional normalization parameter generation network, and a learnable query vector using question-answering labeled datasets with multiple question-answering modes. The parameters of the large language model remain frozen throughout the second and third stages, and the training loss function is the cross-entropy loss of the answer sequence generated by the large language model.

[0031] The third stage involves unfreezing the parameters of each encoder pre-trained in the first stage, and then jointly training them with the fusion layer, the conditional normalization parameter generation network, and the learnable query vector, using the cross-entropy loss as the optimization objective, and employing a differentiated learning rate, wherein the learning rate of the encoder is less than the learning rate of the fusion layer and the conditional normalization parameter generation network.

[0032] The beneficial effects of the present invention: Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. A multi-layered, integrated knowledge system has been constructed. This invention achieves structured representation of knowledge in the grain storage field, time-series management of sensor data, and hierarchical tracking of grain condition events by organically combining a static domain knowledge graph, a sensor time-series database, and a hierarchical regional event state graph, effectively solving the data silo problem in existing technologies.

[0034] 2. Dynamic hierarchical modeling of grain situation events has been achieved. This invention, through a three-level node structure of grain depots, warehouses, and regions, a dynamic establishment mechanism for event nodes, and the construction of cross-level causal event chains, enables grain situation anomalies to gradually aggregate from the regional level to the warehouse and grain depot levels, providing structured support for comprehensive perception of grain situation and root cause tracing.

[0035] 3. Deep fusion of image and sensor data is achieved. This invention achieves deep semantic fusion of visual and sensor modalities through a dual-path design of image encoding path and sensor region encoding path, as well as a cross-modal bidirectional attention mechanism. The attention weight is guided by the prior bias of causal association rule entities, which improves the domain specificity and interpretability of the fusion.

[0036] 4. This invention achieves knowledge-enhanced question-answering generation for a large language model. It modulates the large language model by dynamically generating conditional normalization parameters using graph encoding of joint activation subgraphs, enabling the model to adaptively adjust its generation behavior based on the current context. The generated answers possess both structure and domain-specific accuracy.

[0037] 5. Supports flexible question-and-answer modes. This invention adapts to the question-and-answer needs of different application scenarios through a switching mechanism between image and text mode, plain text mode, pure image mode, and pure sensor mode, and provides users with a reliability reference by evaluating the confidence level of the answers. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of a grain storage graphic question-and-answer system based on graph and multimodal fusion provided by the present invention;

[0039] Figure 2This is a schematic diagram of the multimodal fusion module structure in a grain storage graphic question-and-answer system based on graph and multimodal fusion provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the knowledge enhancement question-answering generation module structure in a grain storage graphic question-answering system based on graph and multimodal fusion provided by the present invention;

[0041] Figure 4 This is a flowchart illustrating the generation of conditional normalization parameters in a grain storage graphic question-and-answer system based on graph and multimodal fusion provided by the present invention.

[0042] Figure 5 This is a flowchart illustrating a grain storage graphic question-and-answer method based on graph and multimodal fusion provided by the present invention.

[0043] Figure 6 This is a training process diagram of a grain storage graphic question-and-answer system based on graph and multimodal fusion provided by the present invention. Detailed Implementation

[0044] This invention addresses the technical problems in existing grain storage management systems, such as severe data silos, lack of hierarchical modeling of grain situation events, limited multimodal data fusion methods, and insufficient question-and-answer interaction capabilities. It provides a grain storage graphic and text-based question-and-answer system based on graph and multimodal fusion.

[0045] like Figure 1 As shown, the grain storage graphic question-answering system based on graph and multimodal fusion includes an input layer, a knowledge system module, a multimodal fusion module, and a knowledge-enhanced question-answering generation module.

[0046] The input layer is used to determine the current question-and-answer mode, which includes text-and-image mode, plain text mode, plain image mode, and plain sensor mode.

[0047] The knowledge system module includes a static domain knowledge graph, a sensor time-series database, and a hierarchical regional event state graph submodule. The static domain knowledge graph is stored in a graph database and includes grain entities, pest entities, mold entities, treatment procedure entities, semantic relationships between entities, and causal association rule entities between the surface and interior of the grain pile. The sensor time-series database is used to store the time-series data of all sensors in the entire warehouse. The hierarchical regional event state graph submodule organizes grain condition events in a three-level node structure of grain depot, warehouse, and region. Each event node of the grain condition event is dynamically created and forms a cross-level causal event chain through directed relationships, which is driven by the bottom-up aggregation of sensor data organized according to the structure.

[0048] The multimodal fusion module, such as Figure 2As shown, the system includes an image encoding path and a sensor region encoding path. The image encoding path switches between real image input, a learnable global query vector, and a learnable region query vector according to a question-and-answer mode, and outputs an image feature sequence. The sensor region encoding path uses real sensor readings as the basic unit and obtains a sensor region feature sequence through lightweight self-attention aggregation within the region. The multimodal fusion module also performs semantic fusion of image features and sensor region features through a cross-modal bidirectional attention mechanism. The attention weights are guided by the prior bias matrix of the activated surface and internal causal association rule entity encodings, and output a unified grain condition representation vector.

[0049] The knowledge-enhanced question-answering generation module, such as Figure 3 As shown, an input sequence is constructed using a unified grain condition representation vector and user questions. The conditional normalization parameters dynamically generated by the joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule after graph encoding modulate the large language model, driving the large language model to generate structured answers. The generated answers are then evaluated for confidence and output. The joint activation subgraph includes the currently active event node in the hierarchical regional event state graph and the entities in the static domain knowledge graph that the currently active event node hits.

[0050] For example, grain entities carry attributes such as variety name, upper limit of safe moisture content, upper limit of safe storage temperature, and thermal conductivity; pest entities carry attributes such as suitable breeding temperature range, humidity range, and damage density threshold; mold entities carry attributes such as lower limit of toxin-producing trigger temperature and toxin-producing trigger water activity; treatment procedure entities carry attributes such as trigger condition logical expression, operation step sequence, and expected effect; directed semantic relationships are established between entities, including grain type → applicable storage conditions → environmental threshold, pest → suitable breeding conditions → temperature and humidity range, grain condition risk → trigger treatment → treatment procedure, etc.; causal relationship rule entities between grain pile surface and interior store the domain causal rules between abnormal internal environment and surface visual changes, such as "internal temperature continuously exceeds the limit for more than 48 hours → high probability of condensation on the surface", "carbon dioxide concentration exceeds the limit → gray-white mold spots may appear on the grain pile surface", "excessive humidity in the warehouse → grain grains on the surface clump", etc. These rule entities serve as the knowledge source of prior bias in cross-modal bidirectional attention computation, and are semantic bridges connecting the two paths of feature perception of image surface and sensor internal perception. In pure image mode, the knowledge system module is the only available source of knowledge for the system, which is especially important for making reasonable judgments about grain conditions in the absence of sensor data.

[0051] In some optional implementations, in the hierarchical regional event state graph submodule, regional nodes divide each warehouse into multiple regions according to a grid. Each region corresponds to a regional node, carrying a corresponding regional event node. The regional event node includes regional aggregated temperature, regional aggregated humidity, regional carbon dioxide concentration, and regional grain condition status label attributes. Warehouse nodes correspond to each independent grain warehouse, carrying a corresponding warehouse event node, which is obtained by aggregating all regional event nodes under the warehouse. Grain depot nodes correspond to the entire grain storage park and are obtained by aggregating all warehouse event nodes under the grain depot. When the grain condition changes, event nodes are automatically created and attached to the corresponding structural organization level node, carrying event type, event level, occurrence timestamp, associated level node, triggering numerical snapshot, and event duration attributes. Event nodes form a cross-level grain condition causal event chain through directed relationships.

[0052] For example, a grain depot node (top layer, 1 node): represents the entire grain storage park, carrying aggregated attributes such as the overall grain condition status label (normal / warning / dangerous), the current highest-risk warehouse number, and the total number of active warning events in the entire park. This status is inferred from the aggregation of the status of all its subordinate warehouse nodes from bottom to top, and is updated synchronously after each warehouse node status update. A warehouse node (middle layer, N nodes): each independent grain depot corresponds to one warehouse node, carrying aggregated attributes such as warehouse number, warehouse type, currently stored grain type (foreign key related to submodule A grain type entity), overall grain condition status label, the current number of warning areas, and the highest-risk area number. This status is inferred from the aggregation of the status of all its subordinate regional nodes, and is updated synchronously after each regional node status update. Regional Nodes (bottom layer, 9 per warehouse, number can be modified according to warehouse size and sensor characteristics): Each warehouse is divided into 9 regions (Northeast, Northwest, East, Center, West, Southeast, South, Southwest) using a 3x3 grid. Each region corresponds to one regional node, carrying the region name, a list of coverage sensor numbers, the region's spatial coordinate range (3D coordinates of the region center and boundary range), the current region's aggregated temperature (maximum value of coverage sensor readings), the current region's aggregated humidity (average value of coverage sensor readings), and the current region's... Attributes include concentration (maximum value of the covered sensor readings) and the current grain condition status label for the area. The area division is configured by the administrator during system initialization based on the actual sensor layout and remains fixed thereafter.

[0053] Event nodes (dynamically created): Automatically created and attached to the corresponding level node when the grain condition changes. Event nodes carry: Event type (regional temperature warning / regional...). Warning / Regional insect infestation warning / Warehouse concurrent warning / Disposal operation / Status recovery / Inspection anomaly), event level (regional / warehouse / grain depot level), occurrence timestamp, associated hierarchical nodes, triggering value snapshot, event duration.

[0054] Event nodes form a cross-level grain condition causal event chain through directed relationships. For example, regional-level events are linked to subsequent warehouse-level events within the same warehouse through a "trigger" relationship; early warning events are linked to subsequent handling events through a "response" relationship; handling events are linked to state recovery events through a "cause" relationship; and "historical similarity" relationships are established between event nodes of the same type (similarity is calculated by the cosine similarity of the event feature vectors, and is established when it exceeds the threshold of 0.85), supporting the retrieval of historical similar cases and the reuse of handling experience.

[0055] Specifically, the knowledge system module also includes retrieving pest and mold entities that meet the current environmental conditions from the static domain knowledge graph based on the newly created event node types and grain-related entities. The hit entities are marked as currently active and associated treatment procedure entities and surface and internal causal association rule entities are activated, forming a joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule, providing data basis for the subsequent generation of conditional normalization parameters.

[0056] In some alternative implementations, the generated conditional normalization parameters, such as Figure 4 As shown, the process includes: performing graph encoding on the joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule, outputting a node embedding matrix, and then performing global attention pooling to obtain a graph-level representation; inputting the graph-level representation into a conditional normalization parameter generation network to generate conditional normalization parameters, which include a conditional scaling vector and a conditional offset vector; the graph encoding is a relational graph convolutional network encoding; the attention weights of the global attention pooling are obtained by normalizing the inner product of the learnable query vector and the embeddings of each node; and the conditional normalization parameter generation network is a multilayer perceptron.

[0057] For example, R-GCN computation (3 layers, with independent transformation matrices corresponding to different relation types) is performed on the joint activation subgraph. The initial node features are linearly projected to 512 dimensions after concatenation of static attribute TransE pre-trained embeddings (256-dimensional) and dynamic state numerical encodings (64-dimensional). After outputting the node embedding matrix, global attention pooling is performed to obtain the graph-level representation. The graph-level representation is then input into a conditional normalization parameter generation network (two-layer MLP plus ReLU) to generate conditional normalization parameters, which include a conditional scaling vector and a conditional offset vector.

[0058] Specifically, for the large language model, the original LayerNorm of each Transformer layer is replaced with Condition LayerNorm (CLN) to dynamically adapt to the current grain condition knowledge context. This integrates structured knowledge from the grain storage domain into the generation process of the large language model, preserving the powerful language generation and logical organization capabilities of the large language model itself. It also injects the extracted structured grain condition knowledge and event information as conditions into the generation process, effectively reducing the computational cost of domain adaptation and improving the accuracy of domain knowledge in the generated answers.

[0059] In some optional implementations, in the sensor region encoding path, the individual sensor features are formed by linearly projecting the concatenated three parts: reading value features, the relative position encoding of the sensor within its region, and the sensor type encoding. All individual sensor features within the same region are input to a self-attention layer for aggregation, and the output is pooled to obtain a fixed-length feature for that region. Features from all regions constitute a region feature sequence, with each region feature appended with a spatial coordinate position encoding of the region's center coordinates. The self-attention layer is a single-layer self-attention layer. The pooling is global average pooling. The reading value features are obtained by linearly projecting the sensor reading values ​​after high-dimensional expansion using sine and cosine functions. The relative position encoding is a position encoding obtained by normalizing the sensor coordinates to the coordinate range of its region.

[0060] For example, for each sensor in the entire warehouse, an individual feature vector is constructed, which consists of three parts concatenated and linearly projected to 512 dimensions: reading numerical features (normalized values ​​of temperature, humidity, or carbon dioxide concentration, expanded into a high-dimensional vector using sine and cosine functions, similar to the position encoding of Transformer, and finally linearly projected to 64 dimensions, preserving the continuity of the values), the sensor's relative position encoding within its region (normalizing the sensor coordinates to the region's coordinate range of 0-1, encoding it as a 64-dimensional position vector, reflecting the sensor's relative spatial position within the region), and sensor type encoding (three unique thermal encoding projections of temperature, humidity, or carbon dioxide concentration, 32 dimensions).

[0061] Lightweight self-attention aggregation within a region: The individual feature vectors of all sensors within the same region (a region typically has 3 to 10 sensor measurement points) are grouped into a small set, which is input into a single-layer lightweight self-attention layer (4 heads, 512 dimensions). This allows the sensor features within the region to perceive each other's relative positions and numerical differences (e.g., temperature gradient information between measurement points at different vertical depths within the same region). The output is a fixed-length feature vector of the region after global average pooling.

[0062] The entire portfolio's feature sequence is constructed as follows: each of the nine regions is aggregated to obtain a feature vector, forming a feature sequence of length 9 for the entire portfolio. Each region's feature vector is appended with a three-dimensional spatial coordinate position code of the region's center coordinates (using standard sine and cosine coding methods along...). , , The three coordinate axes are independently encoded and then concatenated (with a coding dimension of 64), enabling the regional features to implicitly carry their positional information in the three-dimensional space of the grain silo. The positional encoding is then concatenated to the regional feature vector and linearly projected back to 512 dimensions. Image features are also appended with three-dimensional spatial coordinate positional encodings calculated based on the camera calibration parameters, corresponding to the spatial coordinates within the silo. Both types of features express positional information within the same three-dimensional spatial coordinate encoding system, allowing the subsequent cross-modal attention mechanism to naturally learn stronger semantic associations between images and sensor regions that are spatially close.

[0063] In some optional implementations, in the cross-modal bidirectional attention mechanism, image features and sensor features are enhanced by bidirectional cross-attention to obtain image enhancement features and sensor enhancement features, and the image enhancement features and sensor enhancement features are spliced ​​together and linearly projected to obtain a unified grain condition representation vector.

[0064] In some optional implementations, in image-text mode, the image encoding path takes a real image as input, and the knowledge graph query scope is limited to the corresponding region nodes of the image; in plain text mode, the image encoding path outputs a learnable global query vector, and the knowledge graph query is aggregated to the warehouse node layer or grain depot node layer; in pure image mode, the image encoding path takes a real image as input, the sensor encoding path outputs a zero vector with an additional missing marker, and the knowledge graph query only uses the static domain knowledge graph submodule; in pure sensor mode, the image encoding path outputs the learnable region query vector corresponding to the specified region, and the knowledge graph query scope is limited to the specified region nodes.

[0065] In some optional implementations, the confidence level is jointly determined by the model output confidence benchmark, the signal consistency factor, and the data integrity factor; the signal consistency factor is determined based on the attention asymmetry of each region calculated from the bidirectional attention weight matrix after the completion of cross-modal bidirectional attention, and is determined according to the number of attention asymmetries exceeding a preset threshold; the data integrity factor is determined based on the proportion of currently missing sensor data.

[0066] For example, the formula for confidence level is: ,in, This is the confidence level baseline. Represents the signal consistency factor. Indicates the data integrity factor. , , This represents the maximum value operation. This represents the number of all regions where the attention asymmetry exceeds the threshold. It is only meaningful in the text and image mode, and is set to 0 in the other three modes. This indicates the proportion of currently missing sensor data. In pure image mode, it is 1. In other modes, it represents the proportion of missing or overdue data from all sensors.

[0067] This invention also provides a grain storage image-text question-answering method based on graph and multimodal fusion, which employs the above-described grain storage image-text question-answering system trained using the training steps. The method, as described above... Figure 5 As shown, it includes the following steps:

[0068] The knowledge system construction steps are as follows: A static domain knowledge graph submodule, a sensor time-series database submodule, and a hierarchical regional event state graph submodule are constructed. The static domain knowledge graph submodule stores grain entities, pest entities, mold entities, treatment procedure entities, and surface and internal causal relationship rule entities. The hierarchical regional event state graph submodule organizes grain condition events using a three-level node structure of grain depots, warehouses, and regions. Each event node of a grain condition event is dynamically created and forms a cross-level causal event chain through directed relationships, driven by bottom-up aggregation of sensor data.

[0069] The hierarchical dynamic update steps are as follows: continuously monitor sensor data, perform regional state updates and event detection, warehouse-level state aggregation, and grain depot-level state aggregation from bottom to top according to the preset collection cycle, activate relevant entities in the static domain knowledge graph sub-module based on newly created event nodes, and perform graph encoding on the joint activation sub-graph of the hierarchical regional event state graph sub-module and the static domain knowledge graph sub-module to generate conditional normalization parameters.

[0070] Multimodal fusion steps: Switching is performed on the input side of the image encoding path according to the current question-answering mode. The sensor region encoding path obtains the region feature sequence by self-attention aggregation within the region, using the actual sensor reading as the basic unit. The image block features and sensor region features are positionally encoded in the same spatial coordinate encoding system. Semantic fusion is performed through a cross-modal bidirectional attention mechanism. The attention weight of the cross-modal bidirectional attention is guided by the prior bias matrix of the activated surface and internal causal association rule entity encoding, and a unified grain condition representation vector is output.

[0071] Question-answer generation steps: Construct the input sequence of the large language model using the unified grain condition representation vector, modulate the layer normalization of the large language model using the conditional normalization parameter, drive the large language model to generate structured answers, and output the generated answers after confidence evaluation.

[0072] In some alternative implementations, during the hierarchical dynamic update step:

[0073] Regional-level status updates and event detection include querying the latest sensor readings for each regional node, performing regional-level aggregation calculations and updating status attributes, creating regional-level event nodes and establishing associations with historical events when threshold conditions are met;

[0074] Warehouse-level status aggregation includes calculating the current status of all regional nodes under each warehouse, updating the comprehensive status attributes of warehouse nodes, creating warehouse-level concurrent early warning event nodes and establishing aggregation triggering relationships with subordinate regional event nodes when warehouse-level event triggering conditions are met;

[0075] Grain depot-level status aggregation includes statistical analysis of the current status of all warehouse nodes, updating the comprehensive attributes of grain depot nodes, and creating grain depot-level event nodes when the conditions for triggering grain depot-level events are met.

[0076] In some alternative implementations, the training steps, such as Figure 6 As shown, it includes:

[0077] Phase 1: The image encoder performs pre-training on the grain storage scene image dataset for surface anomaly classification task, the sensor region encoder performs pre-training on the historical sensor dataset for regional grain condition classification task, and the graph encoder performs pre-training on the grain storage domain knowledge graph triple dataset for entity embedding.

[0078] Second stage: Freeze all pre-trained encoder parameters from the first stage, and train a cross-modal bidirectional attention fusion layer, a conditional normalization parameter generation network, and a learnable query vector using question-answering labeled datasets with multiple question-answering modes. The parameters of the large language model remain frozen throughout the second and third stages, and the training loss function is the cross-entropy loss of the answer sequence generated by the large language model.

[0079] The third stage involves unfreezing the parameters of each encoder pre-trained in the first stage, and then jointly training them with the fusion layer, the conditional normalization parameter generation network, and the learnable query vector, using the cross-entropy loss as the optimization objective, and employing a differentiated learning rate, wherein the learning rate of the encoder is less than the learning rate of the fusion layer and the conditional normalization parameter generation network.

Claims

1. A grain storage graphic question-and-answer system based on graph and multimodal fusion, characterized in that, The system includes an input layer, a knowledge system module, a multimodal fusion module, and a knowledge-enhanced question-answering generation module; The input layer is used to determine the current question-and-answer mode, which includes text-and-image mode, plain text mode, plain image mode, and plain sensor mode. The knowledge system module includes a static domain knowledge graph, a sensor time series database, and a hierarchical regional event state graph sub-module. The static domain knowledge graph is stored in a graph database, including grain entities, pest entities, mold entities, disposal procedure entities, semantic relationships between entities, and causal association rule entities between the surface and interior of the grain pile; the sensor time-series database is used to store time-series data from all sensors in the entire warehouse; the hierarchical regional event state graph submodule organizes grain condition events with a three-level node structure of grain depot, warehouse, and region. Each event node of the grain condition event is dynamically created and forms a cross-level causal event chain through directed relationships. The update is driven by the bottom-up aggregation of sensor data organized according to the structure. When the grain condition status changes, an event node is automatically created and attached to the corresponding level node as the currently active event node. The multimodal fusion module includes an image encoding path and a sensor region encoding path. The image encoding path switches between real image input, learnable global query vector, and learnable region query vector according to the question-and-answer mode, and outputs an image feature sequence. The sensor region encoding path uses real sensor readings as the basic unit and obtains a sensor region feature sequence through lightweight self-attention aggregation within the region. The multimodal fusion module also performs semantic fusion of image features and sensor region features through a cross-modal bidirectional attention mechanism. The attention weight is guided by the prior bias matrix of the activated grain pile surface and internal causal association rule entity encoding, and outputs a unified grain condition representation vector. The grain pile surface and internal causal association rule entities that satisfy the current active event node and are retrieved from the static domain knowledge graph are used as the activated grain pile surface and internal causal association rule entities. The knowledge-enhanced question-answering generation module constructs an input sequence using a unified grain situation representation vector and user questions. It dynamically modulates a large language model with conditional normalization parameters generated by graph encoding the joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule. This drives the large language model to generate structured answers, and the generated answers are output after confidence evaluation. The joint activation subgraph includes the currently active event node in the hierarchical regional event state graph and the entities in the static domain knowledge graph that the currently active event node hits.

2. The grain storage graphic question-and-answer system based on graph and multimodal fusion according to claim 1, characterized in that, In the hierarchical regional event state graph submodule, regional nodes divide each warehouse into multiple regions according to a grid. Each region corresponds to a regional node, carrying a corresponding regional event node. The regional event node includes regional aggregated temperature, regional aggregated humidity, regional carbon dioxide concentration, and regional grain condition status label attributes. Warehouse nodes correspond to each independent grain warehouse, carrying a corresponding warehouse event node. Warehouse event nodes are obtained by aggregating all regional event nodes under the warehouse. Grain depot nodes correspond to the entire grain storage park and are obtained by aggregating all warehouse event nodes under the grain depot. Event nodes are automatically created and attached to the corresponding organizational level node when the grain situation changes. They carry attributes such as event type, event level, occurrence timestamp, associated level node, triggered numerical snapshot, and event duration. Event nodes form a cross-level causal event chain of grain conditions through directed relationships.

3. The grain storage graphic question-and-answer system based on graph and multimodal fusion according to claim 1, characterized in that, The generated conditional normalization parameters include: performing graph encoding on the joint activation subgraph of the hierarchical regional event state graph submodule and the static domain knowledge graph submodule, outputting a node embedding matrix, and then performing global attention pooling to obtain a graph-level representation; inputting the graph-level representation into the conditional normalization parameter generation network to generate conditional normalization parameters, which include a conditional scaling vector and a conditional offset vector; the graph encoding is a relational graph convolutional network encoding; the attention weights of the global attention pooling are obtained by normalizing the inner product of the learnable query vector and the embeddings of each node; and the conditional normalization parameter generation network is a multilayer perceptron.

4. The grain storage graphic question-and-answer system based on graph and multimodal fusion according to claim 1, characterized in that, In the sensor region coding path, the individual sensor features are composed of three parts: reading value features, relative position encoding of the sensor in its region, and sensor type encoding, which are then linearly projected together. All individual sensor features in the same region are input into a lightweight self-attention layer for aggregation, and the output is pooled to obtain the fixed-length features of that region. The features of all regions in the entire warehouse constitute a region feature sequence, and each region feature is accompanied by a spatial coordinate position code of the center coordinate of the region; the lightweight self-attention layer is a single-layer lightweight self-attention layer; the pooling is global average pooling; the reading value features are obtained by linearly projecting the sensor reading values ​​after high-dimensional expansion using sine and cosine functions; the relative position code is a position code after the sensor coordinates are normalized to the coordinate range of the region to which they belong.

5. The grain storage graphic question-and-answer system based on graph and multimodal fusion according to claim 1, characterized in that, In the cross-modal bidirectional attention mechanism, image features and sensor region features are enhanced by bidirectional cross-attention to obtain image enhancement features and sensor enhancement features. The image enhancement features and sensor enhancement features are then spliced ​​together and linearly projected to obtain a unified grain condition representation vector.

6. The grain storage graphic question-and-answer system based on graph and multimodal fusion according to claim 1, characterized in that, In image-text mode, the image encoding path takes real images as input, and the knowledge graph query scope is limited to the corresponding region nodes of the image; in plain text mode, the image encoding path outputs a learnable global query vector, and the knowledge graph query is aggregated to the warehouse node layer or grain depot node layer. In pure image mode, the image encoding path takes the real image as input, the sensor encoding path outputs a zero vector with a missing label, and the knowledge graph query only uses the static domain knowledge graph submodule; in pure sensor mode, the image encoding path outputs the learnable region query vector corresponding to the specified region, and the knowledge graph query range is limited to the specified region nodes.

7. The grain storage graphic question-and-answer system based on graph and multimodal fusion according to claim 1, characterized in that, The confidence level is determined by the model output confidence benchmark value, the signal consistency factor, and the data integrity factor. The signal consistency factor is calculated based on the attention asymmetry of each region after the completion of cross-modal bidirectional attention, and is determined according to the number of attention asymmetries exceeding a preset threshold. The data integrity factor is determined based on the proportion of currently missing sensor data.

8. A grain storage image-text question-and-answer method based on graph and multimodal fusion, characterized in that, The method, applied to the grain storage image-text question-answering system as described in claim 1 after training using the training steps, includes the following steps: The knowledge system construction steps are as follows: A static domain knowledge graph submodule, a sensor time-series database submodule, and a hierarchical regional event state graph submodule are constructed. The static domain knowledge graph submodule stores grain entities, pest entities, mold entities, disposal procedure entities, and causal relationship rule entities between the surface and interior of grain piles. The hierarchical regional event state graph submodule organizes grain condition events using a three-level node structure of grain depots, warehouses, and regions. Each event node of a grain condition event is dynamically created and forms a cross-level causal event chain through directed relationships, driven by bottom-up aggregation of sensor data. The hierarchical dynamic update steps are as follows: continuously monitor sensor data, perform regional state updates and event detection, warehouse-level state aggregation, and grain depot-level state aggregation from bottom to top according to the preset collection cycle, activate relevant entities in the static domain knowledge graph sub-module based on newly created event nodes, and perform graph encoding on the joint activation sub-graph of the hierarchical regional event state graph sub-module and the static domain knowledge graph sub-module to generate conditional normalization parameters. Multimodal fusion steps: Switching is performed on the input side of the image encoding path according to the current question-answering mode. The sensor region encoding path obtains the sensor region feature sequence by using the actual sensor reading as the basic unit through lightweight self-attention aggregation within the region. The image features and sensor region features are positionally encoded in the same spatial coordinate encoding system. Semantic fusion is performed through a cross-modal bidirectional attention mechanism. The attention weight of the cross-modal bidirectional attention is guided by the prior bias matrix of the encoding of the causal association rule entities on the surface and inside the grain pile, and a unified grain condition representation vector is output. Question-answer generation steps: Construct the input sequence of the large language model using the unified grain condition representation vector, modulate the layer normalization of the large language model using the conditional normalization parameter, drive the large language model to generate structured answers, and output the generated answers after confidence evaluation.

9. The grain storage image-text question-and-answer method based on graph and multimodal fusion according to claim 8, characterized in that, In the hierarchical dynamic update step: Regional-level status updates and event detection include querying the latest sensor readings for each regional node, performing regional-level aggregation calculations and updating status attributes, creating regional-level event nodes and establishing associations with historical events when threshold conditions are met; Warehouse-level status aggregation includes calculating the current status of all regional nodes under each warehouse, updating the comprehensive status attributes of warehouse nodes, creating warehouse-level concurrent early warning event nodes and establishing aggregation triggering relationships with subordinate regional event nodes when warehouse-level event triggering conditions are met; Grain depot-level status aggregation includes statistical analysis of the current status of all warehouse nodes, updating the comprehensive attributes of grain depot nodes, and creating grain depot-level event nodes when the conditions for triggering grain depot-level events are met.

10. The grain storage image-text question-and-answer method based on graph and multimodal fusion according to claim 8, characterized in that, The training steps include: Phase 1: The image encoder performs pre-training on the grain storage scene image dataset for surface anomaly classification task, the sensor region encoder performs pre-training on the historical sensor dataset for regional grain condition classification task, and the graph encoder performs pre-training on the grain storage domain knowledge graph triple dataset for entity embedding. Second stage: Freeze all pre-trained encoder parameters from the first stage, and train a cross-modal bidirectional attention fusion layer, a conditional normalization parameter generation network, and a learnable query vector using question-answering labeled datasets with multiple question-answering modes. The parameters of the large language model remain frozen throughout the second and third stages, and the training loss function is the cross-entropy loss of the answer sequence generated by the large language model. The third stage involves unfreezing the parameters of each encoder pre-trained in the first stage, and then jointly training them with the fusion layer, the conditional normalization parameter generation network, and the learnable query vector, using the cross-entropy loss as the optimization objective, and employing a differentiated learning rate, wherein the learning rate of the encoder is less than the learning rate of the fusion layer and the conditional normalization parameter generation network.

Images