AI multi-modal fusion-based recognition accuracy improvement system and method

Abstract

The application relates to the field of artificial intelligence and multi-modal data processing technology, in particular to an AI multi-modal fusion-based recognition accuracy improvement system and method, which comprises the following steps: acquiring node structured data, unstructured environment data and node behavior feedback data of a distributed data network bottom layer node; inputting a multi-modal feature extraction network containing a main task branch and an implicit state branch, extracting main task features, and fusing to generate a trust entropy vector and a trust entropy value; inputting the trust entropy vector into a deep reinforcement learning intelligent agent, dynamically outputting a first weight and a second weight corresponding to an artificial intelligence prediction model and a robust optimization rule model, and controlling the weight relationship of the two types according to a preset threshold; weighting and fusing the preliminary prediction result and the preliminary rule result to output a final recognition result, and continuously updating the behavior feedback data based on the node feedback, so that the comprehensive recognition accuracy of the system under long-term operation is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal data processing technology, specifically to a system and method for improving recognition accuracy based on AI multimodal fusion. Background Technology

[0002] In existing distributed data networks, it is usually necessary to identify the status of the underlying nodes in order to provide a basis for scheduling control, resource allocation or risk warning. In related technologies, the node status can be predicted using single structured business data; or, the reported information and execution status of the nodes can be judged by combining preset rules to output the corresponding identification results.

[0003] However, when the number of nodes exceeds the set network node scale threshold, external environmental disturbances are frequent, and node feedback fluctuates, the former method does not make sufficient use of unstructured environmental information and behavioral changes, which can easily lead to the recognition results being affected by local data distortion. The latter method, although it has a certain degree of stability and interpretability, has a response delay to complex scene changes that exceeds the preset time standard, making it difficult to reflect the true state of the nodes in a timely manner.

[0004] Therefore, node state identification methods in related technologies struggle to balance identification accuracy, result stability, and adaptability to dynamic risks. Summary of the Invention

[0005] The purpose of this invention is to provide a system and method for improving recognition accuracy based on AI multimodal fusion, and to solve the following technical problems:

[0006] To avoid the system from producing abnormal identification outputs that deviate from the actual execution capabilities of nodes when faced with distorted input data or environmental chain propagation disturbances, and to achieve an adaptive balance between short-term high-precision identification and long-term system data flow stability by dynamically adjusting the fusion weights of the data-driven prediction model and the robust optimization rule model through a deep reinforcement learning agent.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] Methods for improving recognition accuracy based on AI multimodal fusion include:

[0009] The multimodal input data of the underlying nodes of a distributed data network based on network graph topology expansion is obtained, and the feature missing rate and time delay features of the multimodal input data are extracted; wherein, the multimodal input data includes node structured data, unstructured environmental data, and node behavior feedback data;

[0010] The multimodal input data is input into a multimodal feature extraction network; wherein the multimodal feature extraction network includes a main task branch and a hidden state branch;

[0011] Extract main task features through the main task branch;

[0012] The multimodal input data is fused through the latent state branches to calculate the output trust entropy vector;

[0013] Combining the data quality obtained from the feature missing rate and time delay feature evaluation based on the multimodal input data, the trust entropy vector is aggregated by weighted summation through the multilayer perceptron dynamically allocating or pre-configuring the weights of each dimension of the trust entropy vector, to obtain the corresponding trust entropy value.

[0014] Determine the relationship between the trust entropy value and the preset trust entropy threshold, wherein the preset trust entropy threshold is determined by the system based on the average trust entropy of nodes under abnormal conditions in historical periods;

[0015] The action space of the deep reinforcement learning agent is constrained based on the relationship, and configured as follows: if the trust entropy value is less than the preset trust entropy threshold, then the action space is limited to a combination of proportions where the value of the first weight is higher than or equal to the value of the second weight; otherwise, the action space is limited to a combination of proportions where the value of the second weight is higher than or equal to the value of the first weight.

[0016] The trust entropy vector is input into the state space of the deep reinforcement learning agent, which then dynamically outputs the first weight and the second weight within the constrained action space. The first weight corresponds to the artificial intelligence prediction model, and the second weight corresponds to the robust optimization rule model.

[0017] By combining the main task features with the artificial intelligence prediction model and the robust optimization rule model respectively, preliminary prediction results and preliminary rule results are output;

[0018] Based on the first weight and the second weight, the preliminary prediction result and the preliminary rule result are weighted and fused, and the fusion result is output to the bottom node as the final recognition result. The node behavior feedback data is continuously updated based on the feedback of the bottom node to the final recognition result.

[0019] As a further aspect of the present invention:

[0020] Node structured data includes node inventory time-series data and node circulation time-series data;

[0021] The unstructured environmental data includes environmental text sentiment data and propagation topology data;

[0022] The node behavior feedback data includes historical instruction execution deviation rate data.

[0023] As a further aspect of the present invention:

[0024] Hidden state branches include graph neural networks;

[0025] By fusing the multimodal input data through the latent state branches, the output trust entropy vector is calculated, including:

[0026] The graph neural network integrates the node structured data, the unstructured environmental data, and the node behavior feedback data.

[0027] Extract the inter-node feature differences in the fused multimodal input data, construct a negative exponential propagation decay function by combining the number of hops in the connected path of the bottom node in the network graph topology, calculate the abnormal propagation impact degree between the bottom nodes based on the propagation decay function of the inter-node feature differences, and extract the difference between the actual execution result and the expected instruction issued by the system in the node behavior feedback data of the multimodal input data as the execution deviation.

[0028] The anomaly propagation impact and the execution deviation are concatenated and combined to generate the trust entropy vector.

[0029] As a further aspect of the present invention:

[0030] The artificial intelligence prediction model is configured as a data-driven prediction model based on a deep neural network.

[0031] The robust optimization rule model is configured as an interpretable tree model with explicit logical branches, which include historical average rules and geographical proximity rules. The historical average rule is configured such that if the actual outbound average of the bottom-level node in the current period is less than a preset proportion of the actual outbound average in the historical periods, the robust optimization rule model outputs a conservative preliminary rule result. The geographical proximity rule is configured such that if there is a backup bottom-level node within a preset geographical radius and the inventory of the backup bottom-level node is higher than the inventory of the bottom-level node, the robust optimization rule model outputs a conservative preliminary rule result.

[0032] As a further aspect of the present invention:

[0033] The deep reinforcement learning agent includes an action space and is configured with a reward function;

[0034] The action space is configured to dynamically adjust the proportional relationship between the first weight and the second weight;

[0035] When updating the policy of the deep reinforcement learning agent online, if the trust entropy value is greater than or equal to the preset trust entropy threshold, the optimization objective of the reward function is switched from minimizing the current prediction error to minimizing the probability of node output anomalies.

[0036] As a further aspect of the present invention:

[0037] After outputting the fusion result as the final recognition result to the underlying node, the process also includes:

[0038] Continuously monitor the current trust entropy value corresponding to the updated node behavior feedback data;

[0039] The relationship between the current trust entropy value and the preset trust entropy threshold is determined as follows: if the current trust entropy value is less than the preset trust entropy threshold, the value of the first weight is increased by a preset step size and the value of the second weight is decreased by a preset step size; otherwise, the current ratio of the values ​​of the first weight and the second weight is maintained.

[0040] An AI-based multimodal fusion-based system for improving recognition accuracy includes:

[0041] The data acquisition module is used to acquire multimodal input data of the underlying nodes of the distributed data network based on the network graph topology expansion, and to extract the feature missing rate and time delay features of the multimodal input data; wherein, the multimodal input data includes node structured data, unstructured environmental data and node behavior feedback data;

[0042] A feature extraction module is used to input the multimodal input data into a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a main task branch and a hidden state branch; the main task features are extracted through the main task branch; the multimodal input data are fused through the hidden state branch to calculate and output a trust entropy vector; and the trust entropy vector is aggregated by weighted summation by combining the data quality evaluated based on the feature missing rate and time delay features of the multimodal input data, through a multilayer perceptron dynamically allocated or pre-configured weights of each dimension of the trust entropy vector, to obtain the corresponding trust entropy value;

[0043] The weight dynamic adjustment module is used to input the trust entropy vector as the state space into the deep reinforcement learning agent, and dynamically output the first weight and the second weight through the deep reinforcement learning agent; wherein, the first weight corresponds to the artificial intelligence prediction model, and the second weight corresponds to the robust optimization rule model.

[0044] An adaptive output module is used to determine the relationship between the trust entropy value and a preset trust entropy threshold, wherein the preset trust entropy threshold is determined by the system based on the average trust entropy of nodes in a historical period without abnormal states; the action space of the deep reinforcement learning agent is constrained based on the relationship, specifically configured as follows: if the trust entropy value is less than the preset trust entropy threshold, the value of the first weight output by the deep reinforcement learning agent is controlled to be higher than or equal to the value of the second weight.

[0045] Otherwise, the value of the second weight output by the deep reinforcement learning agent is higher than or equal to the value of the first weight; the preliminary prediction result and the preliminary rule result are output by combining the main task features with the artificial intelligence prediction model and the robust optimization rule model respectively; the preliminary prediction result and the preliminary rule result are weighted and fused according to the first weight and the second weight to obtain the final recognition result;

[0046] The feedback closed-loop module is used to output the final recognition result to the bottom node and continuously update the node behavior feedback data based on the feedback from the bottom node to the final recognition result.

[0047] The beneficial effects of this invention are:

[0048] 1. This invention effectively compensates for the shortcomings of single structured data in making full use of environmental disturbances and behavioral changes by simultaneously acquiring node structured data, unstructured environmental data and behavioral feedback data, and extracting temporal dependencies using the main task branch. This deep fusion of multi-dimensional information can more comprehensively depict the real state of nodes, significantly reduce the recognition bias caused by local data distortion, and improve the recognition accuracy in complex dynamic scenarios.

[0049] 2. This invention models the correlation between nodes in the graph neural network within the hidden state branch, calculates the impact of anomaly propagation and the amount of execution deviation, and generates a trust entropy vector representing potential risks. This mechanism can accurately identify the chain propagation of environmental disturbances and systematic deviations, rather than isolated local anomalies. By guiding weight allocation with trust entropy values, it actively increases the weight allocation of the robust optimization rule model when the identified risk exceeds a preset threshold, ensuring the stability and interpretability of the identification results under extreme data fluctuation conditions.

[0050] 3. This invention introduces a deep reinforcement learning agent, which dynamically adjusts the fusion weights of the artificial intelligence prediction model and the rule model based on the trust entropy state space. The system can automatically switch optimization objectives according to the risk level: pursuing sensitivity when the risk is low, and shifting to minimizing the probability of output anomalies when the risk is high. This dynamic game mechanism based on real-time state improves the system's adaptive feedback adjustment capability for node state fluctuations, enabling it to flexibly cope with node feedback fluctuations and frequent disturbances in the external environment.

[0051] 4. This invention continuously monitors the execution compliance of nodes through a feedback closed-loop module, and adopts a gradual recovery strategy with a preset step size increasing the first weight based on the decline of trust entropy. This design avoids high-frequency weight jumps between different model outputs due to improved state, ensuring that the system smoothly transitions from a high-risk conservative mode back to a high-precision recognition mode, and effectively balances short-term risk prevention with long-term accuracy limits. Attached Figure Description

[0052] The invention will now be further described with reference to the accompanying drawings.

[0053] Figure 1 This is a flowchart illustrating the method for improving recognition accuracy based on AI multimodal fusion provided in the embodiments of this application;

[0054] Figure 2 A schematic diagram of the module of the AI-based multimodal fusion-based recognition accuracy improvement system provided in the embodiments of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Please see Figure 1 A method for improving recognition accuracy based on AI multimodal fusion includes: acquiring multimodal input data of the underlying nodes of a distributed data network based on network graph topology expansion, and extracting the feature missing rate and time delay features of the multimodal input data; wherein, the multimodal input data includes node structured data, unstructured environmental data, and node behavior feedback data; and inputting the multimodal input data into a multimodal feature extraction network.

[0057] The multimodal feature extraction network includes a main task branch and a hidden state branch; the main task features are extracted through the main task branch; the multimodal input data are fused through the hidden state branch to calculate the output trust entropy vector.

[0058] Combining the data quality assessment based on the feature missing rate and time delay features of the multimodal input data, the trust entropy vector is aggregated by weighted summation through dynamic allocation or pre-configured weights of each dimension using a multilayer perceptron, resulting in the corresponding trust entropy value. The relationship between the trust entropy value and a preset trust entropy threshold is then determined, wherein the preset trust entropy threshold is calculated and determined by the system based on the average trust entropy of nodes under abnormal conditions in historical periods.

[0059] The action space of the deep reinforcement learning agent is constrained based on the relationship, and configured as follows: if the trust entropy value is less than the preset trust entropy threshold, then the action space is limited to a combination of proportions where the value of the first weight is higher than or equal to the value of the second weight; otherwise, the action space is limited to a combination of proportions where the value of the second weight is higher than or equal to the value of the first weight.

[0060] The trust entropy vector is input into the state space of the deep reinforcement learning agent, which then dynamically outputs the first weight and the second weight within the constrained action space. The first weight corresponds to the artificial intelligence prediction model, and the second weight corresponds to the robust optimization rule model.

[0061] By combining the main task features with the artificial intelligence prediction model and the robust optimization rule model respectively, preliminary prediction results and preliminary rule results are output;

[0062] Based on the first weight and the second weight, the preliminary prediction result and the preliminary rule result are weighted and fused, and the fusion result is output to the bottom node as the final recognition result. The node behavior feedback data is continuously updated based on the feedback of the bottom node to the final recognition result.

[0063] This embodiment provides a method for improving recognition accuracy based on AI multimodal fusion. Specifically, taking a cross-regional emergency material dispatch data network as a unified scenario, the network consists of a provincial dispatch center, regional warehousing nodes, transportation nodes, and hospital receiving nodes. The system needs to continuously identify the real supply and demand status, execution reliability, and abnormal risks of each underlying node, so as to generate reliable input data for subsequent dispatch instructions.

[0064] Specifically, the system collects three types of input data from the underlying nodes: structured data reflects the node's own business operation status, such as inventory quantity, inbound and outbound time series, loading completion rate, and transportation delay; unstructured environmental data reflects external disturbances, such as public opinion texts, event descriptions, image summaries, and propagation relationships related to the node; and node behavior feedback data reflects the execution layer's response to the system output, such as whether the execution was on time in past rounds of scheduling, whether the reported values ​​were frequently changed, and whether there were repeated withdrawals of confirmations.

[0065] A simplified application scenario model can be constructed: the structured vector of a certain warehouse node at the current moment is denoted as:

[0066]

[0067] The environment vector is denoted as a vector. It includes the proportion of negative text and the number of propagation nodes; the behavioral feedback vector is denoted as vector. This includes historical execution deviation rate and number of timeout confirmations; the system will , , After alignment within a unified time window, the data is input into a multimodal feature extraction network; this network can be configured as a dual-branch structure.

[0068] The main task branch is used to extract main task features directly related to target recognition, such as demand fluctuation trends, supply gap levels, and instruction execution reachability. The main task branch is configured as a network structure including a long short-term memory network or a converter encoder, used to extract features of temporal dependencies in the multimodal input data. The latent state branch is used to characterize potential risks that are not easily observed directly from a single business indicator.

[0069] The main task branch can encode the above three types of inputs into main task features. These correspond to the urgency of demand, the feasibility of replenishment, and the timeliness risk, respectively; the implicit state branch performs fusion calculations on the multimodal data and outputs a trust entropy vector. ;

[0070] This vector contains multiple elements, which collectively represent the potential risk state of a node through multi-dimensional features, such as setting a trust entropy vector. These can respectively characterize the intensity of environmental disturbance, the volatility of node execution, and the reliability decay of reporting;

[0071] After obtaining the trust entropy vector, the system performs a weighted sum aggregation process on it to obtain the corresponding scalar trust entropy value denoted as te; this aggregation can be performed using a weighted summation method; in this process, the weights of each dimension participating in the weighted summation are dynamically allocated by a multilayer perceptron based on the data quality of the multimodal data, or are pre-configured as hyperparameters of a fixed proportion.

[0072] For example, assigning weights of 0.5, 0.2, and 0.3 to the three dimensions respectively, the current trust entropy value will be... It can be calculated as At this point, the system does not make a single, rigid judgment based directly on the calculated value, but instead uses a multi-dimensional trust entropy vector. The state features are input to the deep reinforcement learning agent, enabling the agent to output the fusion weights of the two models while preserving fine-grained state information; for ease of explanation, if the agent outputs the first weight... Second weight If , it means that the data-driven prediction result is preferred at the current moment; if the output , This indicates that the result with stronger rule constraints will be given priority; in specific control, the system introduces a trust entropy threshold. The preset trust entropy threshold is determined by the system based on the average trust entropy of nodes under abnormal conditions in historical periods, or it can be set as a fixed boundary constant by those skilled in the art based on actual scheduling experience; for example, setting... ;

[0073] When trust entropy value Less than When the current node and its surrounding environment are still under control, the system allows the first weight to be greater than the second weight to improve the sensitivity of a single identification; when the trust entropy value... Greater than or equal to When this occurs, it indicates a risk of data contamination, node execution deviation, or external disturbance propagation. In this case, the second weight is made greater than the first weight to improve the stability and interpretability of the output results. Continuing with the above scenario model, when... At that time, the system can , Adjusted to 0.4 and 0.6;

[0074] The two types of models combine the main task features respectively. Generate preliminary results; among them, the artificial intelligence prediction model can output preliminary prediction results. For example, if the actual available allocation for the next time period is 95 boxes, and the demand level is high; the rule model can output preliminary rule results. For example, based on historical averages and adjacent warehouse linkage rules, the system determines that the safe allocation quantity is 82 boxes, and the demand level is medium to high; the system then... and Perform weighted merging, such as final available allocation. The boxes, after discretization, output 87 boxes, and provide corresponding risk labels;

[0075] After the result is sent back to the underlying nodes, the system continues to collect subsequent responses from the nodes to form new behavioral feedback data, such as whether the node executes according to 87 boxes, whether it submits duplicate declarations, and whether it confirms within the specified time, thereby updating the input for the next round.

[0076] In an exception handling mechanism, if a certain modality is missing in the current cycle, the system does not terminate the processing flow; for example, if the environment text has not yet arrived, the environment code in the previous time window is used as a short-term placeholder, and at the same time, the confidence of the modality is marked as decreasing, and the missing penalty coefficient is increased in the implicit state branch.

[0077] If the structured data contains data that exceeds the preset normal range, such as inventory values ​​being below the zero baseline or the rate of change in turnover exceeding the preset maximum turnover frequency threshold, data cleaning is performed first to truncate the dimension to the preset boundary and input the anomaly label into the trust entropy vector; if the difference between the two preliminary results exceeds the preset difference threshold, such as exceeding 30%, the system triggers a review step and adds a high divergence label when outputting the final identification result so that it can be incorporated into the feedback during subsequent strategy updates;

[0078] During the nighttime peak hours of cross-regional emergency material dispatch, a municipal-level warehouse node experienced negative online sentiment regarding an impending shortage of supplies, causing the negative sentiment value of external environmental data to exceed the preset security assessment threshold. Simultaneously, the node had repeatedly delayed executing allocation instructions recently. Within the same time window, the system received structured and behavioral data showing that the inventory level met the preset allocation threshold but the feedback was slow. The main task branch identified that the node still had allocation capabilities, and the implicit state branch calculated a high trust entropy value.

[0079] At this point, the system automatically reduces the output proportion of the data-driven model and increases the proportion of the rule model, ultimately providing a dispatch identification result that is more constrained by rules and matches the historical execution characteristics of the node. After the node executes according to this result, the subsequent feedback tends to stabilize, the behavioral deviation in the next cycle decreases, and the system gradually restores the higher first weight.

[0080] The purpose of this step is to unify the accuracy of single-time recognition with the stability of long-term data sources into the same processing chain, so that the system can not only recognize the business object itself, but also recognize whether the input data is being distorted, thereby optimizing the recognition performance under long-term operating conditions.

[0081] In a preferred embodiment of the present invention, the node structured data includes node inventory time-series data and node flow time-series data; the unstructured environmental data includes environmental text sentiment data and propagation topology graph data; and the node behavior feedback data includes historical instruction execution deviation rate data.

[0082] This embodiment provides a mechanism for refining the definition of multimodal input data. Specifically, in the aforementioned emergency material dispatch scenario, if node data and external information are collected indiscriminately, inconsistencies in the granularity and time scale of data from different sources can easily occur, leading to unstable trust entropy extracted subsequently. Therefore, this embodiment further defines the data content of each modality, making the input boundaries clearer and the fusion process more reproducible.

[0083] Specifically, the structured data for nodes preferably includes node inventory time-series data and node circulation time-series data; the inventory time-series data can be composed of multiple consecutive time slices, for example, recording the inventory value once per hour for the most recent 6 hours to form a sequence. ; Flow time-series data can correspond to the number of inbound and outbound transactions or loading and unloading throughput in the most recent 6 hours, for example ;

[0084] Both reflect whether there is continuous clearing, backlog, or abnormal fluctuations at the node; if only the instantaneous inventory value of 120 boxes is used, it is impossible to distinguish between the different risk states of a stable inventory of 120 boxes and a sudden change from 300 boxes to 120 boxes in adjacent time slices. Therefore, adding a time-series structure is beneficial for the main task branch to extract the real supply and demand trend.

[0085] Unstructured environmental data preferably includes environmental text sentiment data and propagation topology graph data; environmental text sentiment data can come from accessed information sources, such as announcements, media messages, and emergency group feedback related to the warehouse node, which are semantically processed to obtain sentiment score sequences. The higher the value, the greater the negative intensity; the propagation topology data is used to represent the diffusion relationship of relevant information between nodes;

[0086] For ease of explanation, assume there are four nodes. , , , ,like The information was transmitted to and , Continue to Then a adjacency matrix The first row and second column, and the first row and third column are both 1, the third row and fourth column are both 1, and the remaining positions are 0; through this topology, the system can identify whether local negative information has spread from a single point to a chain propagation;

[0087] Node behavior feedback data preferably includes historical instruction execution deviation rate data; the deviation rate can be obtained by normalizing the difference between the actual execution value and the system suggested value; for example, if a node received allocation suggestions of 100, 90, and 80 in the past three times, and the actual execution values ​​were 95, 60, and 78 respectively, then the corresponding deviation rate can be approximately expressed as follows: Its average value is approximately 0.135;

[0088] This data is used to characterize the degree to which nodes cooperate with the system output; if the historical instruction execution deviation rate of a specific node is higher than the preset deviation threshold for a long period of time, even if its node structured data is within the preset normal range, it indicates that there is a potential mismatch risk between the data input characteristics of the node and the actual instruction execution.

[0089] In a fault-tolerant processing mechanism, if the time interval of inventory time series data is uneven, or if the reporting frequency of each time period is inconsistent, then resampling is performed according to a unified time window.

[0090] If the amount of environmental text is lower than the preset text quantity threshold and is insufficient to form an effective sentiment score, then the time window is marked as a low coverage state and the proportion of text modalities in the fusion is reduced.

[0091] If an isolated node appears in the propagation topology graph, that is, a node has neither incoming nor outgoing edges, the self-loop information of the node is still retained to prevent the node from being regarded as an invalid object when encoding the graph structure; if the historical deviation rate cannot be calculated due to insufficient samples, it is represented by a default value and a missing label to avoid directly filling in zeros and causing completely unreliable misjudgments.

[0092] In the same emergency supplies dispatch network, a hospital's receiving node reported a stable decline in inventory during the day, but its related negative sentiment value continuously increased at night, and the propagation topology showed that the message had spread from a single social group to multiple regional contact groups; at the same time, the node had executed a small number of allocation instructions after receiving them several times in the past; after the system inputs inventory time sequence, circulation time sequence, text sentiment, propagation topology and execution deviation rate, it can more completely distinguish between real shortages and abnormal reports under the interference of external sudden information.

[0093] The purpose of this step is to provide a clear, measurable, and time-aligned input basis for subsequent feature extraction and state recognition, thereby improving the stability of the multimodal fusion results.

[0094] In a preferred embodiment of the present invention, the hidden state branch includes a graph neural network; the trust entropy vector is calculated by fusing the multimodal input data through the hidden state branch, including: fusing the node structured data, the unstructured environmental data, and the node behavior feedback data through the graph neural network;

[0095] The inter-node feature differences in the fused multimodal input data are extracted, and a negative exponential propagation decay function is constructed by combining the hop count of the connected paths of the bottom-level nodes in the network graph topology. The abnormal propagation impact degree between the bottom-level nodes is calculated based on the propagation decay function of the inter-node feature differences. The difference between the actual execution result and the expected instruction issued by the system in the node behavior feedback data of the multimodal input data is extracted as the execution deviation. The abnormal propagation impact degree and the execution deviation are concatenated and combined to generate the trust entropy vector.

[0096] This embodiment provides a mechanism for generating trust entropy vectors using graph neural networks. Specifically, in the aforementioned scenario, if each node is treated in isolation, although the inventory, turnover, and execution status of a single node can be observed, it is difficult to identify whether the anomaly of a certain node is propagated from surrounding nodes. For example, if multiple warehouse nodes deviate at the same time, it may not be due to their independent failures, but rather to the spread of the same negative information chain. Therefore, this embodiment introduces graph neural networks to model the correlation between nodes.

[0097] Specifically, the system first constructs a node graph; each vertex in the graph represents a bottom-level node, and each edge represents the material flow, information flow, or instruction coordination relationship between nodes;

[0098] Each vertex is accompanied by a pre-fusion feature vector, for example, for nodes The inventory time-series coding results, circulation time-series coding results, text sentiment coding results, propagation topology local embedding, and historical execution deviation coding results can be concatenated into a single data structure. ;

[0099] For ease of explanation, three nodes are provided. , , Their initial characteristics are respectively , , In a graph structure, if and Connected and If the features are connected, the graph neural network can obtain updated features after neighborhood aggregation. , , In this way, each node no longer only reflects its own state, but also absorbs the state influence of its neighboring nodes.

[0100] Based on this, the system extracts the feature differences between nodes; these differences can be calculated using the distance or dissimilarity of the updated feature vectors; for example... and The difference can be expressed as , and The difference can be expressed as To demonstrate a reproducible and simplified calculation process, if we take the average of the absolute differences of each dimension, then... ;

[0101] The system inputs this difference into the propagation attenuation function to calculate the impact of abnormal propagation. The attenuation function can reflect the rule that the farther the distance and the longer the link, the weaker the propagation impact. Specifically, the propagation attenuation function is preferably configured as a negative exponential attenuation function, and the variable for calculating its attenuation coefficient is the number of hops in the connected path of the bottom node in the network graph topology. The larger the number of hops, the smaller the corresponding attenuation coefficient.

[0102] For example, the infection rate can be obtained by multiplying the node difference by the link attenuation coefficient: if arrive If the attenuation coefficient is 0.9, then the infectivity is... ;like arrive Need to go through Propagation, total attenuation coefficient is If this is achieved, the corresponding infection rate can be further reduced; in addition to the infection rate, the system also extracts the execution deviation; the execution deviation comes from the difference between the actual execution result and the expected instruction; for example, the system expects... 100 boxes should have been allocated to the hospital node, but only 70 boxes were actually allocated, so the original deviation value is 30 boxes; to facilitate cross-node comparison, it can be divided by the expected value to form a normalized execution deviation of 0.3;

[0103] If the deviation of a node exceeds a preset deviation threshold for several consecutive rounds, it indicates that the node's mismatch is not a single, isolated incident, but rather a persistent execution inconsistency. The trust entropy vector is obtained by concatenating the impact of anomaly propagation and the execution deviation. Continuing the example above, if the neighborhood infection degree of a node can be summarized as follows: If its current execution deviation is 0.3, then the trust entropy vector corresponding to this node can be expressed as: ;

[0104] The vector constructed in this way takes into account both external disturbance propagation and internal execution deviation, and can characterize the potential state more meticulously than a single anomaly score. Compared with the scheme based solely on the deviation rate of a single node, this feature fusion mechanism can make up for the shortcomings of existing technologies in the identification of propagating environmental disturbances.

[0105] If a graph structure is not introduced. High deviations may be mistaken for local equipment failures; by introducing graph neural networks, the system can detect... , , The presence of similar environmental emotional disturbances in adjacency relationships, and the directional nature of deviation diffusion, allows for more accurate identification of systematic execution deviations rather than isolated anomalies.

[0106] In an anomaly handling mechanism, if a node has no adjacent edges in the current time window, the graph neural network can degenerate into self-updating of its own features without forcibly relying on external nodes; if two nodes are connected but the difference in their collection timestamps exceeds the preset allowable range and exceeds the unified time window, then the edge is weakened or blocked in the current round to prevent outdated information from misleading the infection degree calculation.

[0107] If the execution command value is zero, it is not appropriate to normalize by dividing by the expected value. In this case, the absolute deviation amount can be used instead, with a zero baseline mark attached. If the difference between the dimensions of the infection rate and the deviation amount exceeds the preset ratio, interval normalization should be performed first before splicing to avoid a certain part completely dominating the vector.

[0108] In the emergency material dispatch network, a provincial warehouse A, an adjacent transit warehouse B, and a hospital receiving end C form a continuous link; after the frequency of negative information about shortages in the vicinity of warehouse A exceeds the preset threshold, B begins to delay responding to system instructions, while C reports abnormal needs in advance.

[0109] After aggregation through graph neural networks, the system discovered that the anomaly of B was not simply caused by its inventory decline, but was jointly related to the propagation of environmental disturbances on the A side and the feedback amplification on the C side; at this time, the generated trust entropy vector simultaneously contained the impact of the link anomaly propagation and the actual execution deviation, making the subsequent weight adjustment more in line with the real risk;

[0110] The purpose of this step is to encode the propagation effects between nodes into the latent state identification process, thereby enabling early identification of potential sources of distortion and chain disturbances.

[0111] In a preferred embodiment of the present invention, the artificial intelligence prediction model is configured as a data-driven prediction model based on a deep neural network; the robust optimization rule model is configured as an interpretable tree model with explicit logical branches, the logical branches including historical mean rules and geographical proximity rules; wherein, the historical mean rule is configured as follows: if the actual outbound average of the bottom node in the current period is less than a preset proportion of the actual outbound average of the historical period, then the robust optimization rule model is controlled to output a conservative preliminary rule result; the geographical proximity rule is configured as follows: if there is a backup bottom node within a preset geographical radius and the inventory of the backup bottom node is higher than the inventory of the bottom node, then the robust optimization rule model is controlled to output a conservative preliminary rule result.

[0112] This embodiment provides a mechanism for implementing a dual-model output structure. Specifically, in the aforementioned scenario, if only a deep neural network is used for recognition, although it usually has the ability to fit a preset accuracy standard when there is sufficient data and the environment is stable, the output result may deviate from the range that the on-site personnel can understand when the input contains propagation perturbations or abnormal node responses. Conversely, if only fixed rules are used, it is easy to react slowly when the supply and demand structure changes abruptly.

[0113] Therefore, this embodiment sets up a data-driven prediction model and an interpretable tree model respectively, and then dynamically merges them according to the state. Specifically, the data-driven prediction model can be implemented using a deep neural network; its input is the main task features, such as demand urgency, replenishment feasibility, historical fulfillment rate, transportation delay coding, etc., and the output is the target identification result, such as the suggested allocation quantity, risk level, or node status category; assuming the main task features are... The network outputs after multiple mapping layers. The model can learn nonlinear relationships from complex historical samples. For example, when the local inventory meets the preset allocation threshold but upstream transportation delays and negative texts increase simultaneously, the probability of subsequent shortages will increase rapidly.

[0114] Correspondingly, the interpretable tree model is constructed based on the historical mean rule and the geographical proximity rule; the historical mean rule can be expressed as: if the average actual outbound value of the current node in the most recent three periods is less than 80% of the average value of the past thirty periods, then the allocation capacity of the node is downgraded by one level.

[0115] The geographical proximity rule can be expressed as follows: if there are two or more backup nodes within a 30-kilometer radius and their average inventory is higher than that of this node, then the output of this node will be adjusted to the conservative side; for ease of display, if the average outbound volume of a node in the past three periods is 60 boxes and the long-term average is 100 boxes, then the downward adjustment condition is met.

[0116] If the inventory of the two surrounding nodes is higher than that of this node, the tree model further determines that this node should reduce direct allocations and outputs... Since each decision branch is traceable, it has higher logical interpretability. Compared with the previous solution, the addition of dual-model parallel output can compensate for the single-path distortion under extreme conditions.

[0117] Data-driven models excel at discovering patterns in high-dimensional data, but may produce outputs with large ranges; the prediction error of interpretable tree models is limited by the set rule granularity, but can provide identification basis with logical traceability when the trust entropy value is greater than or equal to the preset trust entropy threshold; after the two are calculated in parallel, the final adoption degree is determined by subsequent weight control.

[0118] In a fault-tolerant processing mechanism, if the deep neural network has too low output confidence due to insufficient sample coverage, the system can automatically reduce its availability level and limit the upper limit of the first weight in this round of fusion.

[0119] If the interpretable tree model hits multiple conflicting rules, such as the historical mean rule suggesting increased allocation while the geographical proximity rule suggests reduced allocation, then it can be decided according to the preset priority, or the branch conflict marker can be output for the fusion layer to refer to; if the output units of the two types of models are different, such as one outputting continuous quantities and the other outputting discrete levels, then it should be converted to a unified result space before performing weighted fusion.

[0120] In the event of a sudden surge in demand from hospitals in a certain region, a deep neural network, combining historical fluctuations and transportation data, determined that a certain storage node could still support the allocation of 95 boxes. However, the interpretable tree model found that the actual outbound capacity of this node in the past three periods was significantly lower than the long-term level, and that neighboring warehouses had the ability to substitute, thus giving a conservative identification result of 82 boxes. The system does not pre-determine which is correct, but rather retains both to select later based on trust entropy. The purpose of this step is to simultaneously take into account the high-precision fitting characteristics of the data-driven model and the logical traceability characteristics of the interpretable tree model, thereby achieving adaptive adjustment of the identification strategy under different risk levels.

[0121] In a preferred embodiment of the present invention, the deep reinforcement learning agent includes an action space, and the deep reinforcement learning agent is configured with a reward function; the action space is configured to dynamically adjust the ratio of the first weight and the second weight; when updating the policy of the deep reinforcement learning agent online, if the trust entropy value is greater than or equal to the preset trust entropy threshold, the optimization objective of the reward function is switched from minimizing the current prediction error to minimizing the probability of node output anomalies.

[0122] This embodiment provides an online weight adjustment and target switching mechanism. Specifically, based on the aforementioned dual-model structure, if a fixed fusion ratio is still used, for example, always setting the first weight to 0.7 and the second weight to 0.3, the system cannot change the decision model weight allocation logic in a timely manner with changes in the environment. When the network enters a high-risk state, continuing to pursue the minimization of single errors may cause the output results to deviate from the actual execution capabilities of the nodes, which may induce subsequent feedback to deviate from the expected execution deviation.

[0123] Therefore, this embodiment uses a deep reinforcement learning agent to adjust the weight ratio of the two types of objects online and switches the reward target when there is high risk. The deep reinforcement learning agent is preferably constructed using a deep quality network or a proximal policy optimization architecture, which includes an evaluation network for evaluating the state and an execution network for outputting weight allocation actions.

[0124] Specifically, the agent's state input includes at least a trust entropy vector, and may further incorporate information such as the previous round's fusion error, node response latency, and whether high-disagreement labels have appeared; the action space is configured as a discrete set of weight ratios, for example... It can also be set to output real numbers within a continuous interval;

[0125] For ease of explanation, if the trust entropy vector corresponding to the current state is Furthermore, the aggregated trust entropy value is 0.59, allowing the agent to select from candidate actions. ;

[0126] The reward function adopts different objectives for low-risk and high-risk situations; if the trust entropy value is lower than the threshold, it means that the system can prioritize minimizing the current recognition error, and the reward can be calculated based on the deviation between the final recognition result and the posterior true result.

[0127] For example, if the actual safe allocation quantity is 90 boxes and the system output is 87 boxes, then the error is lower than the preset first error threshold, and the corresponding reward value is greater than the preset reward benchmark; if the trust entropy value is greater than or equal to the threshold, the system will change its objective to minimizing the probability of abnormal node output.

[0128] The abnormal node output here can be specifically manifested as refusal to execute, delayed confirmation, reverse correction of reported values, and an increase in the deviation rate for two consecutive rounds. At this time, even if the prediction error of the current round does not reach the global minimum, as long as the output result can effectively reduce the probability of abnormal response of the underlying node in the future, the reward value corresponding to the action is still judged to meet the optimization criteria.

[0129] To demonstrate this switching logic, a concrete calculation example can be constructed; assume that there are two candidate actions for the same node when it is at high risk: action A corresponds to a weight. The output error in this round is only 3 boxes, but the probability of the node refusing to execute is estimated to be 0.45.

[0130] Weight corresponding to action B The output error in this round is 8 boxes, but the probability of rejection is estimated to be 0.08. In low-risk periods, action A is better; in high-risk periods, action B is better. In this way, online updates no longer only take the minimum error of the current time step as the sole objective, but also take into account the stability of the subsequent data chain.

[0131] Compared to the previous solution, this section adds a policy update mechanism with switchable objectives, which solves the problem of using the same optimization scale in different risk ranges within the same fusion framework. When the system is approaching the unstable boundary, if the reward objective is not switched, the agent may continue to favor outputs that seem more accurate but are harder to implement, amplifying subsequent behavioral biases.

[0132] In an anomaly handling mechanism, if the node posterior feedback has not been obtained in the current round and the probability of an output anomaly cannot be directly assessed, the anomaly statistics of similar historical states can be used as a substitute. If the reward signal arrives late, the reward can be delayed by using time difference update, and it is not required that the loop be closed immediately in each round.

[0133] If multiple actions yield similar rewards, the exploration probability can be retained to avoid getting stuck in a single proportion for a long time; if some features of a state suddenly become missing, the most recent valid state is used to fill in the missing features, while reducing the update step size in this round to prevent over-adjustment when information is incomplete.

[0134] During peak emergency dispatch periods, a certain warehouse node entered a high-risk zone due to the spread of public opinion. After the system detected that its trust entropy value exceeded the threshold, it no longer evaluated the strategy based solely on the difference between the current identification value and the actual value. Instead, it focused more on whether the node would execute the strategy, whether it would continue to report anomalies, and whether it would amplify the deviation.

[0135] The agent therefore chose a proportion that leaned more towards the rule-based model, making the output results more consistent with on-site experience; although the identification in this round was conservative, the nodes executed on time, the subsequent reporting returned to stability, and the overall link quality actually improved.

[0136] The purpose of this step is to enable the weight adjustment to change the optimization direction according to the level of system risk, thereby achieving an adaptive balance between short-term results and long-term data stability.

[0137] In a preferred embodiment of the present invention, after outputting the fusion result as the final recognition result to the underlying node, the method further includes: continuously monitoring the current trust entropy value corresponding to the updated node behavior feedback data; determining the relationship between the current trust entropy value and the preset trust entropy threshold, configured as follows: if the current trust entropy value is less than the preset trust entropy threshold, then incrementing the value of the first weight by a preset step size and decrementing the value of the second weight by a preset step size; otherwise, maintaining the current ratio of the values ​​of the first weight and the second weight.

[0138] This embodiment provides a closed-loop recovery mechanism after output. Specifically, in the aforementioned scheme, the system will actively increase the proportion of the rule model in the high-risk stage to stabilize the node response. However, if the risk has subsided but the conservative proportion is maintained for a long time, it will limit the system's ability to regain higher recognition accuracy. Therefore, after outputting the final recognition result, this embodiment continuously monitors the current trust entropy corresponding to the behavior feedback, and gradually restores the first weight according to the preset step size after confirming that the risk has subsided.

[0139] Specifically, after each round of output, the system receives actual execution feedback from the nodes, including confirmation delay, execution volume, number of subsequent corrections, and secondary reporting status, and recalculates the current trust entropy value based on these updated behavioral feedback data.

[0140] The current state here does not require the exact same timestamp as the previous round. It only needs to form a new state evaluation within the preset monitoring period. For example, after the output of the previous round, if a node actually executes as suggested, does not withdraw, and the confirmation time is shortened, then the new execution deviation decreases, and the current trust entropy value may drop from 0.59 to 0.48.

[0141] When the current trust entropy value is less than the threshold, the system does not immediately restore the weights to high-precision mode, but adjusts them smoothly according to a preset step size; for example, if the weights in the previous round were... With a preset step size of 0.1, the new round can be adjusted to... If the trust entropy continues to fall below the threshold in the next round, it will be further adjusted to... ;

[0142] This gradual recovery avoids the subsequent abnormal response of nodes caused by quickly switching back to high-sensitivity mode when the trust entropy index drops below the threshold once. If the current trust entropy value is still higher than or equal to the threshold, the existing ratio is maintained, indicating that the system continues to observe and does not directly change the decision model weight allocation logic.

[0143] Compared to the previous solution, this one adds recovery control after the crisis is resolved; without this mechanism, the system will continue to maintain the high weight allocation of the robust optimization rule model under a high trust entropy state, thereby limiting the system's limit recognition accuracy; or it may directly cause discontinuous parameter mutations based on a single state feedback, resulting in high-frequency oscillations in the output results between different models; this embodiment forms a stable recovery path by using preset step size increments and decrements.

[0144] In a fault-tolerant processing mechanism, if the current trust entropy value fluctuates repeatedly around the threshold, for example, 0.54, 0.56, and 0.53 for three consecutive rounds, while the threshold is 0.55, the hysteresis interval can be increased. For example, the weight is restored only when it is below 0.52, and maintained or re-conservative when it is above 0.58, in order to avoid frequent switching.

[0145] If the first weight plus the preset step size exceeds the upper limit of 1, then the value is set to 1, and the second weight is set to 0; if node feedback is missing and the current trust entropy cannot be recalculated, then the current ratio is maintained and the pending confirmation status is recorded.

[0146] If multiple nodes share a single upstream strategy layer, each node can calculate its step size adjustment separately, and then the upper layer will uniformly constrain the allocation of resources across the entire network. In the third round of scheduling after a provincial transit warehouse entered a high-risk phase, the system had previously reduced the weight to [a specific value]. As nodes gradually resumed executing instructions and the spread of negative text weakened, the updated trust entropy remained below the threshold for two consecutive rounds.

[0147] The system first adjusts the weights to Adjust to After observing no abnormalities at the nodes, the weights were restored to the proportions primarily driven by the data-driven model. The entire weight adjustment process was smooth, avoiding abrupt changes in the control parameters.

[0148] The purpose of this step is to restore high-precision identification capabilities in an orderly manner after the risk subsides, thereby achieving a balance between system stability and the upper limit of accuracy.

[0149] Please see Figure 2 A system for improving recognition accuracy based on AI multimodal fusion includes: a data acquisition module, used to acquire multimodal input data of the underlying nodes of a distributed data network based on network graph topology expansion, and extract the feature missing rate and time delay features of the multimodal input data; wherein, the multimodal input data includes node structured data, unstructured environmental data, and node behavior feedback data;

[0150] A feature extraction module is used to input the multimodal input data into a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a main task branch and a hidden state branch; the main task features are extracted through the main task branch; the multimodal input data are fused through the hidden state branch to calculate and output a trust entropy vector; and the trust entropy vector is aggregated by weighted summation by combining the data quality evaluated based on the feature missing rate and time delay features of the multimodal input data, through a multilayer perceptron dynamically allocated or pre-configured weights of each dimension of the trust entropy vector, to obtain the corresponding trust entropy value;

[0151] The weight dynamic adjustment module is used to input the trust entropy vector as the state space into the deep reinforcement learning agent, and dynamically output the first weight and the second weight through the deep reinforcement learning agent; wherein, the first weight corresponds to the artificial intelligence prediction model, and the second weight corresponds to the robust optimization rule model.

[0152] An adaptive output module is used to determine the relationship between the trust entropy value and a preset trust entropy threshold, wherein the preset trust entropy threshold is determined by the system based on the average trust entropy of nodes in a historical period without abnormal states; the action space of the deep reinforcement learning agent is constrained based on the relationship, specifically configured as follows: if the trust entropy value is less than the preset trust entropy threshold, the value of the first weight output by the deep reinforcement learning agent is controlled to be higher than or equal to the value of the second weight.

[0153] Otherwise, the value of the second weight output by the deep reinforcement learning agent is higher than or equal to the value of the first weight; preliminary prediction results and preliminary rule results are output by combining the main task features with the artificial intelligence prediction model and the robust optimization rule model, respectively;

[0154] Based on the first weight and the second weight, the preliminary prediction result and the preliminary rule result are weighted and fused to obtain the final recognition result; the feedback closed-loop module is used to output the final recognition result to the bottom node, and continuously update the node behavior feedback data based on the feedback of the bottom node to the final recognition result.

[0155] This embodiment provides a system structure for implementing the aforementioned method. Specifically, the system can be deployed in the server cluster of an emergency material dispatch center, or it can be functionally divided into edge acquisition terminals, regional computing nodes, and central strategy nodes for collaborative implementation. The modules can exchange data through a message bus or shared memory, as long as the processing order of time-series acquisition, status calculation, dynamic weight allocation, and feedback can be met.

[0156] Specifically, the data acquisition module is responsible for continuously receiving multimodal input data from the underlying nodes; this module may include a structured interface subunit, an unstructured parsing subunit, and a behavioral feedback receiving subunit; the structured interface subunit receives time-series data such as inventory, circulation, and transportation; the unstructured parsing subunit receives text, topological relationships, or summary results; and the behavioral feedback receiving subunit receives execution confirmations, deviation values, and withdrawal records.

[0157] To ensure consistency in subsequent processing, this module can be equipped with a time-aligned cache to encapsulate data from the same node within the same time window into a unified record;

[0158] The feature extraction module corresponds to the multimodal feature extraction network; internally, it can be divided into main task feature extraction unit and latent state extraction unit.

[0159] The former outputs the main task features, which can be used by both types of recognition models; the latter preferably integrates a graph structure processing component to generate a trust entropy vector from multi-node relationships and further aggregate it into a trust entropy value; for easy reproduction, the output of this module can be represented as a dual-path result: one path is the main task features. The other path is state features. and its aggregate value ;

[0160] The weight dynamic adjustment module receives state features Then, the deep reinforcement learning agent is invoked to generate the first and second weights; this module may further include a state cache unit, a policy inference unit, and an online update unit; the state cache unit stores the trust entropy vector and node feedback from the most recent rounds; the policy inference unit outputs the weight ratio according to the current state; the online update unit corrects the policy parameters after obtaining the posterior result; in this way, the system can not only infer the weights in the current round, but also continuously improve and adjust the rules as it runs;

[0161] The adaptive output module is used to perform threshold determination, call the prediction model, and fuse the recognition results; it can first compare the aggregated values. Compared with the preset trust entropy threshold Then, based on the comparison results, constrain the relationship between the first weight and the second weight;

[0162] When the model is invoked, this module will use the main task features respectively. The data is fed into the data-driven model and the rule-based model to obtain preliminary prediction results. and preliminary rule results According to the first weight Second weight Perform weighted fusion and output the final recognition result. If necessary, the currently used mode identifier can also be output synchronously, such as high-precision mode, conservative mode or recovery mode, to facilitate auditing by the upper-level system.

[0163] The feedback loop module is responsible for sending the final identification results to the underlying nodes and continuously collecting the execution results; this module may include a result publishing unit, a feedback monitoring unit, and a feedback storage unit.

[0164] The results publishing unit is responsible for issuing identification results or related scheduling suggestions; the feedback monitoring unit receives node confirmations, execution differences, and subsequent supplementary reports; the feedback storage unit writes these feedbacks into the behavior database for use in the next round of collection and state calculation; with the help of this closed loop, the system can directly transform the execution compliance of the output results into part of the input for the next round;

[0165] In a fault-tolerant processing mechanism, if a module temporarily fails, the system can enter a degradation process; for example, when the unstructured parsing sub-unit fails, the data acquisition module can still continue to collect structured and behavioral feedback data, and mark the environment modality as low confidence in the current round.

[0166] If the latent state extraction unit in the feature extraction module times out, the valid trust entropy vector from the previous round can be used for a one-time replacement, but the magnitude of weight changes is limited. If the dynamic weight allocation module does not return a valid ratio, the adaptive output module can call a preset backup ratio, for example... ;

[0167] If the feedback loop module does not receive a response from the node for a short period of time, it will not immediately determine failure, but will continue to listen within a preset waiting window, and write an exception tag after the timeout; in the cross-regional emergency material dispatch platform, the data acquisition module collects inventory, public opinion and execution information from warehousing nodes, transportation nodes and hospital nodes;

[0168] The feature extraction module identifies that there is an overlap of environmental disturbance diffusion and execution deviation in a certain link; the weight dynamic adjustment module reduces the first weight and increases the second weight accordingly; the adaptive output module outputs a more conservative identification result and sends it down; the feedback closed-loop module detects that the node execution has returned to normal and writes the feedback back to the database for the next round to gradually restore high-precision output;

[0169] The purpose of this system is to organize multimodal acquisition, latent state identification, dynamic weight allocation, adaptive fusion, and closed-loop feedback into a unified engineering implementation architecture, thereby enabling the stable deployment of the aforementioned methods in real-world distributed networks.

[0170] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for improving recognition accuracy based on AI multimodal fusion, characterized in that, include: The multimodal input data of the underlying nodes of a distributed data network based on network graph topology expansion is obtained, and the feature missing rate and time delay features of the multimodal input data are extracted; wherein, the multimodal input data includes node structured data, unstructured environmental data, and node behavior feedback data; The multimodal input data is input into a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a main task branch and a hidden state branch; main task features are extracted through the main task branch; the multimodal input data are fused through the hidden state branch to calculate and output a trust entropy vector; Combining the data quality obtained based on the feature missing rate and the time delay feature assessment, the trust entropy vector is aggregated by weighted summation through a multilayer perceptron dynamically allocating or pre-configuring the weights of each dimension of the trust entropy vector, to obtain the corresponding trust entropy value. Determine the relationship between the trust entropy value and the preset trust entropy threshold, wherein the preset trust entropy threshold is determined by the system based on the average trust entropy of nodes under abnormal conditions in historical periods; The action space of the deep reinforcement learning agent based on the relation constraints is configured as follows: if the trust entropy value is less than the preset trust entropy threshold, then the action space is limited to a combination of proportions where the values ​​of the first weight are higher than or equal to the values ​​of the second weight; otherwise, the action space is limited to a combination of proportions where the values ​​of the second weight are higher than or equal to the values ​​of the first weight. The trust entropy vector is input into the state space of the deep reinforcement learning agent, which then dynamically outputs the first weight and the second weight within the constrained action space. The first weight corresponds to the artificial intelligence prediction model, and the second weight corresponds to the robust optimization rule model. By combining the main task features with the artificial intelligence prediction model and the robust optimization rule model respectively, preliminary prediction results and preliminary rule results are output; Based on the first weight and the second weight, the preliminary prediction result and the preliminary rule result are weighted and fused, and the fusion result is output to the bottom node as the final recognition result. The node behavior feedback data is continuously updated based on the feedback of the bottom node to the final recognition result.

2. The method for improving recognition accuracy based on AI multimodal fusion according to claim 1, characterized in that, The node structured data includes node inventory time-series data and node circulation time-series data; The unstructured environmental data includes environmental text sentiment data and propagation topology data; The node behavior feedback data includes historical instruction execution deviation rate data.

3. The method for improving recognition accuracy based on AI multimodal fusion according to claim 1, characterized in that, The hidden state branches include graph neural networks; By fusing the multimodal input data through the latent state branches, the output trust entropy vector is calculated, including: The graph neural network integrates the node structured data, the unstructured environmental data, and the node behavior feedback data. Extract the inter-node feature differences in the fused multimodal input data, construct a negative exponential propagation decay function by combining the number of hops in the connected path of the bottom node in the network graph topology, calculate the abnormal propagation impact degree between the bottom nodes based on the propagation decay function of the inter-node feature differences, and extract the difference between the actual execution result and the expected instruction issued by the system in the node behavior feedback data of the multimodal input data as the execution deviation. The anomaly propagation impact and the execution deviation are concatenated and combined to generate the trust entropy vector.

4. The method for improving recognition accuracy based on AI multimodal fusion according to claim 1, characterized in that, The artificial intelligence prediction model is configured as a data-driven prediction model based on a deep neural network. The robust optimization rule model is configured as an interpretable tree model with explicit logical branches, which include historical average rules and geographical proximity rules. The historical average rule is configured such that if the actual outbound average of the bottom node in the current period is less than a preset proportion of the actual outbound average in the historical period, the robust optimization rule model is controlled to output a conservative preliminary rule result. The geographical proximity rule is configured such that if there is a backup bottom-level node within a preset geographical radius and the inventory of the backup bottom-level node is higher than the inventory of the bottom-level node, the robust optimization rule model is controlled to output a preliminary rule result that is biased towards conservatism.

5. The method for improving recognition accuracy based on AI multimodal fusion according to claim 1, characterized in that, The deep reinforcement learning agent includes an action space and is configured with a reward function. The action space is configured to dynamically adjust the proportional relationship between the first weight and the second weight; When updating the policy of the deep reinforcement learning agent online, if the trust entropy value is greater than or equal to the preset trust entropy threshold, the optimization objective of the reward function is switched from minimizing the current prediction error to minimizing the probability of node output anomalies.

6. The method for improving recognition accuracy based on AI multimodal fusion according to claim 1, characterized in that, After outputting the fusion result as the final recognition result to the underlying node, the process also includes: Continuously monitor the current trust entropy value corresponding to the updated node behavior feedback data; The relationship between the current trust entropy value and the preset trust entropy threshold is determined as follows: if the current trust entropy value is less than the preset trust entropy threshold, the value of the first weight is increased by a preset step size and the value of the second weight is decreased by a preset step size; otherwise, the current ratio of the values ​​of the first weight and the second weight is maintained.

7. A recognition accuracy improvement system based on AI multimodal fusion, characterized in that, include: The data acquisition module is used to acquire multimodal input data of the underlying nodes of the distributed data network based on the network graph topology expansion, and to extract the feature missing rate and time delay features of the multimodal input data; wherein, the multimodal input data includes node structured data, unstructured environmental data and node behavior feedback data; A feature extraction module is used to input the multimodal input data into a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a main task branch and a hidden state branch; and extracts main task features through the main task branch; By fusing the multimodal input data through the latent state branches, the trust entropy vector is calculated and output. Combined with the data quality evaluated based on the feature missing rate and the time delay feature, the trust entropy vector is aggregated by weighted summation through the multilayer perceptron dynamically allocating or pre-configuring the weights of each dimension of the trust entropy vector to obtain the corresponding trust entropy value. The weight dynamic adjustment module is used to input the trust entropy vector as the state space into the deep reinforcement learning agent, and dynamically output the first weight and the second weight through the deep reinforcement learning agent; wherein, the first weight corresponds to the artificial intelligence prediction model, and the second weight corresponds to the robust optimization rule model. An adaptive output module is used to determine the relationship between the trust entropy value and a preset trust entropy threshold, wherein the preset trust entropy threshold is determined by the system based on the average trust entropy of nodes under abnormal conditions in historical periods. The action space of the deep reinforcement learning agent based on the relation constraints is specifically configured as follows: if the trust entropy value is less than the preset trust entropy threshold, then the action space is limited to a combination of proportions in which the values ​​of the first weight are higher than or equal to the values ​​of the second weight. Otherwise, the action space is limited to a ratio of the second weight values ​​being higher than or equal to the first weight values; preliminary prediction results and preliminary rule results are output by combining the main task features with the artificial intelligence prediction model and the robust optimization rule model, respectively; the preliminary prediction results and the preliminary rule results are weighted and fused according to the first weight and the second weight to obtain the final recognition result; The feedback closed-loop module is used to output the final recognition result to the bottom node and continuously update the node behavior feedback data based on the feedback from the bottom node to the final recognition result.

Images