Method and system for edge-end multi-mode perception and decision collaboration

Abstract

The invention discloses an edge end multi-modal perception and decision cooperation method and system, and relates to the edge calculation and artificial intelligence cross technology field, and the system comprises a multi-modal perception unit used for collecting heterogeneous perception data of an environment; the data processing unit is in communication connection with the multi-mode sensing unit; wherein the data processing unit comprises a dynamic cognitive kernel, and the dynamic cognitive kernel sequentially comprises a modal credibility evaluation layer, a context reasoning layer and a decision verification layer; and the modal credibility evaluation layer is configured to receive the heterogeneous sensing data. According to the edge-end multi-modal sensing and decision-making collaboration method and system, by introducing a dynamic cognitive kernel, real-time evaluation and dynamic weight adjustment of the reliability of multi-modal sensing data are achieved, the defect that the collaboration efficiency of a traditional fixed rule system is reduced when the environment changes is effectively overcome, and the collaborative performance of the system is improved. And the adaptability and decision accuracy of the system in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to the field of edge computing and artificial intelligence, specifically to a method and system for multimodal perception and decision-making collaboration at the edge. Background Technology

[0002] Currently, multimodal perception and decision-making systems deployed in edge computing environments are widely used. These systems typically rely on pre-defined fixed rules or trained static models to achieve coordination and decision-making among multiple perception modalities. However, the real physical environment is highly dynamic and uncertain; for example, sudden changes in lighting, partial sensor occlusion, or weather effects can cause real-time fluctuations in the reliability of perception across different modalities. Existing methods cannot dynamically assess the confidence level of each modality, still relying on pre-set fusion weights or simple conflict voting strategies, leading to a sharp decline in the overall accuracy and robustness of decisions when the environment changes. Simultaneously, the inherent resource constraints of edge terminals require the system to have the ability to autonomously allocate computing power, while traditional static architectures struggle to adapt to real-time situations, resulting in low resource utilization or delays in critical task decisions. Therefore, existing systems lack a mechanism capable of real-time reasoning, dynamic coordination of multimodal perception, and intelligent resource allocation, severely limiting their application effectiveness in complex scenarios. This invention aims to address the problems of rigid coordination mechanisms and insufficient decision reliability in edge multimodal perception systems caused by dynamic environmental changes. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for multimodal perception and decision-making collaboration at the edge, so as to solve the problems mentioned in the background art.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a method and system for edge-end multimodal perception and decision-making collaboration, comprising: A multimodal sensing unit for collecting heterogeneous sensing data of the environment; The data processing unit is communicatively connected to the multimodal sensing unit; The data processing unit includes a dynamic cognitive kernel, which in turn includes a modal credibility assessment layer, a contextual reasoning layer, and a decision verification layer. The modal reliability assessment layer is configured to receive the heterogeneous sensing data and, based on the internal state of each modal data stream and the intermodal relationship, output a health factor characterizing the real-time reliability of each modality. The context reasoning layer is communicatively connected to the modal credibility evaluation layer and stores a scene knowledge graph. It is configured to receive the health factors and environmental parameters and output a dynamic fusion strategy through reasoning. The decision verification layer is communicatively connected to the context reasoning layer and is configured to verify the preliminary decision output by the context reasoning layer and output the final decision instruction after the verification is successful. A decision execution unit that is communicatively connected to the data processing unit and is used to execute the final decision instruction.

[0005] Preferably, the modal credibility assessment layer includes several self-supervised anomaly detection models, each corresponding one-to-one with a perceived modality; each self-supervised anomaly detection model is configured to measure the internal consistency of the modal data by analyzing the temporal stability of its corresponding modal data.

[0006] Preferably, the modal credibility assessment layer is further configured to measure the expected consistency across modalities by analyzing the degree of mutual support between different modal perception data in space or semantics; the health factor is calculated based on the internal consistency and the expected consistency.

[0007] Preferably, the scene knowledge graph constructs mapping rules between environmental context information, modal health status, and fusion decision strategies; the context reasoning layer is configured to substitute the current health factors and environmental parameters into the mapping rules for reasoning, generating a modal fusion weight matrix and conflict resolution strategy suitable for the current scene.

[0008] Preferably, the decision verification layer includes a lightweight adversarial verification network and a historical decision database; The lightweight adversarial verification network is configured to perform stress testing on the preliminary decision and output a first verification result. The historical decision database stores historical successful decision cases and is configured to provide comparison data to the decision verification layer to compare the preliminary decision with the historical successful decision pattern and output a second verification result.

[0009] Preferably, the decision verification layer is configured as follows: when the preliminary decision, the first verification result, and the second verification result are consistent, the preliminary decision is confirmed as the final decision instruction; when the three are inconsistent, the context reasoning layer is triggered to start a re-reasoning process, and the current case data is recorded in the historical decision database to optimize the scene knowledge graph.

[0010] A method for edge-end multimodal perception and decision-making collaboration, applied to the aforementioned edge-end multimodal perception and decision-making collaboration system, includes the following steps: S1: Collect heterogeneous sensing data of the environment through a multimodal sensing unit; S2: The heterogeneous sensing data is processed through the modal credibility assessment layer of the dynamic cognitive kernel to output real-time health factors for each modality; S3: Through the contextual reasoning layer of the dynamic cognitive kernel, the health factors and real-time environmental parameters are combined to perform reasoning, generate a dynamic fusion strategy and output a preliminary decision; S4: The preliminary decision is verified through the decision verification layer of the dynamic cognitive kernel, and the final decision instruction is output after the verification is passed. S5: Execute the final decision instruction through the decision execution unit.

[0011] Preferably, the step of outputting real-time health factors for each modality includes: The internal consistency of the data stream of each modality is analyzed using a self-supervised anomaly detection model corresponding to each modality. Analyze the expected consistency among sensing data from different modalities; The health factor is calculated by combining the analysis results of the internal consistency and the expected consistency.

[0012] Preferably, the step of generating the dynamic fusion strategy includes: Access the scene knowledge graph that stores mapping rules; The health factors and environmental parameters are used as inputs, and inference is performed using the mapping rules. Outputs a dynamic weight matrix and conflict resolution instructions to guide the current multimodal data fusion.

[0013] Preferably, the steps for multi-level consistency verification of the preliminary decision include: The initial decision is input into a lightweight adversarial verification network for stress testing to obtain the first verification result; The preliminary decision is pattern-matched with historical successful decision-making cases to obtain a second verification result; Compare the preliminary decision, the first verification result, and the second verification result; Based on the comparison results, a final decision instruction will be output or a re-reasoning process will be triggered.

[0014] This invention provides a method and system for multimodal perception and decision-making collaboration at the edge. It has the following beneficial effects: This method and system for edge-end multimodal perception and decision collaboration, by introducing a dynamic cognitive kernel, enables real-time assessment of the reliability of multimodal perception data and dynamic weight adjustment. It effectively overcomes the shortcomings of traditional fixed-rule systems where collaboration efficiency declines when the environment changes, and improves the system's adaptability and decision accuracy in complex scenarios.

[0015] This edge-end multimodal perception and decision-making collaboration method and system, through a multi-level decision verification mechanism and closed-loop optimization design, enables the system to perform multiple verifications on decisions before output, ensuring high reliability of decisions. At the same time, it continuously optimizes the knowledge base using historical cases, enhancing the system's long-term learning ability and operational stability, and avoiding system risks caused by a single decision error. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the module interaction of a multimodal perception and decision-making collaboration system at the edge of the present invention; Figure 2 This is a flowchart illustrating a method for edge-side multimodal perception and decision-making collaboration according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 and Figure 2 This invention provides a technical solution: a method and system for multimodal perception and decision-making collaboration at the edge, comprising: A multimodal sensing unit for collecting heterogeneous sensing data of the environment; A data processing unit that communicates with the multimodal sensing unit; The data processing unit includes a dynamic cognitive kernel, which in turn includes a modal credibility assessment layer, a contextual reasoning layer, and a decision verification layer. The modal reliability assessment layer is configured to receive heterogeneous sensing data and output health factors characterizing the real-time reliability of each modality based on the internal state of each modal data stream and the intermodal relationships. The context reasoning layer communicates with the modal credibility assessment layer and stores a scene knowledge graph. It is configured to receive health factors and environmental parameters and output a dynamic fusion strategy through reasoning. The decision verification layer communicates with the context reasoning layer and is configured to verify the preliminary decision output by the context reasoning layer, and output the final decision instruction after the verification is successful. It is a decision execution unit that communicates with the data processing unit to execute the final decision instructions.

[0019] It should be further explained that, in the specific implementation process, the system first collects raw sensing data of the surrounding environment synchronously or asynchronously through different sensors in the multimodal sensing unit, such as cameras, LiDAR, millimeter-wave radar, microphone arrays, or temperature and humidity sensors. After preliminary signal conditioning and format unification, this heterogeneous data is transmitted to the dynamic cognitive kernel in the data processing unit for core processing.

[0020] The operation of the dynamic cognitive kernel begins with the modal reliability assessment layer. This layer runs a lightweight, self-supervised anomaly detection model in parallel for each modal data stream. For the visual modality, the model evaluates its internal consistency by analyzing the sharpness, contrast, or feature point matching rate of consecutive frames; for the acoustic modality, it evaluates it by analyzing the fluctuations in background noise level and signal-to-noise ratio. Simultaneously, this layer performs cross-modal expected consistency analysis. For example, when the visual sensor detects a moving object, the system expects the radar sensor to respond at the corresponding spatial coordinates; if the radar does not respond or the response coordinates are significantly off, cross-modal consistency is considered reduced.

[0021] Based on the analysis results of internal consistency and expected consistency, this layer calculates and outputs a dynamically changing, quantified health factor for each modality, which directly reflects the reliability of the modality under the current environmental conditions.

[0022] Subsequently, the health factors and current environmental parameters are jointly input into the context inference layer. The scene knowledge graph built into this layer stores rules defined by domain knowledge, such as "IF ambient light intensity is below the threshold AND visual health factor decreases THEN Increase the fusion weight of LiDAR modalities" or "IF rainy weather AND LiDAR health factor moderately decreases THEN Enable anti-rain and fog filtering algorithm and prioritize the use of millimeter-wave radar data for dominant decision-making"; where the current environmental parameters include the ambient light intensity obtained through the ambient light sensor and the weather conditions obtained through the weather interface.

[0023] The inference engine matches and reasons about the input based on these rules, ultimately generating a dynamic fusion strategy applicable to the specific scenario. This strategy includes the weight matrix of each modality in data fusion and conflict resolution instructions to be followed when the perception results of different modalities contradict each other. Based on this strategy, multimodal data is fused and calculated to generate a preliminary decision result.

[0024] The initial decision is not implemented immediately, but is instead sent to the decision verification layer for multi-level consistency verification. The verification layer first calls a lightweight adversarial verification network, which stress-tests the initial decision by injecting simulated perceptual noise or simulated extreme scenarios, observes whether the conclusion remains stable, and outputs the first verification result; among them, extreme scenarios include extreme occlusion and strong interference.

[0025] Meanwhile, the verification layer accesses the historical decision database, performs pattern matching and similarity calculation between the features of the current preliminary decision and the historical successful decision cases recorded in the database, and outputs a second verification result. The system compares the preliminary decision, the first verification result, and the second verification result: if the three are consistent, the preliminary decision is confirmed as the final decision instruction; if there is a discrepancy, such as adversarial testing revealing a fragile decision or low historical pattern matching, the verification layer will trigger a signal to require the context reasoning layer to start a new reasoning process, and simultaneously record the current environmental parameters, perceived data, and decision result as a new case in the historical database for subsequent iterative optimization of the scene knowledge graph.

[0026] Finally, the final decision command, having passed all verifications, is sent to the decision execution unit, which may be a vehicle control unit, a robot motion controller, or an industrial alarm trigger, responsible for translating digital commands into concrete physical actions. This closed-loop process enables the system to adapt to dynamic environmental changes, intelligently coordinate multimodal sensing resources, and ensure the reliability of decisions.

[0027] The modal credibility assessment layer comprises several self-supervised anomaly detection models, each corresponding one-to-one with a perceived modality. Each model is configured to measure the internal consistency of its corresponding modality data by analyzing the temporal stability of the data. It should be further noted that, in practice, the modal credibility assessment layer incorporates several self-supervised anomaly detection models to achieve its functionality. These models are pre-established and maintain a one-to-one correspondence with each perceived modality the system needs to process. The core operating mechanism of each self-supervised anomaly detection model serving its corresponding modality lies in continuously and in real-time analyzing the temporal characteristics of the incoming modality data stream, using this as the fundamental basis for evaluating the internal consistency of that modality.

[0028] For the visual modality, the corresponding model analyzes the smoothness of the trajectory of specific feature points and the fluctuations in overall brightness or color distribution between consecutive image frames. If the image sequence exhibits abnormal jitter, sudden changes in brightness, or loss of a large number of feature points, the model determines that its temporal stability has decreased. For the acoustic modality, the model focuses on analyzing the variation of the energy spectrum of the audio signal within a short time window, or the continuity of energy distribution in a specific frequency band. The radar modality model monitors the rate of change of point cloud density and the stability of echo intensity.

[0029] This measurement of temporal stability does not rely on predefined absolute thresholds, but rather on the inherent patterns of the data stream for that modality learned by the model during system initialization or stable operation. The model generates a quantitative evaluation value regarding internal consistency by comparing the degree of deviation between real-time data and the learned patterns. This evaluation value is a key input for subsequent calculations of the comprehensive health factor, aiming to dynamically capture data quality degradation in any modality caused by environmental interference or changes in sensor condition, thereby providing crucial raw judgments reflecting the real-time reliability of each modality for subsequent fusion inference.

[0030] The modal reliability assessment layer is also configured to measure the expected consistency across modalities by analyzing the degree of mutual support between different modal sensing data in space or semantics; the health factor is calculated based on both internal consistency and expected consistency. It should be further explained that, in the specific implementation process, in addition to analyzing the internal consistency of each modality, the modal reliability assessment layer also undertakes the task of evaluating the mutual support and corroboration relationships between different modal sensing data, i.e., the expected consistency across modalities. This system synchronizes data from different sensors with coordinate systems and timestamps through predefined spatiotemporal alignment rules, establishing a foundation for cross-modal analysis. For example, when a camera in the visual modality identifies the outline of an object and provides its two-dimensional image coordinates, the system projects these coordinates into a ray in three-dimensional space based on a pre-calibrated extrinsic parameter matrix, and expects the lidar or millimeter-wave radar modality to detect the corresponding point cloud or reflective point cloud clusters within a specific distance range along the ray direction.

[0031] If radar data successfully detects a target within the expected spatial area, and its reported velocity and size largely match the visual assessment, then the cross-modal expectation consistency is considered high. Conversely, if radar data for that area is missing, or if the reported target attributes conflict inexplicably with the visual assessment, then the expectation consistency is considered low. For semantic-level expectations, if the audio modality's microphone array captures a sharp sound, while the visual modality does not detect a significant event that could produce such a sound at the same time, such as an object collision, the system will also record this semantic expectation inconsistency.

[0032] The built-in logic processing unit in the evaluation layer integrates the temporal stability analysis results within each modality with the aforementioned cross-modal consistency analysis results. Using rule-based or weighted fusion algorithms, it calculates a health factor that more comprehensively and accurately reflects the reliability of the modality in the current integrated environment. This health factor no longer merely reflects the operating state of a single sensor but also implies its reliability within the multi-sensor collaborative system, providing richer and more reliable input for the subsequent inference layer to formulate highly adaptive fusion strategies.

[0033] The scene knowledge graph constructs mapping rules between environmental context information, modal health status, and fusion decision strategies. The context reasoning layer is configured to substitute the current health factors and environmental parameters into the mapping rules for reasoning, generating a modal fusion weight matrix and conflict resolution strategy suitable for the current scenario. It should be further clarified that, in specific implementation, the scene knowledge graph is not a simple database, but a network storing mapping rules abstracted from expert experience and historical data. These rules define the complex causal relationships and logical connections between environmental context information, modal health status, and the final fusion decision strategy. The construction of this graph is based on the analysis of numerous typical scenarios. For example, in the field of autonomous driving, one rule might be expressed as "When the environmental context information is nighttime and the weather conditions are rain and fog, if the health factor of the LiDAR modality remains at a high level while the health factor of the visual modality decreases, then fusion strategy A is triggered"; another rule might be "When the system is in a congested urban road scenario and the acoustic modal health factor shows reliability, if an emergency vehicle siren is detected, then regardless of the immediate results of the visual and radar modalities, strategy B is prioritized for avoidance."

[0034] During runtime, the contextual reasoning layer receives a real-time set of health factors from the evaluation layer and environmental parameters obtained through external interfaces. This real-time data is then used as input facts and matched against rule conditions stored in the knowledge graph. The reasoning process employs a rule-based reasoning engine, which traverses the rules in the graph, searching for entries whose conditional parts match the current input facts. Upon successful matching, the dynamic fusion strategy indicated by the rule's conclusion is activated. This strategy is manifested as a weight matrix guiding multimodal data fusion calculations; for example, explicitly specifying a 20% increase in radar data weight and a 40% decrease in visual data weight under the current conditions; and a series of conflict resolution instructions, such as prioritizing radar classification when visual and radar classifications of the same target are inconsistent. In this way, the system no longer statically applies a fusion algorithm but can dynamically and adaptively select and generate the most suitable collaborative perception and decision-making scheme for the rapidly changing operating conditions based on a deep understanding of the environment.

[0035] The decision verification layer includes a lightweight adversarial verification network and a historical decision database; The lightweight adversarial verification network is configured to perform stress testing on the initial decision and output the first verification result. The historical decision database stores historical successful decision-making cases and is configured to provide comparison data to the decision verification layer to compare the preliminary decision with the historical successful decision-making pattern and output a second verification result.

[0036] It should be further explained that, in the specific implementation process, the decision verification layer integrates two core components to achieve its multi-level verification function: a lightweight adversarial verification network and a historical decision database. The lightweight adversarial verification network is a specially trained, efficient neural network model. Its design goal is not to perform complex perception tasks, but rather to focus on robustness stress testing of the initial decision results output by the context inference layer. This network operates by actively injecting simulated, controllable interference patterns into the perceptual features or data upon which the initial decision is based. These interference patterns simulate typical anomalies that may occur in the edge environment, such as simulating data loss due to partial sensor occlusion, simulating the impact of specific weather conditions on sensing accuracy, or simulating random noise caused by electromagnetic interference; among these, specific weather conditions include rain and fog.

[0037] The network observes whether the initial decision conclusions will undergo disruptive changes after being subjected to these injected disturbances, and outputs a first verification result regarding the robustness of the decision. The historical decision database is a structured repository that continuously records decision cases generated during past system operations that have been subsequently verified as successful, along with their corresponding key contextual features. These key contextual features include the environmental state, health factors for each modality, and the final adopted strategy. After obtaining the initial decision, the decision verification layer accesses this database in real time, matching the features of the current decision with historical cases and calculating similarity. If the current decision is found to be highly similar in pattern to historical successful decisions, a positive second verification result is output; if the current decision exhibits an abnormal or rare pattern, a verification result requiring caution is output. The collaborative work of these two components forms the core foundation of the multi-level verification capability of the decision verification layer, providing dual assurance for the reliability of the final decision instruction.

[0038] The decision verification layer is configured as follows: when the preliminary decision, the first verification result, and the second verification result are consistent, the preliminary decision is confirmed as the final decision instruction; when the three are inconsistent, the context reasoning layer is triggered to start a re-reasoning process, and the current case data is recorded in the historical decision database to optimize the scenario knowledge graph. It should be further explained that, in the specific implementation process, the logic processing unit built into the decision verification layer is responsible for comprehensively evaluating the received preliminary decision, the first verification result from the lightweight adversarial verification network, and the second verification result from the historical decision database. This unit compares the three inputs to determine whether their conclusions point to the same decision direction or compatible decision results. If the three are consistent, the logic processing unit confirms that the preliminary decision has high reliability and outputs it as the final decision instruction, which is then sent to the decision execution unit. If the three are inconsistent, for example, the preliminary decision suggests executing action A, but adversarial testing shows that the decision becomes unstable under interference, or historical records show that successful decisions in similar situations are mostly action B, then the logic processing unit determines that the current decision has a high risk. At this point, the unit will not directly output an unreliable decision, but will generate a trigger signal, which will be sent to the context inference layer to instruct it to start a new inference process.

[0039] In the re-inference process, the contextual reasoning layer may use adjusted parameters, such as assigning a higher distrust weight to the health factor with a lower modality, or trying to call a backup, more conservative fusion rule in the knowledge graph to generate a new preliminary decision for re-verification.

[0040] Simultaneously, the logic processing unit packages all contextual data that currently causes inconsistencies—including environmental parameters, raw perception data from each modality, health factors, preliminary decision results, and verification results—as a special case and transmits it to the historical decision database for storage. This case is marked as a case to be optimized. When background resources are idle or during periodic tasks, the system uses these cases to incrementally learn and optimize the scenario knowledge graph. For example, it adjusts the confidence level of specific rules or adds new rule entries, thereby enabling the entire system to learn from decision disagreements and continuously improve its collaborative and decision-making capabilities.

[0041] A method for edge-end multimodal perception and decision-making collaboration, applied to an edge-end multimodal perception and decision-making collaboration system, includes the following steps: S1: Collect heterogeneous sensing data of the environment through a multimodal sensing unit; S2: The heterogeneous sensing data is processed through the modal credibility assessment layer of the dynamic cognitive kernel to output real-time health factors for each modality; S3: Through the contextual reasoning layer of the dynamic cognitive kernel, the health factors and real-time environmental parameters are combined to perform reasoning, generate a dynamic fusion strategy and output a preliminary decision; S4: The preliminary decision is verified through the decision verification layer of the dynamic cognitive kernel, and the final decision instruction is output after the verification is passed. S5: Execute the final decision instruction through the decision execution unit.

[0042] It should be further explained that, in the specific implementation process, this method begins by synchronously or asynchronously collecting raw sensing data of the surrounding environment through various sensors in the multimodal sensing unit deployed on edge devices. This data can include multiple modalities such as images, point clouds, audio, and temperature. After necessary preprocessing, the collected heterogeneous data is sent to the core processing stage of the system, namely the dynamic cognitive kernel. The processing flow of this kernel is first executed by the modality reliability assessment layer. This layer calls the self-supervised anomaly detection model built into the system, corresponding to each modality, to perform real-time analysis on the continuously flowing modal data streams. By evaluating its own internal characteristics such as temporal stability and the degree of mutual support between different modal data, it comprehensively calculates and outputs a set of health factors that can quantitatively reflect the reliability of each modality under the current environmental conditions.

[0043] These health factors, along with real-time environmental parameters obtained through other interfaces, are then fed into the context inference layer. This layer accesses its internally stored scene knowledge graph, a rule base containing mapping relationships between environmental context, device status, and optimization decision strategies. The inference engine takes the current health factors and environmental parameters as input facts, matches them with the rule conditions in the knowledge graph, generates a dynamic fusion strategy adapted to the current scene through logical reasoning, and applies this strategy to perform fusion calculations on the original perceived data, ultimately producing a preliminary decision result. The fusion strategy includes weight allocation and conflict resolution rules.

[0044] This preliminary decision is not the final result; it must be submitted to the decision verification layer for multi-level consistency verification. This verification layer calls a lightweight adversarial verification network to stress test the decision, while querying the historical decision database for pattern comparison, and comparing the test results and comparison results with the preliminary decision itself. Only when the three conclusions are consistent is the preliminary decision confirmed as the final and reliable decision instruction. If inconsistency occurs, a re-reasoning mechanism is triggered and the system is optimized.

[0045] Ultimately, the verified decision instructions are sent to the decision execution unit and transformed into specific control commands or operational actions, thus completing a full closed loop from perception to collaborative decision-making to execution.

[0046] The steps for outputting real-time health factors for each modality include: The internal consistency of the data stream of each modality is analyzed using a self-supervised anomaly detection model corresponding to each modality. Analyze the expected consistency among sensing data from different modalities; The health factor was calculated by combining the analysis results of internal consistency and expected consistency.

[0047] It should be further explained that, in the specific implementation process, this step is executed by calling the self-supervised anomaly detection model deployed in the system, which has a pre-established correspondence with each sensing modality. For each input sensing data stream, the corresponding model first analyzes the inherent regularity of the data in the time series. For example, for video stream data, the model calculates the smoothness of the motion trajectory of specific image feature points or the fluctuation of the overall brightness distribution between consecutive frames, thereby assessing its internal consistency; for acoustic stream data, it analyzes the energy change pattern of its short-time spectrum or the continuity of signals in specific frequency bands.

[0048] At the same time, the system will analyze the expected consistency between different modal perception data in parallel. This requires mapping data from different sources to a unified reference frame through spatiotemporal alignment, and then judging the mutual support or contradiction between them. For example, when the visual modality identifies an object in a specific spatial region, the system will check whether the lidar modality detects the corresponding point cloud clusters near the corresponding coordinates. If the match is successful, the expected consistency is high. If the lidar does not provide feedback in the region or the feedback attributes are significantly different, the consistency is reduced.

[0049] Ultimately, the system's built-in fusion logic will integrate the internal consistency analysis results for the same modality with the expected consistency analysis results for cross-modal data involving that modality. This integration process can be carried out according to preset rules or weighting strategies. Finally, it will calculate and output a health factor value that can comprehensively reflect the real-time reliability of the modality under the current integration conditions. This value will serve as a key input for subsequent knowledge reasoning.

[0050] The steps to generate a dynamic fusion strategy include: Access the scene knowledge graph that stores mapping rules; Using health factors and environmental parameters as inputs, inference is performed through mapping rules; Outputs a dynamic weight matrix and conflict resolution instructions to guide the current multimodal data fusion.

[0051] It should be further explained that, in the specific implementation process, this step begins with the system accessing its internally stored scene knowledge graph. This graph is a structured rule base in which the mapping relationships between environmental context parameters, the health states of each modality, and specific fusion decision strategies are constructed. These mapping relationships exist in the form of condition-conclusion rules. The system combines the set of real-time health factors output by the modality credibility assessment layer with the real-time environmental parameters obtained through external sensors or interfaces to form the input fact set of the current state.

[0052] The inference engine then matches these input facts with numerous rule conditions stored in the knowledge graph, searching for rule entries whose conditions match the current input facts. The matching process involves logical judgments, such as determining whether the current ambient light intensity is below a certain threshold, and whether the visual modality's health factor is indeed in a declining range. Once a suitable rule is matched, the system activates the specific fusion strategy indicated in the rule's conclusion.

[0053] This strategy is specifically manifested in two operable outputs: First, there is a dynamic modal fusion weight matrix, which clearly defines the proportion of each modal data in the fusion algorithm under the current specific working conditions. For example, in rainy or foggy weather and when the lidar performance is stable, its data weight is increased accordingly. Second, there are conflict resolution instructions. This is a set of pre-defined logical commands that guide the system to take action when irreconcilable contradictions arise in the perception results of different modalities. For example, when the system is inconsistent between visual and radar classifications of objects, it can prioritize the radar classification result or trigger lower-level feature fusion for re-determination.

[0054] Through this series of reasoning actions based on real-time knowledge and context awareness, the system generates a fusion strategy that is highly adapted to the current dynamic environment, thereby achieving a fundamental transformation from rigid fixed rules to intelligent adaptive collaboration.

[0055] The steps for multi-level consistency verification of preliminary decisions include: The initial decision was input into a lightweight adversarial verification network for stress testing to obtain the first verification result; The initial decision was pattern-matched with historical successful decision-making cases to obtain a second validation result; Compare the preliminary decision, the first verification result, and the second verification result; Based on the comparison results, a final decision instruction will be output or a re-reasoning process will be triggered.

[0056] It should be further explained that, in the specific implementation process, this step begins by inputting the preliminary decision results generated by the context reasoning layer into a lightweight adversarial verification network. This network is pre-trained to simulate various abnormal or extreme conditions that may occur in edge environments. It performs stress testing by injecting specific and controllable disturbance patterns into the feature space or data source on which the decision is based. For example, it simulates data loss caused by instantaneous sensor occlusion or the typical impact of specific weather conditions on the quality of perceived data. It then observes whether the preliminary decision conclusion remains stable in the face of these disturbances, thereby obtaining the first verification result on the robustness of the decision.

[0057] Simultaneously, the system accesses a historical decision database, which stores a large number of successful decision cases that have been verified after the fact and their corresponding key context feature vectors. The system uses a pattern matching algorithm to calculate and compare the similarity between the features of the current preliminary decision and the historical cases. If a highly similar historical successful pattern is found, a supporting second verification result is obtained. If the current decision exhibits an abnormal pattern or differs significantly from the historical successful pattern, a warning second verification result is obtained.

[0058] Subsequently, the system's built-in logic unit performs a consistency comparison on the preliminary decision itself, the first verification result, and the second verification result to determine whether they point to the same final action direction. If the comparison result shows consistency among the three, the logic unit confirms that the preliminary decision is reliable and outputs it as the final decision instruction; if any inconsistency exists among the three, the logic unit determines that the current decision-making process has uncertainty or risk, not only refusing to output the preliminary decision, but also automatically triggering a signal to request the context inference layer to start a re-inference process. In this process, the adjusted strategy or parameters are used for re-inference, and the complete case data that caused the inconsistency is recorded in the historical database for subsequent system optimization.

[0059] It should be further explained that, in the specific implementation process, the system collects heterogeneous sensing data of the environment through multimodal sensing units deployed on edge computing devices. This data may include images, laser point clouds, millimeter-wave radar signals, audio, and other sensor data. After necessary preprocessing, the collected raw data is transmitted to the core processing component of the system, namely the dynamic cognitive kernel in the data processing unit.

[0060] The dynamic cognitive kernel is key to achieving intelligent collaborative decision-making, and its operation comprises three main stages. The first stage is dynamic modal credibility assessment. In this stage, the modal credibility assessment layer in the kernel invokes a corresponding lightweight self-supervised anomaly detection model for each type of incoming perceptual data stream. These models are trained before system deployment and are able to learn the data patterns of each modality under normal operating conditions.

[0061] During runtime, each model continuously analyzes the internal state of its corresponding data stream. For example, for an image stream, the model analyzes the stability of specific features or fluctuations in brightness distribution between consecutive frames; for an acoustic stream, it analyzes the continuity of its spectral features. Simultaneously, this layer also performs cross-modal expected consistency analysis, determining whether observations of the same event or target from different sensors support each other.

[0062] For example, when a vision sensor identifies an object, the system checks whether the radar sensor has a corresponding detection response at the corresponding spatial coordinates. Integrating the analysis results of internal consistency and cross-modal consistency, this layer calculates and outputs a quantified health factor for each modality, which reflects the reliability of that modality in real time.

[0063] The second stage is context-based fusion reasoning and strategy generation. Health factors and real-time environmental parameters are jointly input into the context-based reasoning layer. This layer has a built-in scene knowledge graph that stores rules constructed from expert experience and historical data. These rules define the mapping relationships between environmental conditions, device status, and optimization decision-making strategies. The reasoning engine matches the current input facts with the rule conditions in the knowledge graph.

[0064] For example, the rule might stipulate that if the ambient light intensity is low and the visual health factor declines, the weight of visual data should be reduced and the weight of radar data increased. Upon successful matching, the engine activates the corresponding rule conclusion and outputs two parts: first, a dynamic modal fusion weight matrix, explicitly indicating the contribution ratio of each modality's data in subsequent fusion calculations; and second, conflict resolution instructions, specifying the resolution logic to be followed when perception results from different modalities contradict each other.

[0065] The third stage is multi-level verification of decision confidence. The preliminary decision result is sent to the decision verification layer. This layer first uses a lightweight adversarial verification network to stress test the decision. This network verifies the robustness of the decision by simulating anomalies such as sensor occlusion and data noise, and outputs the first verification result. At the same time, the verification layer accesses the historical decision database, performs pattern matching between the features of the current decision and the successful cases recorded in the database, and outputs the second verification result. The system compares the preliminary decision, the first verification result, and the second verification result. If the three are consistent, the decision is confirmed to be reliable, and the final decision instruction is output. If there is a discrepancy, a re-inference process is triggered, requiring the context reasoning layer to adjust the parameters and regenerate the decision, and the current case record is used for the optimization and update of the knowledge graph.

[0066] The final decision instruction is sent to the decision execution unit, which controls the executor to complete the corresponding physical actions according to the instruction, thus forming a complete closed loop from perception and collaborative reasoning to execution. A dynamic cognitive kernel enables real-time assessment of perception reliability, dynamic strategy adjustment based on knowledge reasoning, and multi-level verification of decision outputs, thereby enhancing the perception and decision-making capabilities of the edge system in complex and dynamic environments.

[0067] By introducing a dynamic cognitive kernel, real-time assessment of the reliability of multimodal perception data and dynamic weight adjustment are achieved, effectively overcoming the shortcomings of traditional fixed rule systems where collaborative efficiency declines when the environment changes, and improving the system's adaptability and decision-making accuracy in complex scenarios.

[0068] Through a multi-level decision verification mechanism and closed-loop optimization design, the system can perform multiple verifications on decisions before output, ensuring high reliability of decisions. At the same time, it continuously optimizes the knowledge base using historical cases, enhancing the system's long-term learning ability and operational stability, and avoiding system risks caused by a single decision error.

[0069] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0070] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A system for multimodal perception and decision-making collaboration at the edge, characterized in that, include: A multimodal sensing unit for collecting heterogeneous sensing data of the environment; The data processing unit is communicatively connected to the multimodal sensing unit; The data processing unit includes a dynamic cognitive kernel, which in turn includes a modal credibility assessment layer, a contextual reasoning layer, and a decision verification layer. The modal reliability assessment layer is configured to receive the heterogeneous sensing data and, based on the internal state of each modal data stream and the intermodal relationship, output a health factor characterizing the real-time reliability of each modality. The context reasoning layer is communicatively connected to the modal credibility evaluation layer and stores a scene knowledge graph. It is configured to receive the health factors and environmental parameters and output a dynamic fusion strategy through reasoning. The decision verification layer is communicatively connected to the context reasoning layer and is configured to verify the preliminary decision output by the context reasoning layer and output the final decision instruction after the verification is successful. A decision execution unit that is communicatively connected to the data processing unit and is used to execute the final decision instruction.

2. The system for edge-end multimodal perception and decision-making collaboration according to claim 1, characterized in that: The modal credibility assessment layer includes several self-supervised anomaly detection models, each corresponding one-to-one with a perceived modality. Each self-supervised anomaly detection model is configured to measure the internal consistency of the modal data by analyzing the temporal stability of its corresponding modal data.

3. The system for edge-end multimodal perception and decision-making collaboration according to claim 2, characterized in that: The modal credibility assessment layer is further configured to measure the expected consistency across modalities by analyzing the degree of mutual support between different modal perception data in space or semantics; the health factor is calculated based on the internal consistency and the expected consistency.

4. The system for edge-end multimodal perception and decision-making collaboration according to claim 1, characterized in that: The scenario knowledge graph constructs mapping rules between environmental context information, modal health status, and fusion decision strategies; the context reasoning layer is configured to substitute the current health factors and environmental parameters into the mapping rules for reasoning, generating a modal fusion weight matrix and conflict resolution strategy suitable for the current scenario.

5. The system for edge-end multimodal perception and decision-making collaboration according to claim 1, characterized in that: The decision verification layer includes a lightweight adversarial verification network and a historical decision database; The lightweight adversarial verification network is configured to perform stress testing on the preliminary decision and output a first verification result. The historical decision database stores historical successful decision cases and is configured to provide comparison data to the decision verification layer to compare the preliminary decision with the historical successful decision pattern and output a second verification result.

6. The system for edge-end multimodal perception and decision-making collaboration according to claim 5, characterized in that: The decision verification layer is configured as follows: when the preliminary decision, the first verification result, and the second verification result are consistent, the preliminary decision is confirmed as the final decision instruction; when the three are inconsistent, the context reasoning layer is triggered to start the re-reasoning process, and the current case data is recorded in the historical decision database to optimize the scene knowledge graph.

7. A method for multimodal perception and decision-making collaboration at the edge, characterized in that, The system applied to any one of claims 1 to 6 includes the following steps: S1: Collect heterogeneous sensing data of the environment through a multimodal sensing unit; S2: The heterogeneous sensing data is processed through the modal credibility assessment layer of the dynamic cognitive kernel to output real-time health factors for each modality; S3: Through the contextual reasoning layer of the dynamic cognitive kernel, the health factors and real-time environmental parameters are combined to perform reasoning, generate a dynamic fusion strategy and output a preliminary decision; S4: The preliminary decision is verified through the decision verification layer of the dynamic cognitive kernel, and the final decision instruction is output after the verification is passed. S5: Execute the final decision instruction through the decision execution unit.

8. The method for edge-end multimodal perception and decision-making collaboration according to claim 7, characterized in that: The steps for outputting real-time health factors for each modality include: The internal consistency of the data stream of each modality is analyzed using a self-supervised anomaly detection model corresponding to each modality. Analyze the expected consistency among sensing data from different modalities; The health factor is calculated by combining the analysis results of the internal consistency and the expected consistency.

9. The method for edge-end multimodal perception and decision-making collaboration according to claim 7, characterized in that: The steps for generating the dynamic fusion strategy include: Access the scene knowledge graph that stores mapping rules; The health factors and environmental parameters are used as inputs, and inference is performed using the mapping rules. Outputs a dynamic weight matrix and conflict resolution instructions to guide the current multimodal data fusion.

10. The method for edge-end multimodal perception and decision-making collaboration according to claim 7, characterized in that: The steps for multi-level consistency verification of preliminary decisions include: The initial decision is input into a lightweight adversarial verification network for stress testing to obtain the first verification result; The preliminary decision is pattern-matched with historical successful decision-making cases to obtain a second verification result; Compare the preliminary decision, the first verification result, and the second verification result; Based on the comparison results, a final decision instruction will be output or a re-reasoning process will be triggered.

Images