A method and system of edge-end multi-modal perception and decision-making collaboration

Abstract

The application discloses a kind of edge end multimodal perception and decision-making coordination method and system, the present application relates to edge computing and artificial intelligence cross technical field, including: the multimodal perception unit for collecting the heterogeneous perception data of environment;Data processing unit is communicatively connected with the multimodal perception unit;Wherein, the data processing unit includes dynamic cognitive kernel, the dynamic cognitive kernel successively includes modal credibility evaluation layer, context reasoning layer and decision check layer;The modal credibility evaluation layer is configured to receive the heterogeneous perception data.This edge end multimodal perception and decision-making coordination method and system, by introducing dynamic cognitive kernel, realizes the real-time evaluation and dynamic weight adjustment of the reliability of multimodal perception data, effectively overcome the defect that traditional fixed rule system is coordinated when environment changes Efficiency decline, improve the adaptability and decision accuracy of system in complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing and artificial intelligence, in particular to a method and system for edge multi-modal perception and decision-making collaboration. BACKGROUND

[0002] At present, multi-modal perception and decision-making systems deployed in edge computing environments are widely used. Such systems usually rely on pre-set fixed rules or trained static models to realize the collaboration and decision-making of multiple perception modalities. However, the real physical environment has high dynamicity and uncertainty, such as sudden changes in light, partial shielding of sensors, or weather influence, which can cause real-time fluctuations in the reliability of each modality. Existing methods cannot dynamically evaluate the confidence level of each modality, and still use pre-set fusion weights or simple conflict voting strategies, resulting in a sharp decline in the overall accuracy and robustness of decision-making when the environment changes. At the same time, the inherent resource constraints of edge terminals require the system to have the ability to autonomously allocate computing resources, and traditional static architectures are difficult to adaptively adjust according to real-time situations, resulting in low resource utilization or delay in critical task decision-making. Therefore, the existing system lacks a mechanism that can reason in real time, dynamically coordinate multi-modal perception, and intelligently allocate resources, which seriously restricts its application effect in complex scenarios. The present application aims to solve the problem of rigid collaboration mechanism and insufficient decision-making reliability in edge multi-modal perception systems caused by dynamic changes in the environment. SUMMARY

[0003] The present application aims to provide a method and system for edge multi-modal perception and decision-making collaboration to solve the problems raised in the background.

[0004] To solve the above technical problems, the present application provides the following technical solution: a method and system for edge multi-modal perception and decision-making collaboration, comprising:

[0005] a multi-modal perception unit for collecting heterogeneous perception data of the environment;

[0006] a data processing unit in communication connection with the multi-modal perception unit;

[0007] wherein the data processing unit comprises a dynamic cognitive kernel, and the dynamic cognitive kernel comprises a modality confidence assessment layer, a context reasoning layer, and a decision verification layer in sequence;

[0008] the modality confidence assessment layer is configured to receive the heterogeneous perception data, and based on the internal state of each modality data stream and the relationship between modalities, output health factors representing the real-time reliability of each modality;

[0009] The context reasoning layer is in communication connection with the modality credibility evaluation layer and stores a scene knowledge graph, and is configured to receive the health factor and the environmental parameter, and output a dynamic fusion strategy through reasoning;

[0010] The decision verification layer is in communication connection with the context reasoning layer, and is configured to verify the preliminary decision output by the context reasoning layer, and output a final decision instruction after verification;

[0011] A decision execution unit in communication connection with the data processing unit, used for executing the final decision instruction.

[0012] Preferably, the modality credibility evaluation layer comprises a plurality of self-supervised anomaly detection models, and each self-supervised anomaly detection model corresponds to a sensing modality; each self-supervised anomaly detection model is configured to measure the internal consistency of the modality data by analyzing the stability of the modality data in time sequence.

[0013] Preferably, the modality credibility evaluation layer is further configured to measure the expected consistency between different modalities by analyzing the mutual support degree of the sensing data in space or semantics; the health factor is calculated based on the internal consistency and the expected consistency.

[0014] Preferably, the scene knowledge graph is constructed with mapping rules between environmental context information, modality health state and fusion decision strategy; the context reasoning layer is configured to substitute the current health factor and the environmental parameter into the mapping rules for reasoning to generate a modality fusion weight matrix and a conflict resolution strategy suitable for the current scene.

[0015] Preferably, the decision verification layer comprises a lightweight adversarial verification network and a historical decision database.

[0016] The lightweight adversarial verification network is configured to perform stress testing on the preliminary decision and output a first verification result.

[0017] The historical decision database stores historical successful decision cases, and is configured to provide comparison data to the decision verification layer to compare the preliminary decision with the historical successful decision mode and output a second verification result.

[0018] Preferably, the decision verification layer is configured to: when the preliminary decision, the first verification result and the second verification result are consistent, confirm the preliminary decision as the final decision instruction; when the three are inconsistent, trigger the context reasoning layer to start a re-reasoning process, and record the current case data to the historical decision database to optimize the scene knowledge graph.

[0019] The application relates to a method for edge-end multi-modal perception and decision-making cooperation, which is applied to a system for edge-end multi-modal perception and decision-making cooperation and comprises the following steps.

[0020] S1: collecting heterogeneous perception data of an environment through a multi-modal perception unit;

[0021] S2: processing the heterogeneous perception data through a modal credibility evaluation layer of a dynamic cognitive kernel to output real-time health factors of each mode;

[0022] S3: combining the health factors and real-time environment parameters through a context reasoning layer of the dynamic cognitive kernel to generate a dynamic fusion strategy and output a preliminary decision;

[0023] S4: performing multi-level consistency verification on the preliminary decision through a decision verification layer of the dynamic cognitive kernel, and outputting a final decision instruction after verification;

[0024] S5: executing the final decision instruction through a decision execution unit.

[0025] Preferably, the step of outputting real-time health factors of each mode comprises the following steps:

[0026] using a self-supervised anomaly detection model corresponding to each mode to analyze the internal consistency of each mode data stream;

[0027] analyzing the expected consistency between different modal perception data;

[0028] comprehensively analyzing the internal consistency and the expected consistency to calculate the health factors.

[0029] Preferably, the step of generating a dynamic fusion strategy comprises the following steps:

[0030] accessing a scene knowledge graph storing mapping rules;

[0031] inputting the health factors and the environment parameters to perform reasoning through the mapping rules;

[0032] outputting a dynamic weight matrix and a conflict resolution instruction for guiding current multi-modal data fusion.

[0033] Preferably, the step of performing multi-level consistency verification on the preliminary decision comprises the following steps:

[0034] inputting the preliminary decision into a lightweight adversarial verification network to perform stress testing and obtain a first verification result;

[0035] performing mode matching on the preliminary decision and historical successful decision cases to obtain a second verification result;

[0036] Compare the preliminary decision, the first verification result and the second verification result;

[0037] According to the comparison result, a final decision instruction is output or a re-reasoning process is triggered.

[0038] The present application provides a kind of edge multi-modal perception and decision-making coordination method and system.It has the following beneficial effects:

[0039] The edge multi-modal perception and decision-making coordination method and system, by introducing dynamic cognitive kernel, realizes the real-time evaluation and dynamic weight adjustment to multi-modal perception data reliability, effectively overcome the defect that traditional fixed rule system coordination efficiency declines when environment changes, improve the adaptability and decision accuracy of system in complex scene.

[0040] The edge multi-modal perception and decision-making coordination method and system, through multi-level decision checking mechanism and closed loop optimization design, system can be verified multiple times before outputting decision, ensure the high reliability of decision, while using historical cases to continuously optimize knowledge base, enhance the long-term learning ability and running stability of system, avoid the system risk caused by single decision error. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 For the module interaction schematic diagram of the edge multi-modal perception and decision-making coordination system of the present application;

[0042] Figure 2 For the flowchart of the edge multi-modal perception and decision-making coordination method of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.It is obvious that the described embodiments are only part of the embodiments of the present application, not all the embodiments.Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0044] Please refer to Figure 1 and Figure 2 The present application provides a technical solution: a kind of edge multi-modal perception and decision-making coordination method and system, comprising:

[0045] Multi-modal perception unit for collecting heterogeneous perception data of environment;

[0046] Data processing unit in communication connection with multi-modal perception unit;

[0047] The data processing unit comprises a dynamic cognitive kernel, and the dynamic cognitive kernel comprises a modality credibility evaluation layer, a context reasoning layer and a decision verification layer in sequence.

[0048] The modality credibility evaluation layer is configured to receive heterogeneous perception data, and output health factors representing real-time reliability of each modality based on internal states of each modality data stream and inter-modality relationships.

[0049] The context reasoning layer is in communication connection with the modality credibility evaluation layer and stores a scene knowledge graph, and is configured to receive the health factors and environment parameters and output a dynamic fusion strategy through reasoning.

[0050] The decision verification layer is in communication connection with the context reasoning layer, and is configured to verify a preliminary decision output by the context reasoning layer and output a final decision instruction after verification.

[0051] The decision execution unit in communication connection with the data processing unit is configured to execute the final decision instruction.

[0052] It should be further explained that, in the specific implementation process, the system first synchronously or asynchronously collects original perception data of the environment through different sensors in the multi-modal perception unit, such as a camera, a laser radar, a millimeter wave radar, a microphone array or a temperature and humidity sensor. After preliminary signal conditioning and format unification, the heterogeneous data are transmitted to the dynamic cognitive kernel in the data processing unit for core processing.

[0053] The operation of the dynamic cognitive kernel starts from the modality credibility evaluation layer. This layer runs a lightweight self-supervised anomaly detection model in parallel for each modality data stream. For the visual modality, the model evaluates the internal consistency by analyzing the definition, contrast or feature point matching rate of consecutive frames of images; for the acoustic modality, the model evaluates by analyzing the fluctuation of background noise level and signal-to-noise ratio. At the same time, this layer performs cross-modality expected consistency analysis, for example, when the visual sensor detects a moving object, the system expects the radar sensor to have feedback in the corresponding spatial coordinates; if the radar does not feedback or the feedback coordinates have large deviations, it is determined that the cross-modality consistency is reduced.

[0054] Based on the analysis results of the internal consistency and the expected consistency, this layer calculates and outputs a dynamic and quantitative health factor for each modality, which directly reflects the reliability degree of the modality under the current environmental conditions.

[0055] Subsequently, the health factor and the current environmental parameters are jointly input to the context reasoning layer. The scenario knowledge graph built in this layer stores rules defined by domain knowledge, such as “IF ambient light intensity is lower than threshold AND visual health factor is decreased THEN increase the fusion weight of the laser radar modality” or “IF rainy day AND laser radar health factor is moderately decreased THEN enable the rain and fog filtering algorithm and preferentially use millimeter wave radar data for dominant decision making”; wherein the current environmental parameters include the ambient light intensity obtained through the ambient light sensor and the weather condition obtained through the weather interface.

[0056] The inference engine matches and reasons the input according to these rules, and finally generates a set of dynamic fusion strategies suitable for the current specific scenario, including the weight matrix of each modality in data fusion and the conflict resolution instructions to be followed when the perception results of different modalities are contradictory. According to this strategy, the multi-modal data is fused and calculated to generate a preliminary decision result.

[0057] The preliminary decision result is not immediately executed, but is sent to the decision verification layer for multi-level consistency verification. The verification layer first calls a lightweight adversarial verification network, which performs stress testing on the preliminary decision by injecting simulated perception noise or simulating extreme scenarios to observe whether the conclusion remains stable, and outputs the first verification result; wherein the extreme scenarios include extreme occlusion and strong interference.

[0058] At the same time, the verification layer accesses the historical decision database, and performs pattern matching and similarity calculation between the features of the current preliminary decision and the historical successful decision cases recorded in the database, outputting the second verification result. The system compares the preliminary decision, the first verification result and the second verification result: if they are consistent, the preliminary decision is confirmed as the final decision instruction; if there is inconsistency, for example, the decision is fragile or the historical pattern matching degree is low, the verification layer will trigger a signal to require the context reasoning layer to start the re-reasoning process, and at the same time, the current environmental parameters, perception data and decision result are recorded as a new case in the historical database for subsequent iterative optimization of the scenario knowledge graph.

[0059] Finally, the final decision instruction verified through all the checks is sent to the decision execution unit, which can be a vehicle control unit, a robot motion controller or an industrial alarm trigger, responsible for converting digital instructions into specific physical actions. The entire process runs in a closed loop, enabling the system to adapt to dynamic changes in the environment, intelligently coordinate multi-modal perception resources, and ensure the reliability of the decision.

[0060] The modality credibility evaluation layer includes a plurality of self-supervised anomaly detection models, each of which corresponds to a perception modality; each self-supervised anomaly detection model is configured to measure the internal consistency of the modality data by analyzing the stability of the modality data in time sequence. It needs to be further explained that in the specific implementation process, the modality credibility evaluation layer is internally built with a plurality of self-supervised anomaly detection models to achieve its function. These models are pre-established and maintained in a one-to-one correspondence with each perception modality required by the system when deployed. The core operating mechanism of each self-supervised anomaly detection model serving its corresponding modality is to continuously and real-time analyze the performance characteristics of the incoming modality data stream in the time dimension, which serves as the fundamental basis for evaluating the internal consistency of the modality.

[0061] For the visual modality, the corresponding model analyzes the smoothness of the trajectory of specific feature points between consecutive image frames, the fluctuation of overall brightness or color distribution. If the image sequence shows abnormal jitter, sudden brightness change or a large number of feature points loss, the model will determine that its time sequence stability is reduced. For the acoustic modality, the model focuses on analyzing the variation rule of the energy spectrum of the audio signal in the short time window, or the continuity of the energy distribution of the specific frequency band. The model of the radar modality monitors the change rate of the point cloud density and the stability of the echo intensity.

[0062] This measurement of time sequence stability does not rely on a pre-defined absolute threshold, but is based on the inherent pattern of the modality data stream learned by the model during the system initialization phase or normal running stable period. The model generates a quantitative evaluation value of internal consistency by comparing the deviation between real-time data and learned patterns. This evaluation value is a key input for subsequent calculation of the comprehensive health factor, and its purpose is to dynamically capture the degradation of data quality of any modality caused by environmental interference or changes in the state of the sensor, thereby providing the subsequent fusion reasoning with crucial original judgments reflecting the real-time reliability of each modality.

[0063] The modal credibility evaluation layer is further configured to measure the expected consistency between different modal perception data by analyzing the mutual support degree of the different modal perception data in space or semantics; and the health factor is calculated based on the internal consistency and the expected consistency. It needs to be further explained that in the specific implementation process, in addition to analyzing the internal consistency of each modal itself, the modal credibility evaluation layer also undertakes the task of evaluating the mutual support and confirmation relationship between different modal perception data, i.e., the expected consistency between different modalities. The system synchronizes the data from different sensors in the coordinate system and the time stamp through the pre-defined space-time alignment rule, and establishes the foundation for cross-modal analysis. For example, when the camera of the visual modality recognizes the outline of an object and gives its two-dimensional image coordinates, the system will project the coordinates into a ray in the three-dimensional space based on the pre-calibrated external parameter matrix, and expects that the laser radar or millimeter wave radar modality should be able to detect the corresponding point cloud or reflected point cloud cluster within a certain distance range in the direction of the ray.

[0064] If the radar data successfully detects the target in the expected space region, and the reported speed and size are basically consistent with the visual judgment, it is determined that the cross-modal expected consistency is high; otherwise, if the radar data in the region is missing, or the reported target attributes and the visual judgment have unexplainable conflicts, it is determined that the expected consistency is reduced. For the semantic level expectation, if the microphone array of the audio modality captures a sharp ringing sound, and the visual modality does not detect a significant event that may produce such a sound, such as object collision, at the same time, the system will also record this semantic expected inconsistency.

[0065] The logic processing unit built in the evaluation layer will comprehensively analyze the time sequence stability of each modality and the above-mentioned consistency analysis results between different modalities, and use a rule-based or weighted fusion algorithm to calculate a health factor that can more comprehensively and accurately reflect the reliability of the modality in the current comprehensive environment. The health factor no longer only reflects the working state of a sensor itself, but also contains the credibility of the sensor in the multi-sensor cooperative system, providing more abundant and reliable input basis for the subsequent reasoning layer to formulate a highly adaptive fusion strategy.

[0066] The scene knowledge graph constructs mapping rules between environmental context information, modal health status and fusion decision strategies; the context reasoning layer is configured to substitute the current health factors and environmental parameters into the mapping rules for reasoning to generate a modal fusion weight matrix and a conflict resolution strategy suitable for the current scene. It needs to be further explained that in the specific implementation process, the scene knowledge graph is not a simple database, but a network that stores mapping rules abstracted from expert experience and historical data. These rules define the complex causal relationships and logical associations between environmental context information, modal health status and final fusion decision strategies. The construction of this graph is based on the analysis of a large number of typical scenes. For example, in the field of autonomous driving, one rule may state: "When the environmental context information is night and the weather condition is rain and fog, if the health factor of the laser radar modality is maintained at a high level and the health factor of the visual modality decreases, then trigger fusion strategy A"; another rule may be: "When the system is in a congested urban road scene and the acoustic modality health factor shows reliable, if an emergency vehicle honking sound is detected, then regardless of the immediate results of the visual and radar modalities, strategy B is preferentially executed to avoid."

[0067] The context reasoning layer receives real-time health factors from the evaluation layer and environmental parameters obtained through external interfaces at runtime, and matches these real-time data as input facts with the rule conditions stored in the knowledge graph. The reasoning process uses a rule-based reasoning engine that traverses the rules in the graph to find entries whose condition parts match the current input facts. When a match is successful, the dynamic fusion strategy indicated in the conclusion part of the rule is activated, which is materialized as a weight matrix that guides the multi-modal data fusion calculation, for example, explicitly specifying that the radar data weight increases by twenty percent and the visual data weight decreases by forty percent under the current conditions; and a series of conflict resolution instructions, for example, when the visual and radar classifications of the same target are inconsistent, the radar classification result is preferred. In this way, the system is no longer statically applying one fusion algorithm, but can dynamically and adaptively select and generate the most suitable collaborative perception and decision scheme for the current rapidly changing working conditions based on the in-depth understanding of the environment.

[0068] The decision verification layer includes a lightweight adversarial verification network and a historical decision database;

[0069] The lightweight adversarial verification network is configured to stress test the preliminary decision and output a first verification result;

[0070] The historical decision database stores historical successful decision cases and is configured to provide comparison data to the decision verification layer to compare the preliminary decision with the historical successful decision mode and output a second verification result.

[0071] It needs to be further explained that in the specific implementation process, the decision verification layer integrates two core components to realize its multi-level verification function: a lightweight adversarial verification network and a historical decision database. The lightweight adversarial verification network is a specially trained efficient neural network model, and its design goal is not to perform complex perception tasks, but to focus on robustness stress testing of the preliminary decision results output by the context reasoning layer. This network operates by actively injecting simulated and controllable interference patterns into the perception features or data on which the preliminary decision is based. These interference patterns simulate typical abnormal situations that may occur in edge environment, such as simulating data loss caused by partial sensor occlusion, simulating the impact of specific weather conditions on sensing accuracy, or simulating random noise caused by electromagnetic interference; among them, the specific weather conditions include rain and fog.

[0072] The network observes whether the preliminary decision conclusion will be overturned after being subjected to these injected interferences, and outputs a first verification result about the robustness of the decision accordingly. The historical decision database is a structured repository that continuously records successful decision cases and their corresponding key context features generated in the system's past operation and verified as successful afterwards, where the key context features include environmental state, modality health factors, and the strategy finally adopted. After obtaining the preliminary decision, the decision verification layer accesses the database in real time, matches the features of the current decision with historical cases and calculates the similarity. If it is found that the current decision is highly similar to the historical successful decision in pattern, a positive second verification result is output; if the current decision shows an abnormal or rare pattern, a verification result that needs to be vigilant is output. The cooperative work of these two components constitutes the core foundation of the multi-level verification capability of the decision verification layer, providing double protection for the reliability of the final decision instruction.

[0073] The decision verification layer is configured to: when the preliminary decision, the first verification result and the second verification result are consistent, confirming the preliminary decision as the final decision instruction; when the three are inconsistent, triggering the context reasoning layer to start the re-reasoning process, and recording the current case data to the historical decision database to optimize the scene knowledge graph. It needs to be further explained that in the specific implementation process, the logic processing unit built in the decision verification layer is responsible for comprehensive research and judgment on the received preliminary decision, the first verification result from the lightweight adversarial verification network and the second verification result from the historical decision database comparison. The unit compares the three inputs to determine whether their conclusions point to the same decision direction or compatible decision results. If the three are consistent, the logic processing unit confirms that the preliminary decision has high reliability, and immediately outputs it as the final decision instruction and issues it to the decision execution unit. If the three are inconsistent, for example, the preliminary decision suggests executing action A, but the adversarial test shows that the decision becomes unstable under interference, or the historical record shows that the successful decision in a similar situation is mostly action B, then the logic processing unit determines that the current decision is at high risk. At this time, the unit will not directly output the unreliable decision, but will generate a trigger signal, which is sent to the context reasoning layer, instructing it to start the re-reasoning process.

[0074] In the re-reasoning process, the context reasoning layer may use adjusted parameters, such as giving a lower mode of the health factor a higher distrust weight, or trying to call a backup, more conservative fusion rule in the knowledge graph to generate a new preliminary decision for re-verification.

[0075] At the same time, the logic processing unit will package all the context data that caused the inconsistency, including environmental parameters, original perception data of each modality, health factors, preliminary decision results and verification results, as a special case, and transmit it to the historical decision database for storage. This case is marked as a case to be optimized, and the system will use these cases to incrementally learn and optimize the scene knowledge graph in the background resource idle time or periodic tasks, such as adjusting the confidence of a specific rule or supplementing new rule items, so that the entire system has the characteristics of learning from decision differences and continuously improving its coordination and decision-making ability.

[0076] A method of edge multi-modal perception and decision-making collaboration, applied to a system of edge multi-modal perception and decision-making collaboration, comprising the following steps:

[0077] S1: collecting heterogeneous perception data of the environment through a multi-modal perception unit;

[0078] S2: processing the heterogeneous perception data through the modality credibility evaluation layer of the dynamic cognitive kernel to output real-time health factors of each modality;

[0079] S3: Inference is performed on the health factors and real-time environmental parameters by the context reasoning layer of the dynamic cognitive kernel to generate a dynamic fusion strategy and output a preliminary decision;

[0080] S4: The preliminary decision is verified for multi-level consistency by the decision verification layer of the dynamic cognitive kernel, and a final decision instruction is output after verification;

[0081] S5: The final decision instruction is executed by the decision execution unit.

[0082] It should be further explained that in the specific implementation process, the method starts with the synchronous or asynchronous acquisition of raw perception data of the environment by various sensors in the multi-modal perception unit deployed on the edge device, which can include images, point clouds, audio, temperature and other modalities. After necessary preprocessing, the heterogeneous data is transmitted to the core processing link of the system, i.e., the dynamic cognitive kernel. The processing flow of the kernel is first executed by the modal credibility evaluation layer, which calls the self-supervised anomaly detection model corresponding to each modality built into the system to analyze the continuous inflow of each modality data stream in real time. By evaluating its own time stability and the mutual support degree between different modal data, a set of health factors that can quantitatively reflect the reliability of each modality under the current environmental conditions is calculated and output.

[0083] These health factors are immediately sent to the context reasoning layer along with the real-time environmental parameters obtained through other interfaces; the layer accesses the scenario knowledge graph stored in it, which is a rule base containing the mapping relationship between environmental context, device state and optimization decision strategy. The reasoning engine takes the current health factors and environmental parameters as input facts, matches them with the rule conditions in the knowledge graph, generates a set of dynamic fusion strategies adapted to the current scenario through logical reasoning, and applies the strategies to the fusion calculation of the original perception data, finally outputs a preliminary decision result; among them, the fusion strategy includes weight allocation and conflict resolution rules.

[0084] This preliminary decision result is not the end, but also needs to be submitted to the decision verification layer for multi-level consistency verification; the verification layer calls the lightweight adversarial verification network to test the decision, queries the historical decision database for pattern comparison, and compares the test results and comparison results with the preliminary decision itself; only when the three conclusions are consistent, the preliminary decision is confirmed as the final and reliable decision instruction; if there is inconsistency, the re-reasoning mechanism is triggered and the system is optimized.

[0085] Finally, the verified decision instruction is sent to the decision execution unit, which is converted into specific control commands or operation actions, thereby completing the complete closed loop from perception to collaborative decision to execution.

[0086] The step of outputting the real-time health factor of each modality includes:

[0087] The internal consistency of each modality data stream is analyzed by using a self-supervised anomaly detection model corresponding to each modality;

[0088] The expected consistency between different modality perception data is analyzed;

[0089] The analysis results of internal consistency and expected consistency are integrated to calculate the health factor.

[0090] It should be further explained that in the specific implementation process, this step is executed by calling the self-supervised anomaly detection model corresponding to each perception modality pre-established in the system. For each input perception data stream, the corresponding model will first analyze the internal regularity of the modality data in the time sequence, for example, for video stream data, the model will calculate the smoothness of the motion trajectory of specific image feature points or the fluctuation of the overall brightness distribution, so as to evaluate the internal consistency; for acoustic stream data, the energy change mode of short-time spectrum or the continuity of specific frequency band signal is analyzed.

[0091] At the same time, the system will analyze the expected consistency between different modality perception data in parallel, which needs to map data from different sources to a unified reference system through space-time alignment, and then judge their mutual support or contradiction; for example, when the visual modality recognizes an object in a certain spatial region, the system will check whether the laser radar modality detects a corresponding point cloud cluster near the corresponding coordinates, if the matching is successful, the expected consistency is high, if the radar has no feedback or the feedback attribute has significant difference in the region, the consistency is reduced.

[0092] Finally, the fusion logic built in the system will integrate the internal consistency analysis results for the same modality and the cross-modality expected consistency analysis results involving the modality, and the integration process can be carried out according to the preset rules or weighting strategy, finally a health factor value reflecting the real-time reliability of the modality under the current comprehensive conditions is calculated and output, which will be used as the key input of subsequent knowledge reasoning.

[0093] The step of generating a dynamic fusion strategy includes:

[0094] Accessing a scene knowledge graph storing mapping rules;

[0095] Taking the health factor and the environmental parameter as input, reasoning through the mapping rules;

[0096] Outputting a dynamic weight matrix and conflict resolution instruction for guiding the current multi-modality data fusion.

[0097] It needs to be further explained that in the specific implementation process, this step starts from the system accessing the scene knowledge graph stored in its internal storage, which is a structured rule base in which the mapping relationship between the environmental context parameters, the health status of each modality and the specific fusion decision strategy is constructed. These mapping relationships exist in the form of conditional-conclusion rules. The system combines the real-time health factor set output by the modality credibility evaluation layer with the real-time environmental parameters obtained through external sensors or interfaces to form the input fact set of the current state.

[0098] The reasoning engine then matches these input facts with the numerous rule conditions stored in the knowledge graph, looking for rule entries that match the condition part with the current input facts. The matching process involves logical judgments, such as whether the current environmental light intensity is lower than a certain threshold, while the health factor of the visual modality is also in the declining interval. Once the applicable rule is matched, the system activates the specific fusion strategy indicated by the rule conclusion part.

[0099] This strategy is specifically embodied in two operable outputs:

[0100] One is the dynamic modality fusion weight matrix, which clearly specifies the proportion of each modality data in the fusion algorithm under the current specific working condition, for example, in rainy and foggy weather and when the laser radar performance is stable, the data weight of the corresponding laser radar is increased.

[0101] The second is the conflict resolution instruction, which is a set of pre-set logical commands used to guide the system when the perception results of different modalities appear irreconcilable contradictions, such as instructing the system to preferentially adopt the classification result of the radar when the visual and radar classification of the object is inconsistent, or triggering a more bottom-level feature-level fusion for re-determination.

[0102] Through this series of reasoning actions based on real-time knowledge and context perception, the system generates a highly adaptive fusion strategy for the current dynamic environment, thereby realizing the fundamental change from rigid fixed rules to intelligent adaptive coordination.

[0103] The step of multi-level consistency verification of the preliminary decision includes:

[0104] Input the preliminary decision into a lightweight adversarial verification network for stress testing to obtain the first verification result;

[0105] Pattern matching the preliminary decision with historical successful decision cases to obtain the second verification result;

[0106] Compare the preliminary decision, the first verification result and the second verification result;

[0107] According to the comparison result, decide to output the final decision instruction or trigger the re-reasoning process.

[0108] It needs to be further explained that in the specific implementation process, this step starts from inputting the preliminary decision result generated by the context reasoning layer into a lightweight adversarial verification network, which is pre-trained to simulate the abnormal or extreme working conditions that may occur in various edge environments, and implements stress testing by injecting specific and controllable interference patterns into the feature space or data source on which the decision is based, such as simulating data missing caused by instantaneous sensor occlusion, or simulating the typical impact of specific weather conditions on perception data quality, and observing whether the preliminary decision conclusion remains stable in the face of these disturbances, thereby obtaining the first verification result about the robustness of the decision.

[0109] Synchronously, the system accesses the historical decision database, which stores a large number of successful decision cases and their corresponding key context feature vectors that have been verified after the fact, and performs similarity calculation and comparison between the features of the current preliminary decision and the historical cases through a pattern matching algorithm. If a highly similar historical successful pattern is found, a supportive second verification result is obtained, and if the current decision behaves as an abnormal pattern or differs greatly from the historical successful pattern, a warning second verification result is obtained.

[0110] Thereafter, the logic unit built into the system compares the consistency of the preliminary decision itself, the first verification result and the second verification result, and judges whether they point to the same final action direction. If the comparison result is consistent, the logic unit confirms that the preliminary decision is reliable and outputs it as the final decision instruction; if there is any inconsistency among the three, the logic unit determines that there is uncertainty or risk in the current decision-making process, not only refuses to output the preliminary decision, but also automatically triggers a signal to require the context reasoning layer to start a re-reasoning process, and in this process, the adjusted strategy or parameters are used for re-reasoning, and the complete case data that causes inconsistency is recorded in the historical database for subsequent system optimization.

[0111] It needs to be further explained that in the specific implementation process, the system collects heterogeneous perception data of the environment through the multi-modal perception unit deployed on the edge computing device, which can include images, laser point clouds, millimeter wave radar signals, audio and other sensor data. The raw data collected after necessary preprocessing is transmitted to the core processing component of the system, namely the dynamic cognitive kernel in the data processing unit.

[0112] The dynamic cognitive kernel is the key to realizing intelligent collaborative decision-making, and its operation includes three main stages. The first stage is the dynamic evaluation of modal credibility. In this stage, the modal credibility evaluation layer in the kernel calls a corresponding lightweight self-supervised anomaly detection model for each type of incoming perception data stream. These models are trained before system deployment and can learn the data patterns of each modality under normal working conditions.

[0113] At runtime, each model continuously analyzes the internal state of its corresponding data stream, for example, for image streams, the model analyzes the stability of certain features or the fluctuation of brightness distribution between consecutive frames; for acoustic streams, it analyzes the continuity of spectral features. At the same time, this layer also performs expected consistency analysis across modalities, that is, it judges whether the observations of different sensors on the same event or target support each other.

[0114] For example, when a visual sensor identifies an object, the system checks whether the radar sensor has corresponding detection feedback at the corresponding spatial coordinates. Based on the analysis results of internal consistency and cross-modality consistency, this layer calculates and outputs a quantitative health factor for each modality, which reflects the reliability of the modality in real time.

[0115] The second stage is context knowledge-based fusion reasoning and strategy generation. The health factor and real-time environmental parameters are jointly input into the context reasoning layer. This layer has a built-in scene knowledge graph, which stores rules constructed from expert experience and historical data, defining the mapping relationship between environmental conditions, device states and optimization decision strategies. The reasoning engine matches the current input facts with the rule conditions in the knowledge graph.

[0116] For example, the rule may stipulate that if the environmental light intensity is low and the visual health factor is low, the weight of visual data should be reduced and the weight of radar data should be increased. After successful matching, the engine activates the corresponding rule conclusion and outputs two parts of content: one is a dynamic modal fusion weight matrix, which clearly indicates the contribution proportion of each modality data in subsequent fusion calculation; the other is a conflict resolution instruction, which stipulates the solution logic to be followed when the perception results of different modalities are contradictory.

[0117] The third stage is multi-level verification of decision confidence. The preliminary decision result is sent to the decision verification layer. This layer first uses a lightweight adversarial verification network to stress test the decision, which verifies the robustness of the decision by simulating abnormal situations such as sensor occlusion and data noise, and outputs the first verification result. At the same time, the verification layer accesses the historical decision database, and matches the features of the current decision with the successful cases recorded in the database, outputting the second verification result. The system compares the preliminary decision, the first verification result and the second verification result. If they are consistent, the decision is confirmed to be reliable, and the output is the final decision instruction. If there is inconsistency, the re-reasoning process is triggered, requiring the context reasoning layer to adjust the parameters to generate a new decision, and the current case is recorded for optimization and update of the knowledge graph.

[0118] The final decision instruction is sent to a decision execution unit, which controls the actuator to complete the corresponding physical action according to the instruction, thereby forming a complete closed loop from perception, collaborative reasoning to execution. Through the dynamic cognitive kernel, real-time evaluation of perception reliability, dynamic strategy adjustment based on knowledge reasoning, and multi-level verification of decision output are realized, thereby improving the perception and decision-making ability of the edge system in a complex dynamic environment.

[0119] By introducing the dynamic cognitive kernel, real-time evaluation and dynamic weight adjustment of multi-modal perception data reliability are realized, effectively overcoming the defect that the collaborative efficiency of the traditional fixed rule system decreases when the environment changes, and improving the adaptability and decision accuracy of the system in a complex scene.

[0120] Through the multi-level decision verification mechanism and closed loop optimization design, the system can perform multiple verifications on the decision before output, ensuring the high reliability of the decision, while continuously optimizing the knowledge base using historical cases, enhancing the long-term learning ability and operation stability of the system, and avoiding the system risk caused by a single decision error.

[0121] It should be noted that, in this article, relational terms such as first and second are used merely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or inherent to such a process, method, article, or apparatus. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0122] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A system for multimodal perception and decision-making collaboration at the edge, characterized in that, include: A multimodal sensing unit for collecting heterogeneous sensing data of the environment; The data processing unit is communicatively connected to the multimodal sensing unit; The data processing unit includes a dynamic cognitive kernel, which in turn includes a modal credibility assessment layer, a contextual reasoning layer, and a decision verification layer. The modal reliability assessment layer is configured to receive the heterogeneous sensing data and, based on the internal state of each modal data stream and the intermodal relationship, output a health factor characterizing the real-time reliability of each modality. The modal credibility assessment layer includes several self-supervised anomaly detection models, each corresponding one-to-one with a perceived modality. Each self-supervised anomaly detection model is configured to measure the internal consistency of the modal data by analyzing the temporal stability of its corresponding modal data. The modal credibility assessment layer is further configured to measure the expected consistency across modalities by analyzing the degree of mutual support between different modal perception data in space or semantics; the health factor is calculated based on the internal consistency and the expected consistency. The context reasoning layer is communicatively connected to the modal credibility assessment layer and stores a scene knowledge graph. It is configured to receive the health factors and current environmental parameters, wherein the current environmental parameters include ambient light intensity and weather conditions. It outputs a dynamic fusion strategy through reasoning, and performs fusion and calculation on multimodal data according to the fusion strategy to generate a preliminary decision. The decision verification layer is communicatively connected to the context reasoning layer and is configured to verify the preliminary decision output by the context reasoning layer and output the final decision instruction after the verification is successful. The decision verification layer includes a lightweight adversarial verification network and a historical decision database; The lightweight adversarial verification network is configured to perform stress testing on the preliminary decision and output a first verification result. The historical decision database stores historical successful decision cases and is configured to provide comparison data to the decision verification layer to compare the preliminary decision with the historical successful decision pattern and output a second verification result. The decision verification layer is configured as follows: when the preliminary decision, the first verification result, and the second verification result are consistent, the preliminary decision is confirmed as the final decision instruction; when the three are inconsistent, the context reasoning layer is triggered to start a re-reasoning process, and the current case data is recorded in the historical decision database to optimize the scenario knowledge graph. A decision execution unit that is communicatively connected to the data processing unit and is used to execute the final decision instruction.

2. The system for edge-end multimodal perception and decision-making collaboration according to claim 1, characterized in that: The scenario knowledge graph constructs mapping rules between environmental context information, modal health status, and fusion decision strategies; the context reasoning layer is configured to substitute the current health factors and environmental parameters into the mapping rules for reasoning, generating a modal fusion weight matrix and conflict resolution strategy suitable for the current scenario.

3. A method for multimodal perception and decision-making collaboration at the edge, characterized in that, The system applied to any one of claims 1 to 2 includes the following steps: S1: Collect heterogeneous sensing data of the environment through a multimodal sensing unit; S2: The heterogeneous sensing data is processed through the modal credibility assessment layer of the dynamic cognitive kernel to output real-time health factors for each modality; S3: Through the contextual reasoning layer of the dynamic cognitive kernel, the health factors and real-time environmental parameters are combined to perform reasoning, generate a dynamic fusion strategy and output a preliminary decision; S4: The preliminary decision is verified through the decision verification layer of the dynamic cognitive kernel, and the final decision instruction is output after the verification is passed. S5: Execute the final decision instruction through the decision execution unit.

4. The method for edge-end multimodal perception and decision-making collaboration according to claim 3, characterized in that: The steps for outputting real-time health factors for each modality include: The internal consistency of the data stream of each modality is analyzed using a self-supervised anomaly detection model corresponding to each modality. Analyze the expected consistency among sensing data from different modalities; The health factor is calculated by combining the analysis results of the internal consistency and the expected consistency.

5. The method for edge-end multimodal perception and decision-making collaboration according to claim 3, characterized in that: The steps for generating the dynamic fusion strategy include: Access the scene knowledge graph that stores mapping rules; The health factors and environmental parameters are used as inputs, and inference is performed using the mapping rules. Outputs a dynamic weight matrix and conflict resolution instructions to guide the current multimodal data fusion.

6. The method for edge-end multimodal perception and decision-making collaboration according to claim 3, characterized in that: The steps for multi-level consistency verification of preliminary decisions include: The initial decision is input into a lightweight adversarial verification network for stress testing to obtain the first verification result; The preliminary decision is pattern-matched with historical successful decision-making cases to obtain a second verification result; Compare the preliminary decision, the first verification result, and the second verification result; Based on the comparison results, a final decision instruction will be output or a re-reasoning process will be triggered.

Images