A multi-modal perception based integrated hierarchical decision system, method and robot

Abstract

The application provides a kind of based on multimodal perception comprehensive hierarchical decision system, method and robot, system includes remote processing center and: multimodal sensing component, the multimodal sensing component includes first sensor group of always-on operation and standby sensor group to be activated, the standby sensor group includes second sensor group and third sensor group;Near end processing center, comprising: perception uncertainty assessment unit, for determining whether to activate the sensor of at least one mode in the second sensor group to obtain second raw perception data based on obtaining the first raw perception data of the first sensor group;The perception result including monomodal confidence, global perception uncertainty;Comprehensive decision unit, based on the perception result, determine the execution mode of generating control instruction.Comprehensively optimize one or more of the goals of improving recognition accuracy, real-time performance, robustness and optimizing computing power consumption and sensor energy consumption.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a comprehensive hierarchical decision-making system, method, and robot based on multimodal perception. Background Technology

[0002] Robots need to meet a variety of comprehensive requirements, such as recognition accuracy, real-time performance, and robustness, in scenarios such as personnel detection, abnormal event recognition, environmental risk monitoring, obstacle avoidance, and proactive interaction.

[0003] Chinese patent application number 202510077224.0 discloses a multi-sensor collaborative intelligent robot control method. This method can adjust the reliability of sensor data in real time according to environmental conditions, dynamically allocate fusion weights, and ensure data accuracy through cross-validation of adjacent sensor data. Simultaneously, by combining an ant colony collaboration algorithm with adaptive pheromone evaporation regulation, it can flexibly adjust path exploration strategies according to task complexity and dynamic environmental changes, effectively improving search efficiency, adapting to various environments more quickly, and rapidly finding the optimal path. However, it lacks acoustic sensing coverage, making it unable to detect abnormal mechanical noise; the fixed evaporation rate of the ant colony algorithm easily leads to local optima and slow convergence; and the allocation of communication and computing resources is unreasonable.

[0004] Chinese patent application number 202511171314.2 discloses a robot dynamic risk assessment and decision-making system and method based on multimodal perception. By integrating multi-source sensor data such as vision, acoustics, mechanics, and position, the system can comprehensively capture various risk factors in complex dynamic environments, thus overcoming the shortcomings of traditional single-sensor systems with insufficient perception dimensions. It is particularly suitable for regions with varying terrain and climate conditions. However, it lacks a dynamic sensor role switching mechanism, resulting in insufficient single-point failure response capability; adjacent sensors lack cross-validation, making it difficult to correct systemic biases; the dynamic trade-off between edge computing resource constraints and model accuracy is missing; and the cloud mainly uses federated learning technology to aggregate the operating data of multiple robots to optimize the global risk assessment model without a specific mechanism to respond to individual triggering events.

[0005] In summary, existing indoor service robots suffer from several problems in complex indoor environments, including insufficient utilization of multi-sensor data, significant environmental impact on the reliability of perception results, unreasonable allocation of communication and computing resources, and fragmentation of the perception-communication-computation-execution link.

[0006] Therefore, there is an urgent need for a comprehensive hierarchical decision-making system, method, and robot that can comprehensively balance the improvement of recognition accuracy, real-time performance, robustness, and optimization of computing power consumption and sensor energy consumption. Summary of the Invention

[0007] The purpose of this invention is to provide a comprehensive hierarchical decision-making system, method, and robot that can comprehensively balance and improve recognition accuracy, real-time performance, robustness, and optimize computing power consumption and sensor energy consumption.

[0008] In a first aspect, the present invention provides a comprehensive hierarchical decision-making system based on multimodal perception, comprising a remote processing center and: A multimodal sensing component, comprising a first sensor group that is normally open and a backup sensor group to be activated, wherein the backup sensor group comprises a second sensor group and a third sensor group; The proximal processing center includes: The perception uncertainty assessment unit is used to determine whether to activate at least one mode of sensor in the second sensor group to obtain second original perception data based on the first original perception data obtained from the first sensor group; and to calculate the perception result, including single-mode confidence and global perception uncertainty, based on the first original perception data and the second original perception data. The integrated decision-making unit determines the execution mode of the generated control command based on the perception results. The execution mode includes either processing the command independently by the near-end processing center or requesting the remote processing center to process it collaboratively.

[0009] Preferably, the first sensor group includes an infrared sensor and a millimeter-wave radar sensor; the second sensor group includes one or more of a vision sensor, a microphone sensor, and an air sensor; and the third sensor group includes one or more of a laser rangefinder, an ultrasonic sensor, an inertial sensor, or a door magnetic sensor. The proximal processing center also includes: The preprocessing and feature extraction unit is used to filter, denoise, normalize, segment, encode and extract features from the raw sensing data collected by the multimodal sensing components to form multimodal feature data. The time synchronization and spatial alignment unit is used to perform time synchronization, spatial mapping, and target association on multimodal data with different sampling frequencies, timestamps, and coordinate systems.

[0010] Preferably, the integrated hierarchical decision-making system further includes: The communication scheduling unit is used to determine whether to activate at least one modality sensor in the third sensor group based on the perception results output by the integrated decision unit; and is used to selectively transmit one or more of the following data to the remote processing center based on the perception results: raw data fragments, multimodal feature data, and semantic results processed by the integrated decision unit. The execution control module is used to receive control commands output by the near-end processing center and execute corresponding actions; The remote processing center is used to respond to the data uploaded by the communication scheduling unit and coordinate with the near-end processing center to process the sensed events.

[0011] Preferably, when the execution mode is to request the remote processing center to cooperate in processing, the integrated decision-making unit generates the final control instruction based on the processing result returned by the remote processing center and in combination with the current local perception state, local security constraints and execution context.

[0012] In a second aspect, the present invention provides a comprehensive hierarchical decision-making method based on multimodal perception, applicable to the comprehensive hierarchical decision-making system described in any one of the first aspects, wherein the comprehensive hierarchical decision-making method includes: This puts the first sensor group in a normally open, active state, while the standby sensor group is in a dormant state awaiting activation. Acquire first raw sensing data from the first sensor group, determine whether to activate at least one modality sensor in the second sensor group to obtain second raw sensing data; based on the first raw sensing data and the second raw sensing data, calculate the sensing results including single-modal confidence and global sensing uncertainty. Based on the perception results, the execution mode of the generated control command is determined, and the execution mode includes the near-end processing center processing alone or requesting the far-end processing center to process in cooperation.

[0013] Preferably, the modes of transmitting data to a remote processing center include: When the perceived uncertainty is below the first threshold, the first mode of uploading structured semantic results or state summary information is adopted. When the perception uncertainty is between the first threshold and the second threshold, a second mode is adopted, in which intermediate multimodal feature data extracted from the original perception data is uploaded to the remote processing center for collaborative processing. When the perceived uncertainty is higher than the second threshold, or when high-risk events such as falls, intrusions, calls for help, or smoke are detected, the third mode is used. Specifically, the third mode involves: activating the third sensor group and acquiring third raw sensing data; and uploading a portion of the third raw sensing data, and at least one of the partial raw first raw sensing data and the partial second raw sensing data to a remote processing center for collaborative processing.

[0014] Preferably, the data transmission mode also implements an adaptive communication strategy based on link bandwidth, link latency, and packet loss rate, enabling the integrated hierarchical decision system to maintain minimum available closed-loop capability under link fluctuation conditions.

[0015] Thirdly, the present invention provides a robot including a memory and a processor, wherein the memory stores a program executable on the processor, and when the program is executed by the processor, the robot performs the method described in any of the second aspects.

[0016] Fourthly, the present invention provides a readable storage medium storing a program, which, when executed, implements the method described in any of the second aspects.

[0017] Fifthly, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any of the second aspects.

[0018] The beneficial effects of this invention are as follows: (i) Improve the accuracy of event identification and reduce the risk of misjudgment By integrating multimodal sensor information to dynamically assess perception uncertainty, optimizing uploaded data quality and collaborative computing modes, the event identification accuracy is higher than existing solutions while significantly reducing communication load. At the same time, it effectively reduces false alarm rate and false negative rate, and improves the reliability of risk detection.

[0019] (ii) Significantly reduce bandwidth usage and transmission latency Based on a hierarchical data processing architecture, the edge prioritizes feature extraction and event pre-screening, uploading only local raw data or compressed feature information as needed, significantly reducing the amount of data uploaded per event, shortening the average response latency, and effectively alleviating network bandwidth pressure. This approach is suitable for bandwidth-constrained edge computing scenarios.

[0020] (III) Reduce computing power consumption and sensor energy consumption An on-demand wake-up mechanism based on high-risk event flags is adopted, which triggers the call of enhanced modal sensors and the uploading of local raw data only when the perception uncertainty exceeds the threshold or a high-risk event is detected. This avoids the continuous computing power overhead caused by full-time multimodal perception and significantly reduces the overall energy consumption and computing resource occupation of the system. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the architecture of a comprehensive hierarchical decision-making system based on multimodal perception in one embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating a comprehensive hierarchical decision-making method based on multimodal perception in another embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, but do not exclude other elements or objects.

[0024] To address the limitations of fully utilizing the rich information contained in millimeter-wave data and the inconvenience of real-time data processing, the deep learning-based vehicle imaging road information reconstruction method provided by this invention can fully utilize the information in millimeter-wave data to achieve high-precision real-time reconstruction of road information. The technical solutions of the embodiments of this invention are described below with reference to the accompanying drawings. In the description of the embodiments of this invention, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the invention. The singular expressions “a,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this invention, “at least one” and “one or more” refer to one or more (including two). The term “and / or” is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0025] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the invention. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," and "in still other embodiments" appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized. The term "connection" includes both direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0026] In embodiments of the present invention, "exemplarily" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design described as "exemplarily" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] Firstly, such as Figure 1 As shown, the present invention provides a comprehensive hierarchical decision-making system 1 based on multimodal perception, including a remote processing center 5 and: The multimodal sensing component 2 includes a first sensor group that is normally open and a backup sensor group that is to be activated. The backup sensor group includes a second sensor group and a third sensor group. Proximal processing center 3, which includes: The perception uncertainty evaluation unit 33 is used to determine whether to activate at least one mode of sensor in the second sensor group to obtain second original perception data based on the first original perception data obtained from the first sensor group; and to calculate the perception uncertainty based on the first and second original perception data obtained. The integrated decision-making unit 34 determines the execution mode of the generated control command based on the perception results. The execution mode includes processing by the near-end processing center alone or requesting the remote processing center to process in collaboration.

[0028] The comprehensive decision-making process includes: ① Whether the judgment is completed locally; ② Should we request collaborative analysis from the remote processing center 5? ③ Whether to directly issue control commands to the robot's control module 6.

[0029] By setting activation levels for sensors, the system's operating power consumption is reduced. After initially identifying the target object, a decision is made on whether to activate the second sensor group to collect more modal data. The perception uncertainty is calculated based on the collected raw perception data, and the comprehensive decision-making unit allocates computing resources reasonably based on the perception uncertainty. At the same time, a third sensor group is reserved for activation in case of corresponding response modes. This approach flexibly addresses the usage requirements of different scenarios and comprehensively optimizes one or more of the following objectives: improving recognition accuracy, real-time performance, robustness, and optimizing computing power consumption and sensor energy consumption.

[0030] In some embodiments, continue to refer to Figure 1 The near-end processing center 3 also includes: The preprocessing and feature extraction unit 31 is used to filter, denoise, normalize, segment, encode and extract features from the raw sensing data collected by the multimodal sensing components to form multimodal feature data. The time synchronization and spatial alignment unit 32 is used to perform time synchronization, spatial mapping and target association on multimodal data with different sampling frequencies, timestamps and coordinate systems.

[0031] The first sensor group includes an infrared sensor 21 and a millimeter-wave radar sensor 22; the second sensor group includes one or more of a vision sensor 23, a microphone sensor 24, and an air sensor 25; the third sensor group 26 includes one or more of a laser rangefinder, an ultrasonic sensor, an inertial sensor, or a door magnetic sensor.

[0032] In some embodiments, the integrated hierarchical decision-making system further includes: The communication scheduling unit 4 is used to determine whether to activate at least one modality sensor in the third sensor group based on the perception results output by the integrated decision unit; and is used to selectively transmit one or more of the following data to the remote processing center 5: raw data fragments, multimodal feature data, and semantic results processed by the integrated decision unit, based on the perception results. The raw data consists of data collected from sensors in the multimodal sensing component, such as heat transfer image segments from infrared sensors, millimeter-wave trajectory segments from millimeter-wave radar sensors, and local image segments from visual sensors. Multimodal feature data is generated by filtering, denoising, normalizing, segmenting, encoding, and extracting features from the raw sensing data collected by the multimodal sensing component. For example, target distance, radial velocity, azimuth angle, point features, and trajectory continuity features are extracted from millimeter-wave data; thermal area, heat source intensity, thermal centroid location, and temperature difference features are extracted from infrared data; target bounding boxes, category probabilities, pose features, and scene semantic features are extracted from visual data; audio event categories and confidence levels are extracted from audio data; and pollution levels, concentration change trends, and abnormal threshold trigger states are extracted from air data.

[0033] The following is an example of the semantic results. The final output of the perceived uncertainty evaluation unit 33 can be represented in a structured data format as follows: O = {ym, Cm, Um} ∪ {Dconflict, Uglobal, Cglobal, Level} Where y_m represents the category judgment result of the m-th modality, Cm represents the single-modal confidence of the m-th modality, Um represents the single-modal uncertainty of the m-th modality, Dconflict represents the degree of conflict between the total modalities, Uglobal represents the globally perceived uncertainty, Cglobal represents the globally perceived confidence, and Level represents the uncertainty level information. The integrated decision-making unit 34 can perform subsequent local processing decisions, communication upload decisions, and cloud collaborative decisions based on the structured output results.

[0034] Figure 1 In one exemplary embodiment, the communication scheduling unit 4, as the core scheduling node of the system, not only undertakes data uploading and receiving functions but also performs joint scheduling among the perception layer, communication layer, and computing layer. Compared to the conventional approach where the communication module is merely used as a data forwarding interface, the communication scheduling unit 4 in this embodiment directly receives information from the local processor 3, including basic global perception uncertainty, enhanced perception global uncertainty, task level, high-risk event flag, local computing power utilization, link bandwidth, link latency, and packet loss rate. Based on this information, it selects the data uploading granularity, uploading priority, uploading timing, and whether to request the remote processing center 5 to participate in joint inference.

[0035] For example, the communication scheduling unit 4 switches between at least three communication coordination modes based on the current scenario: When the output results of the basic perception mode are consistent, the global perception uncertainty is lower than the first threshold, and there are no high-risk event markers, the communication scheduling unit 4 adopts a low communication load mode, uploading only structured semantic results or state summary information, without uploading intermediate features and raw data; when the global perception uncertainty is between the first and second thresholds, and there are no high-risk event markers, the communication scheduling unit 4 adopts a feature-level coordination mode, uploading intermediate features of millimeter wave and infrared, as well as the feature vector corresponding to the awakened enhanced perception mode, for the remote processing center 5 to perform joint confirmation; when the global perception uncertainty is higher than the second threshold, or the probability of high-risk events such as falls, intrusions, calls for help, smoke, and gas leaks exceeds a preset threshold, the communication scheduling unit 4 adopts a high-priority coordination mode, controlling the uploading of local raw data fragments, and requesting the remote processing center 5 to perform high-precision joint inference.

[0036] The communication scheduling unit 4 uploads data with at least three granularities: the first is semantic results, including target category, target location, risk level, uncertainty level, and suggested action; the second is intermediate features, including millimeter-wave target point features, infrared thermal zone features, visual embedding features, audio embedding features, air state features, or combinations thereof; and the third is local raw data, including local image fragments, audio fragments before and after events, millimeter-wave target trajectory fragments, infrared thermal image sequences, air sensing time series fragments, and local raw data from other sensors 26. The communication scheduling unit 4 switches between these three data granularities based on the current link status and task priority, thereby avoiding bandwidth waste and processing delays caused by constantly uploading all sensing data in full.

[0037] Secondly, the communication scheduling unit 4 is also used to control the wake-up strategy of the enhanced perception modality. For example, when the system is in the basic dual-modal operating mode, the communication scheduling unit 4 keeps the visual sensor 23, microphone sensor 24, air sensor 25, and other sensors 26 in a turned-off or low-power standby state; when increased conflict between basic modalities, increased global perception uncertainty, or the occurrence of high-risk events are detected, the communication scheduling unit 4 gradually wakes up the enhanced perception modality according to a preset priority, and reassesses whether it is necessary to continue uploading higher-granularity data based on the new perception results after wake-up. Thus, the communication scheduling unit 4 not only controls "what to transmit," but also "when to wake up whom," "when to upload to the cloud," and "when to stop uploading to the cloud."

[0038] In some other embodiments, the communication scheduling unit 4 also executes an adaptive communication strategy based on link bandwidth, link latency, and packet loss rate. For example, when the link bandwidth is higher than a preset bandwidth threshold and the latency is lower than a preset latency threshold, the communication scheduling unit 4 allows the uploading of partial raw data; when the link bandwidth decreases or the latency increases, the communication scheduling unit 4 preferentially degrades to an intermediate feature uploading mode; when the link quality further deteriorates, the communication scheduling unit 4 only retains the semantic result uploading, and the local processor 3 performs a conservative decision. In this way, the system can maintain minimum available closed-loop capability under link fluctuation conditions, avoiding the robot completely losing its judgment and execution capabilities due to network quality degradation.

[0039] Among them, the remote processing center 5 is used to respond to the data uploaded by the communication scheduling unit 4 and coordinate with the near-end processing center 3 to process the sensed events. For example Figure 1 In one robot example, the remote processing center 5 acts as the robot's cloud-based brain, receiving data uploaded by the communication scheduling unit 4 and performing highly complex joint analysis and collaborative reasoning. The remote processing center 5 can be deployed on an edge server, a robot swarm control platform, a local gateway server, or a cloud-based intelligent platform. The processing functions of the remote processing center 5 include, but are not limited to: ① Multimodal joint target recognition; ② Abnormal event identification; ③ Risk level determination; ④ Semantic understanding; ⑤ Path or action suggestion generation.

[0040] After completing the inference, the remote processing center 5 returns the identification results, risk level, suggested actions, control strategies or parameter update information to the communication scheduling unit 4, which then forwards them to the near-end processing center 3.

[0041] The execution control module 6 is used to receive control commands output by the near-end processing center 3 and execute corresponding actions; For example, the execution control module 6 is used to receive control commands output from the local near-end processing center 3 and drive the robot to perform corresponding actions. These actions include, but are not limited to: ① Obstacle avoidance; ② Deceleration; ③ Stop; ④ Path replanning; ⑤ • Proximity confirmation; ⑥ Voice prompts; ⑦ Report the alarm; ⑧ • Link lighting, fresh air system, access control system or other indoor equipment.

[0042] In some embodiments, when the execution mode is to request collaborative processing from the remote processing center 5, the integrated decision-making unit 34 generates the final control command based on the processing result returned by the remote processing center 5 and in combination with the current local awareness state, local security constraints, and execution context. Specifically, after receiving the joint inference result returned by the remote processing center 5, the communication scheduling unit 4 does not directly send it as the final control command to the execution control module 6, but first sends it back to the near-end processing center 3, which then generates the final control command in combination with the current local awareness state, local security constraints, and execution context. This allows the remote processing center 5 to primarily handle high-complexity analysis, the communication scheduling unit 4 to handle collaborative link control, and the near-end processing center 3 to handle final closed-loop control, thus balancing real-time performance, security, and robustness.

[0043] Secondly, refer to Figure 2 This invention provides a comprehensive hierarchical decision-making method based on multimodal perception, applicable to any comprehensive hierarchical decision-making system in the first aspect. The comprehensive hierarchical decision-making method includes: S1. This puts the first sensor group into an active state of normal operation, and the standby sensor group into a dormant state of waiting to be activated. S2. Obtain the first raw sensing data of the first sensor group, and decide whether to activate at least one mode sensor in the second sensor group to obtain the second raw sensing data; based on the obtained first raw sensing data and second raw sensing data, calculate the sensing uncertainty; S3. Based on the perception results, determine the execution mode of the generated control command. The execution mode includes processing by the near-end processing center alone or requesting the remote processing center to process in collaboration.

[0044] In some embodiments, the mode of transmitting data to a remote processing center includes: When the perceived uncertainty is below the first threshold, the first mode of uploading structured semantic results or state summary information is adopted. When the perceived uncertainty is between the first threshold and the second threshold, a second mode is adopted, in which intermediate multimodal feature data extracted from the original perceived data is uploaded to the remote processing center for collaborative processing. The third mode is used when the perceived uncertainty is higher than the second threshold, or when high-risk events such as falls, intrusions, calls for help, or smoke are detected.

[0045] Specifically, the third mode involves: activating the third sensor group and acquiring the third raw sensing data; and uploading one of the local third raw sensing data, and at least the local raw first raw sensing data and the local raw second raw sensing data to a remote processing center for collaborative processing.

[0046] In some embodiments, the data transmission mode also executes an adaptive communication strategy based on link bandwidth, link latency, and packet loss rate, enabling the integrated hierarchical decision system to maintain minimum available closed-loop capability under link fluctuation conditions.

[0047] Calculating perceived uncertainty can be achieved through neural networks, algorithms, or a combination of both.

[0048] Below is an algorithm-based example. Specifically, based on the confidence level of each modal sensing result, data quality, and the degree of intermodal conflict, single-modal uncertainty and global sensing uncertainty are output. The modalities include millimeter-wave mode, infrared mode, visual mode, microphone mode, and air mode. The sensing uncertainty evaluation unit first calculates the single-modal confidence level for each modality, then obtains the corresponding single-modal uncertainty based on the single-modal confidence level, and finally combines the intermodal conflict degree to obtain the global sensing uncertainty.

[0049] Wherein, for the m-th modality, its single-modality confidence is denoted as Cm, and the single-modality confidence is calculated by weighting the class probability output by the modality recognition model, the original data quality score, the spatiotemporal stability score, and the anomaly penalty term.

[0050] Cm = λ1Pm + λ2Qm + λ3Sm − λ4Rm In the above formula, Pm represents the class probability or target probability output by the modality recognition model, Qm represents the original data quality score of the modality, Sm represents the spatiotemporal stability score of the modality, Rm represents the anomaly penalty term of the modality, and λ1, λ2, λ3, and λ4 are weight coefficients that satisfy λ1 + λ2 + λ3 + λ4 = 1. After calculation, Cm is subjected to amplitude limiting to restrict its value range to [0,1].

[0051] The following is an example of single-mode confidence calculation for a specific sensor: (a) Millimeter wave mode The single-mode confidence score Cmm of the millimeter-wave mode is calculated based on the millimeter-wave recognition probability, echo signal-to-noise ratio score, target trajectory continuity score, and micro-motion feature score.

[0052] Cmm = a1Pmm + a2Qsnr + a3Qtrack + a4Qmicro Wherein, Pmm represents the target category probability output by the millimeter-wave network, Qsnr represents the millimeter-wave echo signal-to-noise ratio score, Qtrack represents the target continuous tracking stability score, Qmicro represents the score for breathing, slight swaying, or human micro-motion features, and a1, a2, a3, and a4 are weighting coefficients. For example, the signal-to-noise ratio score can be obtained based on the normalized result of the millimeter-wave signal-to-noise ratio between a preset minimum and maximum signal-to-noise ratio; the trajectory continuity score can be calculated based on the number of continuously tracked frames and the number of lost frames; and the micro-motion feature score can be obtained based on the relationship between micro-motion energy within the target area and a reference threshold.

[0053] (ii) Infrared mode The single-mode confidence score Cir for infrared modes is calculated based on infrared recognition probability, thermal contrast score, hot zone shape score, and hot zone stability score.

[0054] Cir = b1Pir + b2Qcontrast + b3Qshape + b4Qstable Wherein, Pir represents the category probability output by the infrared recognition model, Qcontrast represents the temperature difference contrast score between the hot zone and the background, Qshape represents the matching degree score between the hot zone area, aspect ratio, and the preset target template, Qstable represents the stability score of the hot zone centroid in consecutive frames, and b1, b2, b3, and b4 are weighting coefficients. For example, the thermal contrast score can be obtained by normalizing the difference between the average temperature of the hot zone and the average temperature of the background; the hot zone shape score can be calculated using an exponential function based on the difference between the hot zone area and a preset area reference value, and the difference between the aspect ratio and a reference aspect ratio; and the hot zone stability score can be calculated based on the jitter degree of the hot zone centroid in consecutive frames.

[0055] (iii) Visual modality The single-modal confidence score Cvis for the visual modality is calculated based on the visual recognition probability, image sharpness score, illumination score, and occlusion score.

[0056] Cvis = c1Pvis + c2Qclarity + c3Qlight + c4Qocc Where Pvis represents the target probability output by the visual detection or classification network, Qclarity represents the image sharpness score, Qlight represents the image lighting score, Qocc represents the occlusion score, and c1, c2, c3, and c4 are weighting coefficients. For example, the image sharpness score can be calculated based on the Laplacian variance of the image, the lighting score can be calculated based on the deviation between the average brightness and the target brightness range, and the occlusion score can be obtained based on the ratio of the effective visible area within the target bounding box to the total area of ​​the target bounding box.

[0057] (iv) Microphone mode The single-mode confidence score Cmic of the microphone modality is calculated based on the audio event recognition probability, audio signal-to-noise ratio score, and event duration matching score.

[0058] Cmic = d1Pmic + d2Qaudio + d3Qdur Where Pmic represents the event category probability output by the audio recognition model, Qaudio represents the audio signal-to-noise ratio score, Qdur represents the score of the degree of matching between the event duration and the preset event template, and d1, d2, and d3 are weighting coefficients.

[0059] (v) Air mode The single-mode confidence score Cair for the air mode is calculated based on the air anomaly detection probability, sensor drift stability score, and concentration change gradient score.

[0060] Cair = e1Pair + e2Qdrift + e3Qgrad Where Pair represents the category probability output by the air anomaly identification model, Qdrift represents the baseline drift stability score of the air sensor, Qgrad represents the significance score of the concentration change gradient, and e1, e2, and e3 are weighting coefficients.

[0061] The perceived uncertainty assessment unit obtains the corresponding single-mode uncertainty based on the single-mode confidence level. The single-mode uncertainty of the m-th mode, denoted as Um, can be expressed as: Um = 1 − Cm The value of Um ranges from [0,1], and the larger the value, the less reliable the current sensing result of the modality.

[0062] In other embodiments, to improve the accuracy of uncertainty characterization, the single-mode uncertainty can also be calculated by combining the information entropy of the output category distribution of that mode, expressed as: Um = α(1 − Cm) + (1 − α)H(Pm) / logK Where H(Pm) represents the information entropy of the output category distribution of the m-th modality, K represents the total number of categories, and α is the balance coefficient.

[0063] Furthermore, in some other embodiments, calculating perceived uncertainty also involves calculating the degree of intermodal conflict. Let Dij be the degree of conflict between the i-th and j-th modes, which can be calculated by weighting semantic conflict terms, spatial conflict terms, and temporal conflict terms, and expressed as: Dij = ρ1Dclass_ij + ρ2Dspace_ij + ρ3Dtime_ij Where Dclass_ij represents the semantic conflict degree between the two modalities in terms of target or event category, Dspace_ij represents the difference in spatial location estimation of the corresponding targets in the two modalities, Dtime_ij represents the temporal difference between the two modalities at the time of event occurrence, and ρ1, ρ2, and ρ3 are weighting coefficients. Furthermore, a weighted average of the conflict degrees of all modal pairs can be performed to obtain the total intermodal conflict degree Dconflict.

[0064] Dconflict = Σ(μijDij) / Σμij (i < j) Finally, the perceived uncertainty assessment unit outputs the global perceived uncertainty Uglobal based on the uncertainties of each single mode and the degree of conflict between modes, which can be expressed as: Uglobal = Σ(wmUm) + βDconflict Where M represents the number of modes participating in the current fusion, wm represents the fusion weight of the m-th mode, and β represents the inter-modal conflict penalty coefficient. The fusion weight wm can be a fixed value or adaptively calculated based on the current single-modal confidence level. For example, it can be expressed as: wm = Cm / ΣCk In addition, the perceived uncertainty assessment unit also outputs the global perceived confidence Cglobal, which can be expressed as: Cglobal = 1 − Uglobal Example

[0065] refer to Figure 1 It is understood that the indoor service robot 1 is used for nighttime inspections of elderly care rooms. The millimeter-wave radar sensor 21 and the infrared sensor 22 are continuously operational.

[0066] The preset rules of this embodiment are: The first threshold T1 is set to 0.30, and the second threshold T2 is set to 0.60. When Uglobal < T1, the current scenario is determined to be in a low uncertainty state, and the first mode of independent processing by the proximal processing center is adopted; when T1 ≤ Uglobal < T2, the current scenario is determined to be in a medium uncertainty state, and the comprehensive decision-making unit 34 adopts the second mode of uploading the intermediate multi-modal feature data extracted from the original sensing data to the remote processing center for collaborative processing; when Uglobal ≥ T2, the current scenario is determined to be in a high uncertainty state, and the comprehensive decision-making unit 34 adopts the third mode, that is, the mode of "local enhanced sensing + local raw data uploading + collaborative analysis by the remote processing center". Alternatively, even if Uglobal < T2, if the system detects high-risk events such as falls, break-ins, calls for help, smoke, abnormal gas leaks, etc., it can also be directly upgraded to the high-risk handling mode, that is, the third mode. In a specific embodiment, the system does not only switch modes based on the uncertainty level, but also combines the high-risk event flag H for joint determination. Let the high-risk event threshold be PH, which can preferably be set to 0.75. When the probability of at least one of the fall event probability, break-in event probability, call for help event probability, smoke event probability, and gas leakage event probability exceeds the high-risk event threshold, it is determined that H = 1; otherwise, it is determined that H = 0. The comprehensive decision-making unit 34 outputs a candidate mode based on Uglobal and H, and the communication scheduling unit 4 then performs scheduling accordingly. The response to the sensing event can be determined according to the following rules: ① When Uglobal < T1 and H = 0, the local direct processing mode is adopted; ② When T1 ≤ Uglobal < T2 and H = 0, the local processing plus intermediate feature uploading mode is adopted; ③ When Uglobal ≥ T2 or H = 1, the local enhanced sensing plus local raw data uploading and requesting collaborative analysis by the remote processing center 5 mode is adopted.

[0067] When the millimeter-wave radar sensor 21 detects an obvious height change and low-speed micro-movement in the bedside area, and the infrared sensor 22 detects that the hot zone has transferred from the bed surface to the ground area, it decides whether to activate the visual sensor and the microphone sensor in the second sensor group. The target category probability Pmm output by the millimeter-wave mode is 0.82, the echo signal-to-noise ratio score Qsnr = 0.80, and the trajectory continuity score Qtrack = 0.90; the category probability Pir output by the infrared mode is 0.76, the thermal contrast score Qcontrast = 0.85, the hot zone shape score Qshape = 0.72, and the hot zone stability score Qstable = 0.80; due to weak light at night in the visual mode, its category probability Pvis = 0.58, the clarity score Qclarity = 0.35, the illumination score Qlight = 0.30, and the occlusion score Qocc = 0.60; the event probability Pmic output by the microphone mode is 0.81, the audio signal-to-noise ratio score Qaudio = 0.70, and the duration matching score Qdur = 0.90.

[0068] Under the preset weights, the uncertainty of the millimeter-wave mode Umm = 0.16, the uncertainty of the infrared mode Uir = 0.222, the uncertainty of the visual mode Uvis = 0.49, and the uncertainty of the microphone mode Umic = 0.194 can be obtained respectively. Further combining the degree of conflict between modes Dconflict = 0.22, the global perception uncertainty Uglobal ≈ 0.287 is calculated. However, since the millimeter-wave detects a rapid decrease in the target height and the infrared detects that the hot zone has transferred from the bed surface to the ground area, the system calculates that the probability of a fall event Pfall≥0.75, and thus determines that the high-risk event flag H = 1. Although Uglobal < T1, the system still adopts the third mode according to the high-risk event priority rule. At this time, the communication scheduling unit 4 schedules the awakened visual sensor 23 and microphone sensor 24, obtains the image of the target area and the audio information before and after the event, and selects to upload the intermediate features or local raw data to the remote processing center 5 for collaborative confirmation according to the current link state. The remote processing center 5 returns the result of "suspected fall" and the action suggestions of "proximity confirmation + voice reminder + alarm reporting". The proximal processing center 3 receives and issues corresponding control instructions to the execution control module 6. Embodiment

[0069] Reference Figure 1Using the same preset rules as in Example 1, the system's basic bimodal mode operates first for suspected unauthorized intrusion scenarios at the entrance of elderly care rooms or hotel rooms at night. The millimeter-wave mode outputs a target category probability Pmm=0.55, an echo signal-to-noise ratio score Qsnr=0.52, and a trajectory continuity score Qtrack=0.48; the infrared mode outputs a category probability Pir=0.46, a thermal contrast score Qcontrast=0.42, a hot zone shape score Qshape=0.38, and a hot zone stability score Qstable=0.50. Under the influence of backlighting at the entrance, partial occlusion, and rapid passage of people, the system calculates that the conflict between the basic bimodal modes increases, further yielding a basic global perception uncertainty Uglobal≈0.66. Since Uglobal≥T2, the current scene is determined to be in a high uncertainty state.

[0070] At this time, the communication scheduling unit 4 controls the third mode to be adopted, waking up other sensors 26 to provide enhanced perception information. In this embodiment, when the other sensors 26 are door magnetic sensors, they can obtain the door opening and closing status. The communication scheduling unit 4 uploads the local raw data of the basic mode and the enhanced mode to the remote processing center 5 for joint reasoning. The remote processing center 5 returns the probability of unfamiliar intrusion, target category, and handling suggestions. Based on the returned results, the near-end processing center 3 issues control commands to the execution control module 6 to stop approaching, conduct voice inquiry, report an alarm, or link the access control. Example

[0071] Indoor service robot 1 is used in hotel delivery scenarios. Millimeter-wave radar sensor 21 detects a low-lying moving target in front, infrared sensor 22 detects only a weak heat source, and vision sensor 23, after being woken up, identifies the target as a sweeping robot. The perception uncertainty assessment unit 33 uses a neural network to determine that when the perception uncertainty is below a first threshold, the comprehensive decision-making unit 34 no longer requests deep involvement from the remote processing center 5, and the local near-end processing center 3 directly issues deceleration and detour control commands to the machine execution control module 6.

[0072] Thirdly, the present invention provides a robot including a memory and a processor. The memory stores a program that can run on the processor. When the program is executed by the processor, the robot performs the method described in any of the second aspects. For example, the processor may be implemented using a heterogeneous computing platform such as an MCU, embedded SoC, AI edge computing board, or FPGA + processor.

[0073] Fourthly, the present invention provides a readable storage medium storing a program, which, when executed, implements the method of any one of the second aspects.

[0074] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the second aspects.

[0075] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A comprehensive hierarchical decision-making system based on multimodal perception, characterized in that, Including remote processing centers and: A multimodal sensing component, comprising a first sensor group that is normally open and a backup sensor group to be activated, wherein the backup sensor group comprises a second sensor group and a third sensor group; The proximal processing center includes: The perception uncertainty assessment unit is used to determine whether to activate at least one mode of sensor in the second sensor group to obtain second original perception data based on the first original perception data obtained from the first sensor group; and to calculate the perception result, including single-mode confidence and global perception uncertainty, based on the first original perception data and the second original perception data. The integrated decision-making unit determines the execution mode of the generated control command based on the perception results. The execution mode includes either processing the command independently by the near-end processing center or requesting the remote processing center to process it collaboratively.

2. The comprehensive hierarchical decision-making system according to claim 1, characterized in that, The first sensor group includes an infrared sensor and a millimeter-wave radar sensor; the second sensor group includes one or more of a vision sensor, a microphone sensor, and an air sensor; the third sensor group includes one or more of a laser rangefinder, an ultrasonic sensor, an inertial sensor, or a door magnetic sensor. The proximal processing center also includes: The preprocessing and feature extraction unit is used to filter, denoise, normalize, segment, encode and extract features from the raw sensing data collected by the multimodal sensing components to form multimodal feature data. The time synchronization and spatial alignment unit is used to perform time synchronization, spatial mapping, and target association on multimodal data with different sampling frequencies, timestamps, and coordinate systems.

3. The comprehensive hierarchical decision-making system according to claim 1, characterized in that, The integrated hierarchical decision-making system also includes: The communication scheduling unit is used to determine whether to activate at least one modality sensor in the third sensor group based on the perception results output by the integrated decision unit; and is used to selectively transmit one or more of the following data to the remote processing center based on the perception results: raw data fragments, multimodal feature data, and semantic results processed by the integrated decision unit. The execution control module is used to receive control commands output by the near-end processing center and execute corresponding actions; The remote processing center is used to respond to the data uploaded by the communication scheduling unit and coordinate with the near-end processing center to process the sensed events.

4. The comprehensive hierarchical decision-making system according to claim 3, characterized in that, When the execution mode is to request the remote processing center to cooperate in processing, the integrated decision-making unit generates the final control instruction based on the processing result returned by the remote processing center and in combination with the current local perception state, local security constraints and execution context.

5. A comprehensive hierarchical decision-making method based on multimodal perception, characterized in that, Applied to the comprehensive hierarchical decision-making system as described in any one of claims 1-4, characterized in that the comprehensive hierarchical decision-making method includes: This puts the first sensor group in a normally open, active state, while the standby sensor group is in a dormant state awaiting activation. Acquire first raw sensing data from the first sensor group, determine whether to activate at least one modality sensor in the second sensor group to obtain second raw sensing data; based on the first raw sensing data and the second raw sensing data, calculate the sensing results including single-modal confidence and global sensing uncertainty. Based on the perception results, the execution mode of the generated control command is determined, and the execution mode includes the near-end processing center processing alone or requesting the far-end processing center to process in cooperation.

6. The comprehensive hierarchical decision-making method according to claim 5, characterized in that, The modes of transmitting data to remote processing centers include: When the perceived uncertainty is below the first threshold, the first mode of uploading structured semantic results or state summary information is adopted. When the perception uncertainty is between the first threshold and the second threshold, a second mode is adopted, in which intermediate multimodal feature data extracted from the original perception data is uploaded to the remote processing center for collaborative processing. When the perceived uncertainty is higher than the second threshold, or when high-risk events such as falls, intrusions, calls for help, or smoke are detected, the third mode is used. Specifically, the third mode involves: activating the third sensor group and acquiring third raw sensing data; and uploading a portion of the third raw sensing data, and at least one of the partial raw first raw sensing data and the partial second raw sensing data to a remote processing center for collaborative processing.

7. The comprehensive hierarchical decision-making method according to claim 6, characterized in that, The data transmission mode also implements an adaptive communication strategy based on link bandwidth, link latency, and packet loss rate, enabling the comprehensive hierarchical decision-making system to maintain minimum available closed-loop capability under link fluctuation conditions.

8. A robot, characterized in that, It includes a memory and a processor, wherein the memory stores a program that can run on the processor, and when the program is executed by the processor, causes the robot to perform the method of any one of claims 5-7.

9. A readable storage medium storing a program, characterized in that, When the program is executed, it implements the method of any one of claims 5-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 5-7.

Images