Autonomous collaborative decision-making method, system and medium for industrial robots based on multimodal perception

Abstract

The present application provides an autonomous collaborative decision-making method, system and medium for industrial robots based on multimodal perception, which belongs to the field of intelligent control technology for industrial robots. The method includes: collecting visual, tactile and auditory information, performing cross-modal spatiotemporal alignment to eliminate spatiotemporal differences in data, combining a dynamic weight adjustment mechanism and an attention calculation model to achieve multimodal feature fusion, and generating a joint decision-making strategy through a deep learning optimization model. The system dynamically allocates sensor weights according to the task type, and introduces a multi-objective optimization mechanism for energy consumption, accuracy and safety, while setting fault-tolerant rules to automatically restore weights or recalibrate sensors. The present invention solves the problems of rigid data fusion and single optimization dimension in traditional multimodal decision-making, and significantly improves the accuracy, response speed and environmental adaptability of collaborative operations of industrial robots.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology for industrial robots, and more specifically, to a method, system, and medium for autonomous collaborative decision-making of industrial robots based on multimodal perception. Background Art

[0002] Existing technologies in the field of multimodal perception for industrial robots suffer from significant shortcomings. First, multi-source data fusion is insufficient. Differences in the acquisition frequencies and spatial coordinate systems of visual, tactile, and auditory sensors lead to temporal and spatial misalignment of data, severely impacting decision accuracy. Second, existing methods often employ fixed weight allocation strategies, failing to dynamically adjust the priority of each sensor based on the task type, resulting in inefficient resource utilization. Furthermore, traditional optimization models often focus on a single metric, neglecting multi-objective collaborative optimization such as energy consumption and safety, making them difficult to adapt to complex and ever-changing industrial scenarios. Finally, the system suffers from weak fault tolerance and lacks a real-time verification mechanism for decision-making, preventing rapid backtracking or sensor calibration in the event of task failure. These issues collectively hinder the performance improvement of autonomous collaborative decision-making for industrial robots. As robotic application scenarios become increasingly complex, dynamic, and diverse, single-modal perception is unable to meet the needs of autonomous collaborative decision-making. There is an urgent need for methods and systems that can integrate multimodal information to achieve more accurate and reliable decision-making. Summary of the Invention

[0003] The purpose of this application is to provide an autonomous collaborative decision-making method, system, and medium for industrial robots based on multimodal perception. By synchronously collecting visual, tactile, and auditory information, time synchronization and coordinate conversion technologies are used to achieve cross-modal spatiotemporal alignment and solve the problem of data misalignment. To address the weight distribution problem, a task-driven dynamic weight adjustment mechanism is introduced, combined with an attention calculation model to achieve adaptive generation of feature fusion parameters. Deep learning is used to construct a multi-objective optimization framework, which integrates indicators such as energy consumption, accuracy, and safety to generate joint decisions and iteratively optimize. At the same time, a two-layer fault-tolerant mechanism is set up to ensure system stability and promote the intelligent upgrade of industrial robots.

[0004] This application provides an autonomous collaborative decision-making method for industrial robots based on multimodal perception, comprising the following steps:

[0005] Collect sensory information within a preset period of the autonomous collaborative system, including visual information, tactile information, and auditory information;

[0006] Performing cross-modal spatiotemporal alignment of the visual information, tactile information, and auditory information using a preset method to generate first perceptual feature information;

[0007] Extracting visual feature data, tactile feature data, and auditory feature data through a preset mechanism based on the first perception feature information and dynamically adjusting weights to generate feature fusion parameters;

[0008] Generate a multimodal joint decision-making strategy based on the feature fusion parameters, and iteratively optimize the decision parameters through a preset optimization model;

[0009] Obtain decision parameter optimization effect data and compare it with the preset effect threshold to determine whether the optimization effect meets the requirements.

[0010] Among them, in the industrial robot autonomous collaborative decision-making method based on multimodal perception described in this application, the cross-modal spatiotemporal alignment includes:

[0011] The time synchronization module accurately aligns the images captured by the camera, the force changes detected by the tactile sensor, and the sounds collected by the microphone according to the acquisition time.

[0012] The position information detected by different sensors is unified into the three-dimensional space coordinate system of the robot through coordinate transformation.

[0013] Among them, in the industrial robot autonomous collaborative decision-making method based on multimodal perception described in this application, the dynamic adjustment of weights includes:

[0014] The importance ratio of each sensor is automatically set according to the current task type. In the fine assembly task, the weight of the tactile sensor is set to the first weight ratio.

[0015] When it is necessary to detect machine abnormalities, the weight of the sound sensor is increased to the second weight ratio, and the weight of the camera image is reduced to the third weight ratio.

[0016] Among them, in the industrial robot autonomous collaborative decision-making method based on multimodal perception described in this application, the generation process of the feature fusion parameters is:

[0017] Analyze the degree of correlation between visual, tactile, and auditory data through attention computational models;

[0018] The comprehensive impact value of each sensor is automatically assigned according to the degree of association to ensure that the sum of the impact values ​​of vision, touch, and hearing is 1.

[0019] Among them, in the autonomous collaborative decision-making method of industrial robots based on multimodal perception described in this application, the preset optimization model adopts a deep learning model, wherein:

[0020] The robot's movement status, surrounding environment conditions, and mission objectives together form the basis for decision-making;

[0021] The output control instructions include movement route parameters, operation force parameters, and collaboration timing parameters;

[0022] Continuously optimize decision-making results through comprehensive scoring of energy consumption, operational accuracy, and safety.

[0023] Among them, in the industrial robot autonomous collaborative decision-making method based on multimodal perception described in this application, when comparing the optimization effects:

[0024] When the task completion accuracy is lower than the preset first completion rate or the response delay exceeds the preset time, the previous weight setting is automatically restored;

[0025] If the decision fails to meet the target for three consecutive times, all sensors are recalibrated and the data fusion rules are updated.

[0026] In a second aspect, the present application provides an industrial robot autonomous collaborative decision-making system based on multimodal perception, the system comprising: a memory and a processor, the memory comprising a program of an industrial robot autonomous collaborative decision-making method based on multimodal perception, and the program of the industrial robot autonomous collaborative decision-making method based on multimodal perception, when executed by the processor, implements the following steps:

[0027] Collect sensory information within a preset period of the autonomous collaborative system, including visual information, tactile information, and auditory information;

[0028] Performing cross-modal spatiotemporal alignment of the visual information, tactile information, and auditory information using a preset method to generate first perceptual feature information;

[0029] Extracting visual feature data, tactile feature data, and auditory feature data through a preset mechanism based on the first perception feature information and dynamically adjusting weights to generate feature fusion parameters;

[0030] Generate a multimodal joint decision-making strategy based on the feature fusion parameters, and iteratively optimize the decision parameters through a preset optimization model;

[0031] Obtain decision parameter optimization effect data and compare it with the preset effect threshold to determine whether the optimization effect meets the requirements.

[0032] In the industrial robot autonomous collaborative decision-making system based on multimodal perception described in this application, the cross-modal spatiotemporal alignment includes:

[0033] The time synchronization module accurately aligns the images captured by the camera, the force changes detected by the tactile sensor, and the sounds collected by the microphone according to the acquisition time.

[0034] The position information detected by different sensors is unified into the three-dimensional space coordinate system of the robot through coordinate transformation.

[0035] In the industrial robot autonomous collaborative decision-making system based on multimodal perception described in this application, the dynamic adjustment of weights includes:

[0036] The importance ratio of each sensor is automatically set according to the current task type. In the fine assembly task, the weight of the tactile sensor is set to the first weight ratio.

[0037] When it is necessary to detect machine abnormalities, the weight of the sound sensor is increased to the second weight ratio, and the weight of the camera image is reduced to the third weight ratio.

[0038] On the third aspect, the present application also provides a computer-readable storage medium, which includes an industrial robot autonomous collaborative decision-making method program based on multimodal perception. When the industrial robot autonomous collaborative decision-making method program based on multimodal perception is executed by a processor, the steps of the industrial robot autonomous collaborative decision-making method based on multimodal perception as described in any one of the above items are implemented.

[0039] As can be seen from the above, the method, system, and medium for autonomous collaborative decision-making based on multimodal perception for industrial robots provided in the embodiments of the present application fundamentally solve the problem of data spatiotemporal misalignment by synchronously collecting visual, tactile, and auditory information and utilizing a time synchronization module and coordinate transformation technology to achieve cross-modal spatiotemporal alignment. To address the rigidity of weight allocation, the system introduces a task-driven dynamic weight adjustment mechanism, prioritizing tactile data in fine assembly tasks and enhancing the weight of auditory sensors during anomaly detection. At the same time, it combines an attention calculation model to analyze multimodal correlations and achieve adaptive generation of feature fusion parameters. To address the problem of a single optimization dimension, a deep learning model is used to construct a multi-objective optimization framework. This framework comprehensively considers energy consumption, accuracy, and safety indicators to generate joint decisions on movement routes, operation force, and collaboration timing, and continuously improves decision parameters through iterative optimization. The system also innovatively implements a two-layer fault-tolerance mechanism: when the task accuracy falls below a threshold or the response times out, it automatically falls back to the historical weight configuration; after three consecutive decision failures, it triggers a full sensor calibration and rule update, significantly improving system stability and effectively promoting the intelligent upgrade of industrial robots.

[0040] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flowchart of an autonomous collaborative decision-making method for industrial robots based on multimodal perception provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of cross-modal spatiotemporal alignment of an autonomous collaborative decision-making method for industrial robots based on multimodal perception provided in an embodiment of the present application;

[0044] Figure 3 A flowchart of dynamic weight adjustment of the autonomous collaborative decision-making method for industrial robots based on multimodal perception provided in an embodiment of the present application;

[0045] Figure 4 A flowchart of generating feature fusion parameters for an autonomous collaborative decision-making method for industrial robots based on multimodal perception provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0047] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0048] Please refer to Figure 1 , Figure 1 This is a flow chart of an autonomous collaborative decision-making method for industrial robots based on multimodal perception in some embodiments of the present application. This autonomous collaborative decision-making method for industrial robots based on multimodal perception is used in terminal devices, such as computers, mobile phones, etc. This autonomous collaborative decision-making method for industrial robots based on multimodal perception includes the following steps:

[0049] S101, collecting sensory information within a preset period of the autonomous collaborative system, including visual information, tactile information, and auditory information;

[0050] S102, performing cross-modal spatiotemporal alignment of the visual information, tactile information, and auditory information using a preset method to generate first perceptual feature information;

[0051] S103: extracting visual feature data, tactile feature data, and auditory feature data according to the first perception feature information through a preset mechanism and dynamically adjusting weights to generate feature fusion parameters;

[0052] S104: generating a multimodal joint decision-making strategy based on the feature fusion parameters, and iteratively optimizing the decision parameters through a preset optimization model;

[0053] S105: Obtain decision parameter optimization effect data and compare it with a preset effect threshold to determine whether the optimization effect meets the requirements.

[0054] First, multi-source sensors synchronously collect multimodal perception information within a preset period, including visual information (such as workpiece pose and environmental images captured by high-resolution cameras), tactile information (such as end-effector contact force / torque detected by six-dimensional force sensors), and auditory information (such as mechanical transmission noise audio captured by directional microphones). Next, cross-modal spatiotemporal alignment technology is used to uniformly process heterogeneous data, achieving timestamp synchronization of sensor data and eliminating timing deviations caused by sampling frequency differences. Simultaneously, based on the hand-eye calibration matrix and the robot kinematic model, the visual image coordinate system and the tactile sensor coordinate system are transformed into the robot base coordinate system, constructing a three-dimensional spatial data mapping relationship, resolving spatial reference inconsistencies, and generating spatiotemporally consistent first-order perception feature information. Subsequently, a dynamic weight adaptive adjustment mechanism is introduced, using a deep neural network to extract high-order features from each modality. Combined with the real-time task type (such as assembly, inspection, and exception handling) and scene complexity (light intensity and noise level), an attention mechanism is used to calculate cross-modal correlation weights, ultimately generating optimized feature fusion parameters. Based on the fusion parameters, a multi-objective joint decision-making model is constructed. A deep reinforcement learning framework is used to encode the robot's motion state, environmental constraints, and task objectives into a state space. The model then outputs a decision instruction set containing path planning parameters, operation force thresholds, and collaboration timing. This is then iteratively optimized using a multi-objective optimization function until the decision score reaches convergence. Finally, the system monitors the performance of decision execution (such as task completion time, positioning deviation, and energy consumption data) in real time and compares it against preset thresholds (such as an accuracy threshold of ±0.1mm and an upper limit on response delay of 200ms). If the optimization effect falls short, a weight backtracking mechanism is triggered or an online sensor calibration process is initiated, forming a closed-loop control loop of perception, decision-making, and verification, ensuring the system's robustness and adaptability in dynamic industrial environments.

[0055] Please refer to Figure 2 , Figure 2 This is a flow chart of cross-modal spatiotemporal alignment of an autonomous collaborative decision-making method for industrial robots based on multimodal perception in some embodiments of the present application. According to an embodiment of the present invention, the cross-modal spatiotemporal alignment includes:

[0056] S201, accurately aligning the images captured by the camera, the force changes detected by the tactile sensor, and the sounds collected by the microphone according to the acquisition time through a time synchronization module;

[0057] S202: Unify the position information detected by different sensors into the three-dimensional space coordinate system of the robot through coordinate conversion.

[0058] At the temporal synchronization level, a hardware clock synchronization module using a precise time protocol aligns the data acquisition timestamps of the camera, tactile sensors (such as six-dimensional force sensors), and microphones. Interpolation compensation algorithms eliminate timing deviations caused by differences in sensor sampling frequencies, ensuring strict synchronization of visual images, tactile force feedback, and collision sound data for the same event (e.g., the moment a robotic arm contacts a workpiece). At the spatial alignment level, a multi-level coordinate transformation model is established. First, the camera extrinsic matrix is ​​obtained through hand-eye calibration, and image pixel coordinates are mapped to the robot's base coordinate system. Simultaneously, the tactile sensor's mounting pose matrix is ​​used to transform the detected contact force / torque from the tool coordinate system to the base coordinate system. For audio localization, the direction of arrival of the sound source is estimated using the microphone array, and the three-dimensional coordinates of the sound source are calculated using the robot's kinematic model. Finally, all sensor data are unified into the three-dimensional spatial coordinate system used for robot motion control through a homogeneous coordinate transformation, forming a temporally and spatially consistent multimodal perception benchmark. This dual alignment mechanism completely eliminates the data fusion distortion caused by differences in temporal and spatial benchmarks in traditional multi-sensor systems, providing high-precision input for subsequent dynamic weight allocation and collaborative decision-making.

[0059] Please refer to Figure 3 , Figure 3 This is a flowchart of the dynamic weight adjustment of the industrial robot autonomous collaborative decision-making method based on multimodal perception in some embodiments of the present application. According to an embodiment of the present invention, the dynamic weight adjustment includes:

[0060] S301, automatically setting the importance ratio of each sensor according to the current task type, and setting the weight of the tactile sensor in the fine assembly task to a first weight ratio;

[0061] S302: When it is necessary to detect machine abnormality, the weight of the sound sensor is increased to the second weight ratio, and the weight of the camera image is reduced to the third weight ratio.

[0062] The dynamic weight allocation mechanism enables intelligent allocation of multimodal perception resources through task type identification and sensor performance evaluation. The system's built-in task semantic parsing module extracts keywords and classifies intent from task instructions based on natural language processing. It also generates task type labels based on scene perception data (e.g., workpiece dimensional tolerance ±0.05mm, ambient noise level >75dB). For precision assembly tasks, the system utilizes a pre-set priority rule library to increase the weight of tactile sensors to the first level (e.g., 60%-70%), prioritizes six-dimensional force sensor data for contact force feedback, and reduces the weight of vision to an auxiliary level, retaining only critical posture verification. When switching to device anomaly diagnosis mode, the frequency domain analysis module identifies acoustic signal characteristics, dynamically increasing the weight of the sound sensor to the second level (e.g., 65%-75%). Wavelet noise reduction and MFCC feature extraction are used to enhance voiceprint recognition accuracy, while reducing the weight of vision to the third level (e.g., 10%-15%) to reduce computational overhead. The weight allocation process is continuously optimized through a reinforcement learning model, and the weight boundary values ​​are dynamically corrected based on historical operation data (such as assembly success rate and anomaly detection accuracy) to ensure that sensor resource allocation is accurately matched to task requirements. Compared with traditional fixed weight schemes, it can improve task execution efficiency.

[0063] Please refer to Figure 4 , Figure 4 This is a flow chart of generating feature fusion parameters for an autonomous collaborative decision-making method for industrial robots based on multimodal perception in some embodiments of the present application. According to an embodiment of the present invention, the process of generating the feature fusion parameters is as follows:

[0064] S401, analyzing the degree of correlation between visual, tactile, and auditory data through an attention calculation model;

[0065] S402. Automatically assign the comprehensive impact value of each sensor according to the degree of association to ensure that the sum of the impact values ​​of vision, touch, and hearing is 1.

[0066] Among them, the multimodal correlation analysis and weight distribution method of the present invention realizes cross-modal data collaborative optimization based on the improved multi-head attention mechanism. Specifically, a cross-modal feature interaction matrix is ​​first constructed, the spatial features of visual data are extracted by CNN, the temporal mechanical features of tactile data are encoded by LSTM, and the frequency domain features of auditory data are extracted by Mel spectrum graph convolution network. Subsequently, the internal feature correlation of the single modality is modeled by the self-attention calculation module, and the cross-modal interaction weights of vision-touch, vision-auditory, and touch-auditory are calculated using the cross-attention branch to form a correlation scoring matrix. Based on this design, a dynamic normalized weight allocator is finally used, and the visual, tactile, and auditory weights are constrained by the softmax function to meet the sum of the influence values ​​of 1. Compared with the traditional weighted average strategy, this method can improve the efficiency of multimodal data fusion and avoid the problem of weight overfitting through preset normalization constraints.

[0067] According to an embodiment of the present invention, the preset optimization model adopts a deep learning model, wherein:

[0068] The robot's movement status, surrounding environment conditions, and mission objectives together form the basis for decision-making;

[0069] The output control instructions include movement route parameters, operation force parameters, and collaboration timing parameters;

[0070] Continuously optimize decision-making results through comprehensive scoring of energy consumption, operational accuracy, and safety.

[0071] The preset optimization model of the present invention is constructed based on a deep reinforcement learning framework and achieves intelligent decision-making through multi-dimensional state perception and multi-objective collaborative optimization. The model uses the robot's real-time movement state, surrounding environment information, and mission objectives as core decision-making criteria. The robot's movement state includes dynamic data such as its current position, speed, and posture; the surrounding environment includes information such as obstacle distribution, the location of other equipment, and environmental changes; and the mission objectives clearly define the specific operational requirements and expected results. Based on this comprehensive decision-making basis, the model outputs control instructions containing movement route parameters, operation force parameters, and collaboration timing parameters to precisely control the robot's actions. Furthermore, to ensure the scientific and efficient decision-making, the model constructs a comprehensive scoring system for energy consumption, operation accuracy, and safety. By performing a multi-dimensional quantitative evaluation of each decision result and using the evaluation feedback to optimize the decision parameters, continuous iterative improvement of the decision results is achieved, enabling industrial robots to make more energy-efficient, accurate, and safe decisions in complex and changing operating environments.

[0072] According to an embodiment of the present invention, when comparing the optimization effects:

[0073] When the task completion accuracy is lower than the preset first completion rate or the response delay exceeds the preset time, the previous weight setting is automatically restored;

[0074] If the decision fails to meet the target for three consecutive times, all sensors are recalibrated and the data fusion rules are updated.

[0075] Among them, when evaluating and comparing the optimization effects of decision parameters, the present invention sets a double guarantee mechanism to ensure the stability and reliability of the industrial robot decision-making system. When the task completion accuracy is lower than the preset first completion rate standard, or the system response delay exceeds the preset time threshold, it indicates that the current decision parameters and weight configuration fail to meet the task requirements. At this time, the system will automatically restore to the weight settings that have been proven effective in practice in the previous stage, quickly correct the decision deviation, and maintain operational stability. If the decision results for three consecutive times do not meet the preset standards, it is determined that the system may have sensor data deviation or data fusion rule failure problems. The system will trigger a comprehensive calibration program to recalibrate all sensors and update the data fusion rules at the same time. By deeply optimizing the perception and decision-making links, the problem of substandard decision-making is solved from the root, thereby constructing a dynamic feedback and self-repair mechanism to effectively improve the fault tolerance and continuous operation efficiency of the industrial robot autonomous collaborative decision-making system under complex working conditions.

[0076] According to an embodiment of the present invention, the comprehensive score optimization process further includes:

[0077] Real-time monitoring of ambient noise levels and light intensity. When the ambient noise exceeds the preset noise threshold, the auditory sensor's frequency band filtering strength is dynamically increased and the visual weight is reduced.

[0078] When the light intensity is lower than the preset intensity threshold, the infrared fill light module is activated and the depth of the visual feature extraction network is increased, while the tactile weight is increased to compensate for the impact of the decline in visual perception ability.

[0079] During the comprehensive score optimization process, the system uses the environmental perception module to monitor ambient noise levels and light intensity parameters in real time. When the ambient noise level exceeds a preset threshold (e.g., 70dB), the system automatically triggers a dynamic noise reduction mechanism. First, by increasing the frequency filtering strength of the auditory sensor, the system enhances the signal processing of specific noise frequency bands to improve the accuracy of abnormal sound recognition. Second, based on a multimodal weight allocation strategy, the system simultaneously reduces the decision weight of the visual sensor to mitigate the impact of visual information misjudgment caused by noise interference. When the detected light intensity falls below a preset threshold (e.g., 150 lux), the system initiates an environmental adaptive compensation mechanism. First, the infrared fill light module is activated to perform spectral compensation on the visual acquisition area. The depth of the convolutional layer of the visual feature extraction network is increased to improve feature resolution in low-light images. Simultaneously, the decision weight of the tactile sensor is dynamically increased to 1.2-1.5 times the baseline value, increasing the decision-making participation of tactile feedback to compensate for the information loss caused by reduced visual perception. This multimodal dynamic adjustment mechanism, based on adaptive environmental parameters, effectively ensures the robustness of perception and the reliability of decision-making for industrial robots in complex working conditions.

[0080] According to an embodiment of the present invention, the further embodiment includes:

[0081] Collecting the robot's working environment temperature during the dynamic weight adjustment process;

[0082] When the working environment temperature is greater than a first temperature threshold, the sampling frequency of the tactile sensor is increased to compensate for the material deformation error caused by the high temperature, and the visual weight is reduced to avoid image distortion interference;

[0083] When the working environment temperature is lower than the second temperature threshold, the low-frequency enhancement mode of the auditory sensor is enabled to improve the recognition of metallic noises, and the tactile weight is increased to enhance the force control accuracy at low temperatures.

[0084] During this dynamic weight adjustment process, the system collects real-time ambient temperature data from a temperature sensor integrated into the robot body. It then triggers a differentiated multimodal perception optimization strategy based on preset first and second temperature thresholds (e.g., 45°C and 5°C). When the ambient temperature is detected above 45°C, the system activates a high-temperature compensation mechanism, increasing the tactile sensor sampling frequency from a baseline value, for example, from 100Hz to 200Hz. This high-frequency data acquisition compensates for contact force measurement deviations caused by thermal expansion of the metal material. Simultaneously, the visual weight is reduced to minimize the impact of image distortion caused by high-temperature air waves on workpiece positioning accuracy. When the ambient temperature drops below 5°C, the system activates a low-temperature enhancement mode, applying low-frequency bandpass filtering to the auditory sensor to enhance feature extraction of abnormal metal fracture noise. Simultaneously, the tactile weight baseline is increased to offset the mechanical transmission hysteresis caused by low temperatures by increasing the weight of force feedback in decision-making. This temperature adaptive adjustment mechanism dynamically balances the performance degradation of multimodal sensors with scenario requirements, ensuring that the robot's comprehensive perception accuracy meets the requirements under extreme working conditions such as foundries (high temperature) and cold chain warehouses (low temperature), significantly outperforming traditional fixed parameter solutions.

[0085] According to an embodiment of the present invention, when generating a multimodal joint decision-making strategy, the method further includes:

[0086] If the completion rate of a robot's task is lower than the preset progress threshold, its path planning priority will be automatically increased by 2 levels, and other robots will avoid its work area;

[0087] When the target positions of multiple robots overlap, access rights are dynamically allocated based on the urgency of the tasks;

[0088] The remaining power of each robot is calculated in real time, and the movement speed weight of the robot whose power is lower than the preset power threshold is reduced to extend the collaboration cycle.

[0089] When generating a multimodal joint decision-making strategy, the system uses the task progress monitoring module to evaluate the task completion status of each robot in real time. If a robot's task completion rate is detected to be below a preset progress threshold (e.g., 30%), the system automatically triggers a lag compensation mechanism, raising the robot's path planning priority by two levels (for example, from the default priority 3 to priority 1) and marking its work area as a high-priority zone. The remaining robots then adjust their paths in real time based on the obstacle avoidance planning model, avoiding the lagging robot's work area at a safe distance. In the event of a conflict involving overlapping target positions of multiple robots, the system invokes the task urgency assessment model and dynamically allocates access rights based on a preset rule base (e.g., an urgency weight of 0.9 for hazardous materials handling tasks and 0.4 for general assembly tasks). The high-urgency robot receives priority access and uses the laser navigation module to define an exclusive path. At the same time, the system's integrated power monitoring unit calculates each robot's remaining battery life in real time. When it detects that the battery level falls below a preset threshold (e.g., 20%), it activates an energy-balancing strategy, lowering the baseline speed weight coefficient for the low-battery robot. Using a kinematic model, it replans its movement speed and extends the collaborative task cycle (e.g., adjusting the handling cycle from 5 minutes per time to 5.5-5.8 minutes per time). This prevents task interruptions while extending the robot cluster's continuous working time by over 2.3 hours. This multi-dimensional collaborative optimization mechanism improves the overall production capacity of the multi-robot assembly line, reduces task conflicts, and reduces low-battery downtime.

[0090] The present invention also discloses an industrial robot autonomous collaborative decision-making system based on multimodal perception, comprising a memory and a processor. The memory includes an industrial robot autonomous collaborative decision-making method program based on multimodal perception. When the industrial robot autonomous collaborative decision-making method program based on multimodal perception is executed by the processor, the following steps are implemented:

[0091] Collect sensory information within a preset period of the autonomous collaborative system, including visual information, tactile information, and auditory information;

[0092] Performing cross-modal spatiotemporal alignment of the visual information, tactile information, and auditory information using a preset method to generate first perceptual feature information;

[0093] Extracting visual feature data, tactile feature data, and auditory feature data through a preset mechanism based on the first perception feature information and dynamically adjusting weights to generate feature fusion parameters;

[0094] Generate a multimodal joint decision-making strategy based on the feature fusion parameters, and iteratively optimize the decision parameters through a preset optimization model;

[0095] Obtain decision parameter optimization effect data and compare it with the preset effect threshold to determine whether the optimization effect meets the requirements.

[0096] First, multi-source sensors synchronously collect multimodal perception information within a preset period, including visual information (such as workpiece pose and environmental images captured by high-resolution cameras), tactile information (such as end-effector contact force / torque detected by six-dimensional force sensors), and auditory information (such as mechanical transmission noise audio captured by directional microphones). Next, cross-modal spatiotemporal alignment technology is used to uniformly process heterogeneous data, achieving timestamp synchronization of sensor data and eliminating timing deviations caused by sampling frequency differences. Simultaneously, based on the hand-eye calibration matrix and the robot kinematic model, the visual image coordinate system and the tactile sensor coordinate system are transformed into the robot base coordinate system, constructing a three-dimensional spatial data mapping relationship, resolving spatial reference inconsistencies, and generating spatiotemporally consistent first-order perception feature information. Subsequently, a dynamic weight adaptive adjustment mechanism is introduced, using a deep neural network to extract high-order features from each modality. Combined with the real-time task type (such as assembly, inspection, and exception handling) and scene complexity (light intensity and noise level), an attention mechanism is used to calculate cross-modal correlation weights, ultimately generating optimized feature fusion parameters. Based on the fusion parameters, a multi-objective joint decision-making model is constructed. A deep reinforcement learning framework is used to encode the robot's motion state, environmental constraints, and task objectives into a state space. The model then outputs a decision instruction set containing path planning parameters, operation force thresholds, and collaboration timing. This is then iteratively optimized using a multi-objective optimization function until the decision score reaches convergence. Finally, the system monitors the performance of decision execution (such as task completion time, positioning deviation, and energy consumption data) in real time and compares it against preset thresholds (such as an accuracy threshold of ±0.1mm and an upper limit on response delay of 200ms). If the optimization effect falls short, a weight backtracking mechanism is triggered or an online sensor calibration process is initiated, forming a closed-loop control loop of perception, decision-making, and verification, ensuring the system's robustness and adaptability in dynamic industrial environments.

[0097] According to an embodiment of the present invention, the cross-modal spatiotemporal alignment includes:

[0098] The time synchronization module accurately aligns the images captured by the camera, the force changes detected by the tactile sensor, and the sounds collected by the microphone according to the acquisition time.

[0099] The position information detected by different sensors is unified into the three-dimensional space coordinate system of the robot through coordinate transformation.

[0100] At the temporal synchronization level, a hardware clock synchronization module using a precise time protocol aligns the data acquisition timestamps of the camera, tactile sensors (such as six-dimensional force sensors), and microphones. Interpolation compensation algorithms eliminate timing deviations caused by differences in sensor sampling frequencies, ensuring strict synchronization of visual images, tactile force feedback, and collision sound data for the same event (e.g., the moment a robotic arm contacts a workpiece). At the spatial alignment level, a multi-level coordinate transformation model is established. First, the camera extrinsic matrix is ​​obtained through hand-eye calibration, and image pixel coordinates are mapped to the robot's base coordinate system. Simultaneously, the tactile sensor's mounting pose matrix is ​​used to transform the detected contact force / torque from the tool coordinate system to the base coordinate system. For audio localization, the direction of arrival of the sound source is estimated using the microphone array, and the three-dimensional coordinates of the sound source are calculated using the robot's kinematic model. Finally, all sensor data are unified into the three-dimensional spatial coordinate system used for robot motion control through a homogeneous coordinate transformation, forming a temporally and spatially consistent multimodal perception benchmark. This dual alignment mechanism completely eliminates the data fusion distortion caused by differences in temporal and spatial benchmarks in traditional multi-sensor systems, providing high-precision input for subsequent dynamic weight allocation and collaborative decision-making.

[0101] According to an embodiment of the present invention, the dynamic weight adjustment includes:

[0102] The importance ratio of each sensor is automatically set according to the current task type. In the fine assembly task, the weight of the tactile sensor is set to the first weight ratio.

[0103] When it is necessary to detect machine abnormalities, the weight of the sound sensor is increased to the second weight ratio, and the weight of the camera image is reduced to the third weight ratio.

[0104] The dynamic weight allocation mechanism enables intelligent allocation of multimodal perception resources through task type identification and sensor performance evaluation. The system's built-in task semantic parsing module extracts keywords and classifies intent from task instructions based on natural language processing. It also generates task type labels based on scene perception data (e.g., workpiece dimensional tolerance ±0.05mm, ambient noise level >75dB). For precision assembly tasks, the system utilizes a pre-set priority rule library to increase the weight of tactile sensors to the first level (e.g., 60%-70%), prioritizes six-dimensional force sensor data for contact force feedback, and reduces the weight of vision to an auxiliary level, retaining only critical posture verification. When switching to device anomaly diagnosis mode, the frequency domain analysis module identifies acoustic signal characteristics, dynamically increasing the weight of the sound sensor to the second level (e.g., 65%-75%). Wavelet noise reduction and MFCC feature extraction are used to enhance voiceprint recognition accuracy, while reducing the weight of vision to the third level (e.g., 10%-15%) to reduce computational overhead. The weight allocation process is continuously optimized through a reinforcement learning model, and the weight boundary values ​​are dynamically corrected based on historical operation data (such as assembly success rate and anomaly detection accuracy) to ensure that sensor resource allocation is accurately matched to task requirements. Compared with traditional fixed weight schemes, it can improve task execution efficiency.

[0105] According to an embodiment of the present invention, the process of generating the feature fusion parameters is as follows:

[0106] Analyze the degree of correlation between visual, tactile, and auditory data through attention computational models;

[0107] The comprehensive impact value of each sensor is automatically assigned according to the degree of association to ensure that the sum of the impact values ​​of vision, touch, and hearing is 1.

[0108] Among them, the multimodal correlation analysis and weight distribution method of the present invention realizes cross-modal data collaborative optimization based on the improved multi-head attention mechanism. Specifically, a cross-modal feature interaction matrix is ​​first constructed, the spatial features of visual data are extracted by CNN, the temporal mechanical features of tactile data are encoded by LSTM, and the frequency domain features of auditory data are extracted by Mel spectrum graph convolution network. Subsequently, the internal feature correlation of the single modality is modeled by the self-attention calculation module, and the cross-modal interaction weights of vision-touch, vision-auditory, and touch-auditory are calculated using the cross-attention branch to form a correlation scoring matrix. Based on this design, a dynamic normalized weight allocator is finally used, and the visual, tactile, and auditory weights are constrained by the softmax function to meet the sum of the influence values ​​of 1. Compared with the traditional weighted average strategy, this method can improve the efficiency of multimodal data fusion and avoid the problem of weight overfitting through preset normalization constraints.

[0109] According to an embodiment of the present invention, the preset optimization model adopts a deep learning model, wherein:

[0110] The robot's movement status, surrounding environment conditions, and mission objectives together form the basis for decision-making;

[0111] The output control instructions include movement route parameters, operation force parameters, and collaboration timing parameters;

[0112] Continuously optimize decision-making results through comprehensive scoring of energy consumption, operational accuracy, and safety.

[0113] The preset optimization model of the present invention is constructed based on a deep reinforcement learning framework and achieves intelligent decision-making through multi-dimensional state perception and multi-objective collaborative optimization. The model uses the robot's real-time movement state, surrounding environment information, and mission objectives as core decision-making criteria. The robot's movement state includes dynamic data such as its current position, speed, and posture; the surrounding environment includes information such as obstacle distribution, the location of other equipment, and environmental changes; and the mission objectives clearly define the specific operational requirements and expected results. Based on this comprehensive decision-making basis, the model outputs control instructions containing movement route parameters, operation force parameters, and collaboration timing parameters to precisely control the robot's actions. Furthermore, to ensure the scientific and efficient decision-making, the model constructs a comprehensive scoring system for energy consumption, operation accuracy, and safety. By performing a multi-dimensional quantitative evaluation of each decision result and using the evaluation feedback to optimize the decision parameters, continuous iterative improvement of the decision results is achieved, enabling industrial robots to make more energy-efficient, accurate, and safe decisions in complex and changing operating environments.

[0114] According to an embodiment of the present invention, when comparing the optimization effects:

[0115] When the task completion accuracy is lower than the preset first completion rate or the response delay exceeds the preset time, the previous weight setting is automatically restored;

[0116] If the decision fails to meet the target for three consecutive times, all sensors are recalibrated and the data fusion rules are updated.

[0117] Among them, when evaluating and comparing the optimization effects of decision parameters, the present invention sets a double guarantee mechanism to ensure the stability and reliability of the industrial robot decision-making system. When the task completion accuracy is lower than the preset first completion rate standard, or the system response delay exceeds the preset time threshold, it indicates that the current decision parameters and weight configuration fail to meet the task requirements. At this time, the system will automatically restore to the weight settings that have been proven effective in practice in the previous stage, quickly correct the decision deviation, and maintain operational stability. If the decision results for three consecutive times do not meet the preset standards, it is determined that the system may have sensor data deviation or data fusion rule failure problems. The system will trigger a comprehensive calibration program to recalibrate all sensors and update the data fusion rules at the same time. By deeply optimizing the perception and decision-making links, the problem of substandard decision-making is solved from the root, thereby constructing a dynamic feedback and self-repair mechanism to effectively improve the fault tolerance and continuous operation efficiency of the industrial robot autonomous collaborative decision-making system under complex working conditions.

[0118] According to an embodiment of the present invention, the comprehensive score optimization process further includes:

[0119] Real-time monitoring of ambient noise levels and light intensity. When the ambient noise exceeds the preset noise threshold, the auditory sensor's frequency band filtering strength is dynamically increased and the visual weight is reduced.

[0120] When the light intensity is lower than the preset intensity threshold, the infrared fill light module is activated and the depth of the visual feature extraction network is increased, while the tactile weight is increased to compensate for the impact of the decline in visual perception ability.

[0121] During the comprehensive score optimization process, the system uses the environmental perception module to monitor ambient noise levels and light intensity parameters in real time. When the ambient noise level exceeds a preset threshold (e.g., 70dB), the system automatically triggers a dynamic noise reduction mechanism. First, by increasing the frequency filtering strength of the auditory sensor, the system enhances the signal processing of specific noise frequency bands to improve the accuracy of abnormal sound recognition. Second, based on a multimodal weight allocation strategy, the system simultaneously reduces the decision weight of the visual sensor to mitigate the impact of visual information misjudgment caused by noise interference. When the detected light intensity falls below a preset threshold (e.g., 150 lux), the system initiates an environmental adaptive compensation mechanism. First, the infrared fill light module is activated to perform spectral compensation on the visual acquisition area. The depth of the convolutional layer of the visual feature extraction network is increased to improve feature resolution in low-light images. Simultaneously, the decision weight of the tactile sensor is dynamically increased to 1.2-1.5 times the baseline value, increasing the decision-making participation of tactile feedback to compensate for the information loss caused by reduced visual perception. This multimodal dynamic adjustment mechanism, based on adaptive environmental parameters, effectively ensures the robustness of perception and the reliability of decision-making for industrial robots in complex working conditions.

[0122] According to an embodiment of the present invention, the further embodiment includes:

[0123] Collecting the robot's working environment temperature during the dynamic weight adjustment process;

[0124] When the working environment temperature is greater than a first temperature threshold, the sampling frequency of the tactile sensor is increased to compensate for the material deformation error caused by the high temperature, and the visual weight is reduced to avoid image distortion interference;

[0125] When the working environment temperature is lower than the second temperature threshold, the low-frequency enhancement mode of the auditory sensor is enabled to improve the recognition of metallic noises, and the tactile weight is increased to enhance the force control accuracy at low temperatures.

[0126] During this dynamic weight adjustment process, the system collects real-time ambient temperature data from a temperature sensor integrated into the robot body. It then triggers a differentiated multimodal perception optimization strategy based on preset first and second temperature thresholds (e.g., 45°C and 5°C). When the ambient temperature is detected above 45°C, the system activates a high-temperature compensation mechanism, increasing the tactile sensor sampling frequency from a baseline value, for example, from 100Hz to 200Hz. This high-frequency data acquisition compensates for contact force measurement deviations caused by thermal expansion of the metal material. Simultaneously, the visual weight is reduced to minimize the impact of image distortion caused by high-temperature air waves on workpiece positioning accuracy. When the ambient temperature drops below 5°C, the system activates a low-temperature enhancement mode, applying low-frequency bandpass filtering to the auditory sensor to enhance feature extraction of abnormal metal fracture noise. Simultaneously, the tactile weight baseline is increased to offset the mechanical transmission hysteresis caused by low temperatures by increasing the weight of force feedback in decision-making. This temperature adaptive adjustment mechanism dynamically balances the performance degradation of multimodal sensors with scenario requirements, ensuring that the robot's comprehensive perception accuracy meets the requirements under extreme working conditions such as foundries (high temperature) and cold chain warehouses (low temperature), significantly outperforming traditional fixed parameter solutions.

[0127] According to an embodiment of the present invention, when generating a multimodal joint decision-making strategy, the method further includes:

[0128] If the completion rate of a robot's task is lower than the preset progress threshold, its path planning priority will be automatically increased by 2 levels, and other robots will avoid its work area;

[0129] When the target positions of multiple robots overlap, access rights are dynamically allocated based on the urgency of the tasks;

[0130] The remaining power of each robot is calculated in real time, and the movement speed weight of the robot whose power is lower than the preset power threshold is reduced to extend the collaboration cycle.

[0131] When generating a multimodal joint decision-making strategy, the system uses the task progress monitoring module to evaluate the task completion status of each robot in real time. If a robot's task completion rate is detected to be below a preset progress threshold (e.g., 30%), the system automatically triggers a lag compensation mechanism, raising the robot's path planning priority by two levels (for example, from the default priority 3 to priority 1) and marking its work area as a high-priority zone. The remaining robots then adjust their paths in real time based on the obstacle avoidance planning model, avoiding the lagging robot's work area at a safe distance. In the event of a conflict involving overlapping target positions of multiple robots, the system invokes the task urgency assessment model and dynamically allocates access rights based on a preset rule base (e.g., an urgency weight of 0.9 for hazardous materials handling tasks and 0.4 for general assembly tasks). The high-urgency robot receives priority access and uses the laser navigation module to define an exclusive path. At the same time, the system's integrated power monitoring unit calculates each robot's remaining battery life in real time. When it detects that the battery level falls below a preset threshold (e.g., 20%), it activates an energy-balancing strategy, lowering the baseline speed weight coefficient for the low-battery robot. Using a kinematic model, it replans its movement speed and extends the collaborative task cycle (e.g., adjusting the handling cycle from 5 minutes per time to 5.5-5.8 minutes per time). This prevents task interruptions while extending the robot cluster's continuous working time by over 2.3 hours. This multi-dimensional collaborative optimization mechanism improves the overall production capacity of the multi-robot assembly line, reduces task conflicts, and reduces low-battery downtime.

[0132] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for an industrial robot autonomous collaborative decision-making method based on multimodal perception. When the program for an industrial robot autonomous collaborative decision-making method based on multimodal perception is executed by a processor, the steps of the industrial robot autonomous collaborative decision-making method based on multimodal perception as described in any one of the above items are implemented.

[0133] The present invention discloses an autonomous collaborative decision-making method, system, and medium for industrial robots based on multimodal perception, which achieves high-precision autonomous operation in industrial scenarios through multi-source sensor collaborative perception and intelligent decision-making mechanisms. Specifically, the system first uses a high-resolution camera, a six-dimensional force sensor, and a directional microphone to synchronously collect visual, tactile, and auditory information, and adopts cross-modal spatiotemporal alignment technology to solve the problem of multi-source data fusion: in the time dimension, timestamp alignment is achieved based on the precise time protocol hardware clock synchronization module, and the sampling frequency difference is eliminated by combining the interpolation compensation algorithm; in the spatial dimension, the visual pixel coordinates are mapped to the robot base coordinate system through the hand-eye calibration matrix, the tactile sensor installation posture matrix is ​​used to convert the force / torque data, and the three-dimensional coordinates of the sound source are estimated through the microphone array wave direction of arrival, finally constructing a unified multimodal perception benchmark in time and space. Subsequently, the system realizes the optimal configuration of multimodal perception resources based on a dynamic weight allocation mechanism: the current operation type is identified through the task semantic parsing module, the sensor weight ratio is dynamically adjusted in combination with environmental parameters, and the cross-modal correlation is calculated using an improved multi-head attention mechanism, and the feature fusion parameters are generated through softmax normalization. The decision optimization layer uses a deep reinforcement learning framework, taking the robot's motion state, environmental constraints, and mission objectives as input. It outputs a control instruction set that includes path planning parameters, operation force thresholds, and collaboration timing. It continuously optimizes the decision model through multi-objective scoring of energy consumption, accuracy, and safety. The system also integrates an environmental adaptive compensation mechanism: when the temperature exceeds the set threshold, the tactile sampling frequency and visual weight are adjusted; when the temperature is below the set threshold, auditory low-frequency enhancement is enabled and the tactile weight is increased. At the task execution layer, efficient multi-robot collaboration is achieved through a hysteresis compensation mechanism, an urgency arbitration model, and a power balancing strategy. Ultimately, a closed-loop verification mechanism is used to achieve adaptive optimization of perception-decision-making-execution, significantly improving overall accuracy and collaboration efficiency in complex industrial scenarios.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0135] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0136] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories, random access memories, magnetic disks or optical disks, and other media that can store program codes.

[0138] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as standalone products, they can also be stored on a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This software product, stored on a storage medium, includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as removable storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. An autonomous collaborative decision-making method for industrial robots based on multimodal perception, characterized by: The following steps are involved: Collect sensory information within a preset period of the autonomous collaborative system, including visual information, tactile information, and auditory information; Performing cross-modal spatiotemporal alignment of the visual information, tactile information, and auditory information using a preset method to generate first perceptual feature information; Extracting visual feature data, tactile feature data, and auditory feature data through a preset mechanism based on the first perception feature information and dynamically adjusting weights to generate feature fusion parameters; Generate a multimodal joint decision-making strategy based on the feature fusion parameters, and iteratively optimize the decision parameters through a preset optimization model; Obtain decision parameter optimization effect data and compare it with the preset effect threshold to determine whether the optimization effect meets the requirements; The dynamic weight adjustment includes: The importance ratio of each sensor is automatically set according to the current task type. In the fine assembly task, the weight of the tactile sensor is set to the first weight ratio. When it is necessary to detect machine abnormalities, the weight of the sound sensor is increased to the second weight ratio, while the weight of the camera image is reduced to the third weight ratio; The generation process of the feature fusion parameters is as follows: Analyze the degree of correlation between visual, tactile, and auditory data through attention computational models; Automatically assign the comprehensive impact value of each sensor based on the degree of association to ensure that the sum of the impact values ​​of vision, touch, and hearing is 1; The preset optimization model adopts a deep learning model, wherein: The robot's movement status, surrounding environment conditions, and mission objectives together form the basis for decision-making; The output control instructions include movement route parameters, operation force parameters, and collaboration timing parameters; Continuously optimize decision-making results through comprehensive scoring of energy consumption, operational accuracy, and safety; The optimization process of the comprehensive score also includes: Real-time monitoring of ambient noise levels and light intensity. When the ambient noise exceeds the preset noise threshold, the auditory sensor's frequency band filtering strength is dynamically increased and the visual weight is reduced. When the light intensity is lower than the preset intensity threshold, the infrared fill light module is activated and the depth of the visual feature extraction network is increased, while the tactile weight is increased; Also includes: Collecting the robot's working environment temperature during the dynamic weight adjustment process; When the working environment temperature is greater than a first temperature threshold, the sampling frequency of the tactile sensor is increased to compensate for the material deformation error caused by the high temperature, and the visual weight is reduced to avoid image distortion interference; When the working environment temperature is lower than the second temperature threshold, the low-frequency enhancement mode of the auditory sensor is enabled to improve the recognition of metallic noises, and the tactile weight is increased to enhance the force control accuracy at low temperatures.

2. The method for autonomous collaborative decision-making of industrial robots based on multimodal perception according to claim 1, characterized in that: The cross-modal spatiotemporal alignment includes: The time synchronization module accurately aligns the images captured by the camera, the force changes detected by the tactile sensor, and the sounds collected by the microphone according to the acquisition time. The position information detected by different sensors is unified into the three-dimensional space coordinate system of the robot through coordinate transformation.

3. The method for autonomous collaborative decision-making of industrial robots based on multimodal perception according to claim 1, characterized in that: When comparing the optimization effects: When the task completion accuracy is lower than the preset first completion rate or the response delay exceeds the preset time, the previous weight setting is automatically restored; If the decision fails to meet the target for three consecutive times, all sensors are recalibrated and the data fusion rules are updated.

4. An industrial robot autonomous collaborative decision-making system based on multimodal perception, characterized by: It includes a memory and a processor, the memory includes a program of an industrial robot autonomous collaborative decision-making method based on multimodal perception, and when the program of the industrial robot autonomous collaborative decision-making method based on multimodal perception is executed by the processor, the steps of the industrial robot autonomous collaborative decision-making method based on multimodal perception as described in any one of claims 1 to 3 are implemented.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium includes an industrial robot autonomous collaborative decision-making method, system and medium program based on multimodal perception. When the industrial robot autonomous collaborative decision-making method, system and medium program based on multimodal perception are executed by a processor, the steps of the industrial robot autonomous collaborative decision-making method based on multimodal perception as described in any one of claims 1 to 3 are implemented.

Images