An autonomous robot decision-making system based on multimodal perception fusion

Abstract

The present invention discloses an autonomous robot decision-making system based on multimodal perception fusion, and relates to the field of multimodal perception technology. The autonomous robot decision-making system based on multimodal perception fusion includes a clamping scheme acquisition module, a motion path matching module, and a robot motion state evaluation module. By integrating advanced image processing technology and sophisticated motion control strategies, the system can adjust the robot's clamping mode and motion path in real time, thereby improving operational efficiency and safety, reducing errors and item damage risks during operation, and is particularly suitable for automated warehousing and intelligent manufacturing scenarios, and can effectively solve the problem of how to accurately and stably grasp and carry various objects in complex and changing operating environments.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal perception, and in particular to an autonomous robot decision-making system based on multimodal perception fusion. Background Art

[0002] The autonomous robot decision-making system with multimodal perception fusion is an advanced technology framework that enhances the robot's environmental perception ability by integrating multiple perception modes (such as vision, hearing, touch, etc.). This system uses machine learning and artificial intelligence technologies to process and analyze data from different sensors, allowing the robot to more accurately understand its surrounding environment and make more complex and adaptive decisions based on this. The goal of this decision-making system is to improve the robot's autonomy and efficiency, enabling it to operate independently in a changing and unknown environment.

[0003] The Chinese invention patent application with publication number CN117873100A discloses a multi-modal fusion perception architecture and method for multiple mobile robots, the architecture includes n robots, a server and a group control scheduling system; the robot includes a multi-type sensor system, a body map and update unit, a multi-modal fusion calculation unit, a scheduling instruction receiving and parsing unit, a perception / decision calculation control unit, a robot motion control unit and a communication module. The present invention is directed to a generation architecture of a multi-modal scene map of multi-modal sensor data fusion for multiple mobile robots, forming a real-time, comprehensive multi-modal perception capability for the scene map, and providing the multi-modal perception data to individual robots in the scene. The individual robots can better realize mobile navigation and collaborative operation based on this multi-modal perception map.

[0004] However, existing handling robots used in workshops have the problem of being unable to accurately and stably grasp and carry various objects in complex and changing operating environments. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an autonomous robot decision-making system based on multimodal perception fusion, which solves the problem of how to accurately and stably grasp and carry various objects in a complex and changing operating environment.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an autonomous robot decision-making system based on multimodal perception fusion, comprising a clamping scheme acquisition module, a motion path matching module and a robot motion state evaluation module, wherein: the clamping scheme acquisition module is used to acquire cargo data through image perception, and acquire a clamping scheme based on the cargo data, wherein the cargo data includes cargo shape, cargo size, cargo quality and cargo material, and the clamping scheme includes a clamping position and a clamping force; the motion path matching module is used to acquire motion parameters of each motion node of the robot based on the cargo position and stacking position and the clamping scheme decision, wherein the motion parameters include the node rotation angle, the power output oil pipe oil pressure when the node posture changes, and the node motion speed; the robot motion state evaluation module is used to monitor and acquire the actual motion parameters of each motion node of the robot after the robot clamps the cargo through the clamping scheme, evaluate the robot motion state based on the actual motion parameters of each motion node and the motion parameters of each motion node of the robot, obtain an evaluation index, and provide feedback on the motion state of the autonomous robot based on the evaluation index, wherein the actual motion parameters include the actual rotation angle of the node, the actual oil pressure of the power output oil pipe when the node posture changes, and the actual motion speed of the node.

[0007] Furthermore, the decision-making based on cargo data to obtain a clamping solution includes the following steps: combining cargo shape and cargo material to form an index combination; comparing the index combination with each matching index combination stored in a database, and obtaining a cargo matching data set stored in the database corresponding to a matching index combination identical to the index combination, wherein the cargo matching data set includes a plurality of cargo matching data, and the cargo matching data includes a cargo matching size and a cargo matching mass; combining the cargo size and the cargo mass to form solution decision data; comparing the solution decision data with a plurality of cargo matching data in the cargo matching data set one by one through a comparison model to obtain a comparison coefficient; and obtaining a clamping solution stored in the database corresponding to the cargo matching data corresponding to the minimum comparison coefficient.

[0008] Furthermore, the calculation formula of the comparison model is:

[0009] ;

[0010] In the formula, is the comparison coefficient, For the cargo size, For the quality of goods, Match the size of the goods, Match the quality of the goods, is the similarity function between the cargo size and the cargo matching size.

[0011] Furthermore, the method of acquiring motion parameters of each motion node of the robot based on the cargo position, stacking position and clamping scheme decision includes the following steps: combining the cargo position and stacking position into a position combination; comparing the position combination with each matching position combination stored in the database, and acquiring a cargo handling scene stored in the database corresponding to the matching position combination identical to the position combination; in a determined cargo handling scene, acquiring a motion parameter matching guidance data set corresponding to the cargo handling scene stored in the database, wherein the motion parameter matching guidance data set includes a plurality of motion parameter matching guidance data, and the motion parameter matching guidance data includes matching clamping position and matching clamping force; using a matching guidance model to fuse and analyze the clamping scheme with the motion parameter matching guidance data, and obtaining a guidance coefficient; and acquiring the motion parameters of each motion node stored in the database corresponding to the motion parameter matching guidance data corresponding to the minimum guidance coefficient.

[0012] Furthermore, the calculation formula of the matching guidance model is:

[0013] ;

[0014] In the formula, is the guidance coefficient, is the clamping position, is the clamping force, To match the clamping position, To match the clamping force, is the similarity function between the clamping position and the matching clamping position.

[0015] Furthermore, the motion state of the robot is evaluated based on the actual motion parameters of each motion node and the motion parameters of each motion node of the robot to obtain an evaluation index, including the following steps: obtaining the allowable motion deviation parameters of each motion node, the allowable motion deviation parameters including the allowable deviation of the node rotation angle, the allowable deviation of the power output oil pipe oil pressure when the node posture changes, and the allowable deviation of the node motion speed; the actual motion parameters of each motion node, the motion parameters of each motion node and the allowable motion deviation parameters of each motion node are integrated to obtain an evaluation index.

[0016] Furthermore, the calculation formula of the evaluation index is:

[0017] ;

[0018]

[0019] In the formula, To evaluate the index, is the node number, , is the total number of nodes, For the The node rotation angle of each node, For the When the node attitude of each node changes, the oil pressure of the power output oil pipe is For the The node movement speed of each node, For the The actual rotation angle of the node, For the The actual oil pressure of the power output oil pipe when the node attitude of each node changes, For the The actual movement speed of the nodes, For the The node rotation angle of each node is allowed to deviate. For the The allowable deviation of the oil pressure in the power output oil pipe when the node attitude of each node changes. For the The node movement speed tolerance of each node is is a natural constant, For the transfer function.

[0020] Furthermore, the acquisition of the allowable motion deviation parameters of each motion node includes the following steps: acquiring motion parameter influencing data, wherein the motion parameter influencing data include robot operating temperature, robot power supply current standard deviation and instruction response delay; analyzing the motion parameter influencing data with each motion parameter matching influencing data stored in a database based on a fusion model to obtain a comparison index, wherein the motion parameter matching influencing data include robot matching operating temperature, robot power supply current matching standard deviation and instruction matching response delay; and acquiring the motion allowable deviation parameters of each motion node stored in the database corresponding to the motion parameter matching influencing data corresponding to the minimum comparison index.

[0021] Furthermore, the calculation formula of the fusion model is as follows:

[0022] ;

[0023] In the formula, is the comparison index, is the robot operating temperature, The standard deviation of the current supplying the robot, is the command response delay, Match the robot's operating temperature, Match the standard deviation of the current supplying the robot, is the instruction matching response delay, is a natural constant.

[0024] Furthermore, the feedback of the motion state of the autonomous robot based on the evaluation index includes the following steps: comparing the evaluation index with the evaluation threshold set in the database to determine whether the evaluation index is greater than the evaluation threshold set in the database: if the evaluation index is greater than the evaluation threshold set in the database, sending an alarm instruction to the alarm; if the evaluation index is not greater than the evaluation threshold set in the database, continuing to execute the handling task.

[0025] The present invention has the following beneficial effects:

[0026] (1) This autonomous robot decision-making system based on multimodal perception fusion integrates advanced image processing technology and sophisticated motion control strategies. The system can adjust the robot's gripping method and motion path in real time, thereby improving operational efficiency and safety and reducing the risk of errors and damage to items during operation. It is particularly suitable for automated warehousing and intelligent manufacturing scenarios, and can effectively solve the problem of how to accurately and stably grasp and carry various objects in complex and changing operating environments.

[0027] (2) The autonomous robot decision-making system based on multimodal perception fusion can ensure that the robot operates within the precise control range and reduce operational errors by continuously monitoring and evaluating the deviation of motion parameters from preset standards. The evaluation index can serve as an early warning system to help detect motion anomalies in a timely manner, so as to make adjustments or maintenance to prevent larger failures or accidents. Based on the evaluation index, the robot can be maintained and adjusted in a more targeted manner, improving maintenance efficiency and reducing unnecessary maintenance costs.

[0028] (3) Compare the impact data monitored in real time with the matching impact data stored in the database. This comparison can reveal the differences between the current working conditions and the ideal or historical conditions. Use a comparison index to quantify these differences. The smallest comparison index means that the current conditions are closest to the conditions of a successful case in the database. Set the current motion allowable deviation parameters based on the historical data that is closest to the comparison result, which means adjusting the motion control strategy according to the actual working environment to ensure optimal performance. By adjusting the motion allowable deviation parameters in real time to adapt to different working conditions, the operation of the robot can be more flexible and precise, especially in a changing working environment. Accurate deviation settings can optimize the performance of the robot, reduce the rate of false faults caused by environmental changes or system fluctuations, and enhance the reliability of the robot.

[0029] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of the autonomous robot decision-making system based on multimodal perception fusion of the present invention. DETAILED DESCRIPTION

[0031] The embodiment of the present application uses an autonomous robot decision-making system based on multimodal perception fusion. By integrating advanced image processing technology and sophisticated motion control strategies, the system can adjust the robot's clamping method and motion path in real time, thereby improving operational efficiency and safety, reducing errors and the risk of damage to items during operation, and is particularly suitable for automated warehousing and intelligent manufacturing scenarios. It can effectively solve the problem of how to accurately and stably grasp and carry various objects in complex and changing operating environments.

[0032] See also Figure 1 The embodiment of the present invention provides a technical solution: an autonomous robot decision-making system based on multimodal perception fusion, including a clamping scheme acquisition module, a motion path matching module and a robot motion state evaluation module. By accurately controlling the clamping scheme and the motion path, the robot can complete the task more quickly and safely, reducing errors and repeated operations. Multimodal perception enables the robot to cope with various objects and environments, enhancing the flexibility and scope of application of the system. Fine motion control and real-time state evaluation can prevent dangerous situations that may occur during robot operation, protect goods from damage, and ensure the safety of the operating site.

[0033] The clamping scheme acquisition module is used to acquire cargo data through image perception, and to acquire a clamping scheme based on the cargo data. The cargo data includes cargo shape, cargo size, cargo quality, and cargo material, and the clamping scheme includes clamping position and clamping force. The robot uses a high-resolution camera to capture images of cargo. These cameras may be equipped with a variety of sensors (such as visible light, infrared, and depth sensors) to acquire different types of image data, which helps to describe the appearance and environmental conditions of the cargo in detail. Through image processing algorithms, such as object detection or image segmentation, the system can identify the cargo in the image and locate its position in the image. The cargo size, cargo quality, and cargo material can be obtained through the label information on the cargo. Through optical character recognition (OCR) technology, these label information can be read from the image, and the information on the label is parsed and converted into digital data for use by the robot system.

[0034] The motion path matching module is used to obtain the motion parameters of each motion node of the robot based on the cargo position, stacking position and clamping solution decision. The motion parameters include the node rotation angle, the power output oil pipe oil pressure when the node posture changes, and the node movement speed.

[0035] The robot motion state evaluation module is used to monitor and obtain the actual motion parameters of each motion node of the robot after the robot clamps the goods through the clamping scheme, evaluate the robot motion state based on the actual motion parameters of each motion node and the motion parameters of each motion node of the robot, obtain an evaluation index, and provide feedback on the motion state of the autonomous robot based on the evaluation index. The actual motion parameters include the actual rotation angle of the node, the actual oil pressure of the power output oil pipe when the node posture changes, and the actual motion speed of the node.

[0036] The actual rotation angle of the node is usually measured by an encoder, which is installed on the robot joint or rotating part and can provide accurate angular position information. The power output oil pressure is measured by pressure sensors, which are usually installed on the oil pipes of the hydraulic system. The pressure sensor can monitor the pressure in the oil pipes in real time to ensure the normal operation of the robot's hydraulic system. The movement speed of the node can be measured by a variety of methods, such as using a speed encoder, which can directly measure the rotation speed, especially on rotating nodes.

[0037] Specifically, the method for obtaining a clamping solution based on cargo data decision includes the following steps: combining cargo shape and cargo material to form an index combination; comparing the index combination with each matching index combination stored in a database, and obtaining a cargo matching data set stored in the database corresponding to a matching index combination identical to the index combination, wherein the cargo matching data set includes a plurality of cargo matching data, and the cargo matching data includes a cargo matching size and a cargo matching mass; combining the cargo size and the cargo mass to form solution decision data; comparing the solution decision data with a plurality of cargo matching data in the cargo matching data set one by one through a comparison model to obtain a comparison coefficient; and obtaining a clamping solution stored in the database corresponding to the cargo matching data corresponding to the minimum comparison coefficient.

[0038] A unique index is created for each item by combining its shape and material. This step can be thought of as creating a "fingerprint" or identifier that can be used to quickly identify and classify items. The generated index is compared with a pre-stored matching index in the database, which reflects knowledge and experience from previous operations, including which gripping strategies are most effective for specific types of goods. Through index matching, the system can quickly find historical operation data similar to the current item, which contains gripping parameters that successfully handled similar items.

[0039] Combined with the size and quality data of the current goods, a decision data set is formed for subsequent clamping solution decisions. Through the comparison model, the system compares the decision data of the current goods with the goods matching data in the database one by one, calculates the comparison coefficient, and finds the best match. The best clamping solution is selected based on the smallest comparison coefficient, which means selecting the verified clamping strategy that best matches the current goods characteristics.

[0040] The calculation formula of the comparison model is:

[0041] ;

[0042] In the formula, is the comparison coefficient, For the cargo size, For the quality of goods, Match the size of the goods, Match the quality of the goods, is the similarity function between the cargo size and the cargo matching size.

[0043] In this embodiment, It can be a Euclidean distance function or other similarity function, measuring the similarity between the actual size of the goods (A) and the matching size (SA) recorded in the database. Part of the calculation considers the absolute difference between the actual mass of the goods (Ms) and the matched mass (Mc) and takes its square root. This treatment is intended to smooth the impact of large mass differences on the contrast coefficient, because mass differences are also an important factor in the robot's gripping strategy.

[0044] Size is one of the basic factors in determining how to clamp goods. It affects the selection of clamping devices, the determination of clamping positions, and the adjustment of force. Correctly understanding and processing dimensional data is the key to ensuring that the robot can effectively grasp goods without causing damage. Quality affects the stability and safety of robot operation. Heavier goods may require stronger clamping force or a specific type of clamping mechanism to prevent slipping or structural damage during handling. By comparing with the dimensions stored in the database, previous experience and parameters for successful clamping of similar goods can be quickly found. This "learning" and "memory" function makes the robot more efficient and accurate when handling similar goods. Similarly, comparing quality information can help the robot choose the appropriate force and technology to handle goods of a specific weight. This matching can reduce operational risks caused by weight errors.

[0045] By matching and selecting the best clamping solution, the system can perform clamping operations accurately and reduce the need for errors and adjustments. The systematic indexing and matching process accelerates decision-making, enabling the robot to quickly respond to different operational requirements and improve overall operation speed. Previously successful operation cases are stored in the database and utilized in future operations. Operation strategies can be continuously optimized and adjusted to improve the intelligence and adaptability of the overall system.

[0046] Specifically, the method for obtaining motion parameters of each motion node of the robot based on the cargo position, stacking position and clamping scheme decision includes the following steps: combining the cargo position and stacking position into a position combination; comparing the position combination with each matching position combination stored in a database, and obtaining a cargo handling scene stored in the database corresponding to the matching position combination identical to the position combination; in a determined cargo handling scene, obtaining a motion parameter matching guidance data set corresponding to the cargo handling scene stored in the database, wherein the motion parameter matching guidance data set includes a plurality of motion parameter matching guidance data, and the motion parameter matching guidance data includes matching clamping position and matching clamping force; using a matching guidance model to fuse and analyze the clamping scheme with the motion parameter matching guidance data, and obtaining a guidance coefficient; obtaining the motion parameters of each motion node stored in the database corresponding to the motion parameter matching guidance data corresponding to the minimum guidance coefficient.

[0047] The creation of a position combination, that is, the combination of the cargo position and the stacking position (the cargo position and the stacking position can be obtained through RFID tags. RFID tags are installed near the cargo and stacking positions. The RFID reader equipped with the robot can identify these tags and obtain relevant position information), helps to define the starting point and end point of the robot's handling of the cargo. This combination is input into the database query as a key parameter. By comparing with the pre-stored position combinations in the database, the system can find historical cases similar to the current task conditions for rapid identification of similar historical handling scenarios. Once a matching handling scenario is found, the system can access the associated motion parameter matching guidance data set. These data sets include motion parameters optimized for successful completion of similar tasks, such as clamping position and force. Using the matching guidance model, the current clamping scheme is fused and analyzed with the motion parameter matching guidance data. Such analysis can calculate a guidance coefficient. The guidance coefficient helps determine the optimal motion parameter configuration. Selecting the parameters corresponding to the minimum guidance coefficient means selecting the motion parameter configuration closest to the ideal model prediction, so as to ensure the optimality of the operation.

[0048] The design improves the accuracy and efficiency of operations by reusing proven successful strategies, reducing the need for trial and error, allowing the robot to adapt and complete tasks more quickly. By using motion parameters verified in historical data, the risk of operational errors can be significantly reduced, protecting cargo from damage while ensuring the safe operation of the robotic system. By analyzing historical data of multiple handling scenarios and motion parameters, the system can adapt to a wider range of task types and more complex operating environments.

[0049] The calculation formula of the matching guidance model is:

[0050] ;

[0051] In the formula, is the guidance coefficient, is the clamping position, is the clamping force, To match the clamping position, To match the clamping force, is the similarity function between the clamping position and the matching clamping position.

[0052] In this embodiment, It can be the Euclidean distance formula or other similarity functions. It measures the actual gripping position in the current task. Similarity between the clamping positions SC and the best practices in the database (historical data). The difference between the actual clamping force Ls and the ideal clamping force Lc recorded in the database is calculated and the square root is taken. The square root can make the influence of the force difference more linear and reduce the influence of extreme values.

[0053] Specifically, the robot motion state is evaluated based on the actual motion parameters of each motion node and the motion parameters of each motion node of the robot to obtain an evaluation index, including the following steps: obtaining the allowable motion deviation parameters of each motion node, the allowable motion deviation parameters including the allowable deviation of the node rotation angle, the allowable deviation of the power output oil pipe oil pressure when the node posture changes, and the allowable deviation of the node motion speed; the actual motion parameters of each motion node, the motion parameters of each motion node and the allowable motion deviation parameters of each motion node are integrated to obtain an evaluation index.

[0054] For each motion node (such as a joint or actuator), define the allowable deviation of its motion parameters (rotation angle, oil pressure, speed, etc.), which are set based on design tolerance, safety standards, and actual operating requirements. The allowable deviation is to ensure that the robot performs within the normal operating range while providing a certain degree of flexibility to deal with minor errors and environmental changes. Monitor and record the actual motion parameters of each motion node in real time, such as the actual rotation angle, oil pressure, and speed, and compare these actual parameters with the predetermined motion parameters in the robot system. Taking into account the actual motion parameters, predetermined parameters, and allowable deviation parameters, calculate the deviation of each node, which involves comparing the difference between the actual parameters and the predetermined parameters, and comparing the difference with the allowable deviation.

[0055] By continuously monitoring and evaluating the deviation of motion parameters from preset standards, it is possible to ensure that the robot operates within a precise control range and reduce operational errors. The evaluation index can serve as an early warning system to help detect motion anomalies in a timely manner, so that adjustments or maintenance can be made to prevent larger failures or accidents. Based on the evaluation index, the robot can be maintained and adjusted in a more targeted manner, improving maintenance efficiency and reducing unnecessary maintenance costs.

[0056] The calculation formula of the evaluation index is:

[0057] ;

[0058]

[0059] In the formula, To evaluate the index, is the node number, , is the total number of nodes, For the The node rotation angle of each node, For the When the node attitude of each node changes, the oil pressure of the power output oil pipe is For the The node movement speed of each node, For the The actual rotation angle of the node, For the The actual oil pressure of the power output oil pipe when the node attitude of each node changes, For the The actual movement speed of the nodes, For the The node rotation angle of each node is allowed to deviate. For the The allowable deviation of the oil pressure in the power output oil pipe when the node attitude of each node changes. For the The node movement speed tolerance of each node is is a natural constant, For the transfer function.

[0060] In this implementation, the formula uses natural logarithms and exponential functions to handle deviations. This method can amplify smaller deviations, making the evaluation index more sensitive to deviations and easier to capture small changes in motion states. The accumulation method ensures that the deviations of all nodes are taken into account, providing a quantitative indicator for the overall motion state of the robot.

[0061] The rotation angle refers to the change in angle of each joint or rotating part of the robot. This parameter directly affects the accuracy of the robot's execution of tasks, such as grasping, handling or assembly. Accurate control of the rotation angle is the basis for ensuring that the robot operates according to the predetermined path and position. For hydraulically driven robot systems, oil pressure is the main source of force to drive the movement of joints and actuators. The stability and accuracy of the oil pressure directly affect the robot's movement force and response speed, which is especially important for the robot to perform complex tasks or tasks that require fine force control. The movement speed of the node involves the speed of movement of each part of the robot, which is directly related to the efficiency and timeliness of the overall operation. Speed ​​control is also related to the smoothness and synchronization of the robot's movement, which is crucial for performing precise and coordinated tasks.

[0062] In order to achieve precise position control, the robot's movement speed and rotation angle need to be precisely coordinated. For example, in a grasping task, the speed of the actuator must be synchronized with the angle change to reach the target position to ensure the accuracy and timeliness of the grasping action. The stable output of oil pressure is the key to ensuring that the robot node moves at a constant speed and a predetermined angle. If the oil pressure is unstable or mismatched, it may cause delays or sudden acceleration in the robot's movement, affecting the accuracy and safety of task execution. Effective management of rotation angle, oil pressure and speed not only improves the robot's operating accuracy and response speed, but also helps to optimize energy consumption and extend the service life of mechanical components. The improvement of system efficiency is directly related to operation and maintenance costs and system sustainability.

[0063] These three parameters are key indicators of the robot's dynamic performance. They are interdependent and together determine the robot's operational performance. The formula can be used to very accurately evaluate the robot's motion state and promptly identify any slight deviations that may cause machine dysfunction. The formula allows real-time monitoring and evaluation of each motion node of the robot, quickly responding to possible deviations and making timely adjustments to maintain the robot's optimal performance.

[0064] Specifically, the method of obtaining the allowable motion deviation parameters of each motion node includes the following steps: obtaining motion parameter influencing data, wherein the motion parameter influencing data include the robot operating temperature, the robot power supply current standard deviation, and the command response delay; analyzing the motion parameter influencing data with each motion parameter matching influencing data stored in a database based on a fusion model to obtain a comparison index, wherein the motion parameter matching influencing data include the robot matching operating temperature, the robot power supply current matching standard deviation, and the command matching response delay; obtaining the motion allowable deviation parameters of each motion node stored in the database corresponding to the motion parameter matching influencing data corresponding to the minimum comparison index.

[0065] Temperature can affect the thermal expansion of mechanical parts and the efficiency of electronic components, thus affecting the accuracy of movement. Current fluctuations may reflect the stability of the power supply system, affecting the power output and smoothness of the robot's movement. The length of the response time directly affects the speed and timeliness of the robot's execution of instructions, and is related to the synchronization and coordination of the overall operation.

[0066] The real-time monitored impact data is compared with the matching impact data stored in the database. This comparison can reveal the difference between the current working conditions and the ideal or historical conditions. The comparison index is used to quantify these differences. The smallest comparison index indicates that the current conditions are closest to the conditions of a successful case in the database. The current motion tolerance parameters are set according to the historical data that is closest to the comparison result, which means adjusting the motion control strategy according to the actual working environment to ensure the best performance.

[0067] By adjusting the motion tolerance parameters in real time to adapt to different working conditions, the robot can operate more flexibly and accurately, especially in a changing working environment. Accurate deviation settings can optimize the robot's performance, reduce the rate of misjudgment of faults caused by environmental changes or system fluctuations, and enhance the robot's reliability.

[0068] The calculation formula of the fusion model is as follows:

[0069] ;

[0070] In the formula, is the comparison index, is the robot operating temperature, The standard deviation of the current supplying the robot, is the command response delay, Match the robot's operating temperature, Match the standard deviation of the current supplying the robot, is the instruction matching response delay, is a natural constant.

[0071] In this embodiment, , , The absolute differences of operating temperature, supply current standard deviation and command response delay between the current state and the matching state are measured respectively. The use of natural exponentials to enhance the weights of current and delay differences indicates that these factors may have nonlinear effects on robot performance, especially at extreme values. The entire expression is divided by 3 and then added to 1 and a hyperbolic tangent function is applied to perform a normalization process, which not only compresses the range of the comparison index, but also increases the nonlinear characteristics of the output, making small and medium deviations more obvious. The working error of the robot can be determined by the current working state.

[0072] Specifically, the feedback of the motion state of the autonomous robot based on the evaluation index includes the following steps: comparing the evaluation index with the evaluation threshold set in the database to determine whether the evaluation index is greater than the evaluation threshold set in the database: if the evaluation index is greater than the evaluation threshold set in the database, sending an alarm instruction to the alarm; if the evaluation index is not greater than the evaluation threshold set in the database, continuing to execute the handling task.

[0073] In this embodiment, the evaluation threshold value set in the database is usually derived based on historical data and statistical analysis. By collecting a large amount of robot motion state data, including parameters such as rotation angle, power output oil pipe oil pressure and node motion speed, the performance data under normal operation and fault conditions are recorded and analyzed. Using this data, the average value of these parameters can be calculated, and then the weighted sum is used to obtain the threshold value.

[0074] The evaluation index reflects the deviation between the actual motion parameters of each motion node of the robot and the preset parameters. The evaluation index is obtained through a certain calculation formula (as shown in the above model), which can quantify whether the motion state is normal or not. The evaluation threshold preset in the database is set based on the performance data and historical experience of the robot under normal operation. This threshold is used as a criterion for judging whether the robot is operating normally. If the evaluation index exceeds the set threshold, it indicates that the robot's motion state may be abnormal, such as excessive deviation or potential failure. If the evaluation index does not exceed the threshold, it indicates that the robot's motion state is within the normal range and can continue to perform the task.

[0075] When the evaluation index exceeds the threshold, the system automatically sends an alarm command to the alarm, which may trigger a visual or sound alarm to notify the operator or maintenance team to check and intervene. If the evaluation index is normal, the robot does not need additional intervention and continues to perform handling or other tasks. This mechanism can monitor the robot's motion status in real time, detect problems in time and take measures to reduce delays and possible damage caused by failures.

[0076] An electronic device comprises: a processor and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the autonomous robot decision-making system based on multimodal perception fusion as described above.

[0077] A computer-readable storage medium is used to store a program, which, when executed by a processor, implements the autonomous robot decision-making system based on multimodal perception fusion as described above.

[0078] In summary, this application has at least the following effects:

[0079] By integrating advanced image processing technology and sophisticated motion control strategies, the system can adjust the robot's gripping method and motion path in real time, thereby improving operational efficiency and safety, reducing errors and the risk of damage to items during operation. It is particularly suitable for automated warehousing and smart manufacturing scenarios, and can effectively solve the problem of how to accurately and stably grasp and carry various objects in complex and changing operating environments.

[0080] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0085] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An autonomous robot decision-making system based on multimodal perception fusion, characterized in that: It includes a clamping solution acquisition module, a motion path matching module and a robot motion state evaluation module, among which: The clamping scheme acquisition module is used to acquire cargo data through image perception, and acquire a clamping scheme based on the cargo data, wherein the cargo data includes cargo shape, cargo size, cargo quality, and cargo material, and the clamping scheme includes a clamping position and a clamping force; The motion path matching module is used to obtain the motion parameters of each motion node of the robot based on the cargo position and stacking position and the clamping solution decision, and the motion parameters include the node rotation angle, the power output oil pipe oil pressure when the node posture changes, and the node motion speed; The robot motion state evaluation module is used to monitor and obtain the actual motion parameters of each motion node of the robot after the robot clamps the goods through the clamping scheme, evaluate the robot motion state based on the actual motion parameters of each motion node and the motion parameters of each motion node of the robot, obtain an evaluation index, and provide feedback on the motion state of the autonomous robot based on the evaluation index, wherein the actual motion parameters include the actual rotation angle of the node, the actual oil pressure of the power output oil pipe when the node posture changes, and the actual motion speed of the node; The step of evaluating the robot motion state based on the actual motion parameters of each motion node and the motion parameters of each motion node of the robot to obtain an evaluation index comprises the following steps: Acquire the permissible motion deviation parameters of each motion node, wherein the permissible motion deviation parameters include the permissible deviation of the node rotation angle, the permissible deviation of the power output oil pipe oil pressure when the node posture changes, and the permissible deviation of the node motion speed; The actual motion parameters of each motion node, the motion parameters of each motion node and the motion allowable deviation parameters of each motion node are integrated to obtain an evaluation index; The calculation formula of the evaluation index is: ； ； In the formula, To evaluate the index, is the node number, , is the total number of nodes, For the The node rotation angle of each node, For the When the node attitude of each node changes, the oil pressure of the power output oil pipe is For the The node movement speed of each node, For the The actual rotation angle of the node, For the The actual oil pressure of the power output oil pipe when the node attitude of each node changes, For the The actual movement speed of the nodes, For the The node rotation angle of each node is allowed to deviate. For the The allowable deviation of the oil pressure in the power output oil pipe when the node attitude of each node changes. For the The node movement speed tolerance of each node is is a natural constant, For the transfer function.

2. The autonomous robot decision-making system based on multimodal perception fusion according to claim 1, characterized in that: The method of obtaining a clamping solution based on cargo data decision-making includes the following steps: Combine the cargo shape and cargo material to form an index combination; Compare the index combination with each matching index combination stored in the database, and obtain a cargo matching data set stored in the database corresponding to the matching index combination identical to the index combination, wherein the cargo matching data set includes a plurality of cargo matching data, and the cargo matching data includes cargo matching size and cargo matching quality; Combine cargo size and cargo mass to form solution decision data; The scheme decision data is compared with several cargo matching data in the cargo matching data set one by one through the comparison model to obtain the comparison coefficient; The clamping solution stored in the database and corresponding to the cargo matching data corresponding to the minimum comparison coefficient is obtained.

3. The autonomous robot decision-making system based on multimodal perception fusion according to claim 2 is characterized in that: The calculation formula of the comparison model is: ； In the formula, is the comparison coefficient, For the cargo size, For the quality of goods, Match the size of the goods, Match the quality of the goods, is the similarity function between the cargo size and the cargo matching size.

4. The autonomous robot decision-making system based on multimodal perception fusion according to claim 1, characterized in that: The method of obtaining the motion parameters of each motion node of the robot based on the cargo position, the stacking position and the clamping scheme decision includes the following steps: Combining cargo positions and stacking positions into position combinations; Compare the position combination with each matching position combination stored in the database, and obtain the cargo handling scene stored in the database corresponding to the matching position combination identical to the position combination; In a determined cargo handling scenario, a motion parameter matching guidance data set corresponding to the cargo handling scenario stored in a database is obtained, wherein the motion parameter matching guidance data set includes a plurality of motion parameter matching guidance data, and the motion parameter matching guidance data includes a matching clamping position and a matching clamping force; The matching guidance model is used to fuse and analyze the clamping scheme and the motion parameter matching guidance data to obtain the guidance coefficient; The motion parameter corresponding to the minimum guidance coefficient is obtained to match the motion parameter of each motion node stored in the database corresponding to the guidance data.

5. The autonomous robot decision-making system based on multimodal perception fusion according to claim 4 is characterized in that: The calculation formula of the matching guidance model is: ； In the formula, is the guidance coefficient, is the clamping position, is the clamping force, To match the clamping position, To match the clamping force, is the similarity function between the clamping position and the matching clamping position.

6. The autonomous robot decision-making system based on multimodal perception fusion according to claim 1 is characterized in that The step of obtaining the motion allowable deviation parameter of each motion node comprises the following steps: Acquiring motion parameter influence data, wherein the motion parameter influence data includes robot operating temperature, robot power supply current standard deviation and command response delay; Based on the fusion model, the motion parameter influence data is analyzed with the motion parameter matching influence data stored in the database to obtain a comparison index, wherein the motion parameter matching influence data includes the robot matching operating temperature, the robot power supply current matching standard deviation and the instruction matching response delay; The motion parameter matching influence data corresponding to the minimum comparison index is obtained, and the motion allowable deviation parameters of each motion node stored in the database are affected.

7. The autonomous robot decision-making system based on multimodal perception fusion according to claim 6, characterized in that: The calculation formula of the fusion model is as follows: ； In the formula, is the comparison index, is the robot operating temperature, The standard deviation of the current supplying the robot, is the command response delay, Match the robot's operating temperature, Match the standard deviation of the current supplying the robot, is the instruction matching response delay, is a natural constant.

8. The autonomous robot decision-making system based on multimodal perception fusion according to claim 1, characterized in that: The method of providing feedback on the motion state of the autonomous robot based on the evaluation index comprises the following steps: Compare the evaluation index with the evaluation threshold set in the database to determine whether the evaluation index is greater than the evaluation threshold set in the database: If the evaluation index is greater than the evaluation threshold set in the database, an alarm instruction is sent to the alarm; If the evaluation index is not greater than the evaluation threshold set in the database, the transport task continues to be executed.

Images