Mobile robot control system, method and medium based on end-side model

Abstract

The present application provides a mobile robot control system, method and medium based on an end-side model, including: a data acquisition module: used to acquire multimodal perception data, the multimodal perception data including visual data, lidar data and inertial measurement unit data; a calculation module: used to calculate the scene complexity based on the multimodal perception data; a model selection module, used to adaptively select an end-side workload model based on the scene complexity. The embodiment of the present application can dynamically match the end-side workload model according to the task performed by the mobile robot, so that its own hardware computing resources can achieve higher adaptability, avoid waste of computing power or insufficient performance, and reduce power consumption.

Description

Technical Field

[0001] The present application relates to the field of intelligent robot technology, and in particular to a mobile robot control system, method, and medium based on a terminal-side model. Background Art

[0002] With the increasing complexity of mobile (composite) robot application scenarios in intelligent manufacturing (such as industrial inspection, warehousing and logistics, workshop operations, production line integration operations, etc.), the limitations of traditional cloud-based large models that rely on network transmission are becoming increasingly prominent, with problems such as poor real-time performance, high privacy risks, and high energy consumption.

[0003] Small on-device models address these pain points through localized deployment, but mobile robotics still face challenges such as limited computing power, multimodal perception fusion, and dynamic environment adaptation. Existing technologies, such as pruned quantization models, reduce parameter size but lack specialized architectures designed for robotic control scenarios. Furthermore, on-device model deployment often relies on fixed hardware configurations, making it difficult to adapt to diverse tasks and complex, dynamically changing scenarios. Summary of the Invention

[0004] The purpose of this application is to provide a mobile robot control system, method and medium based on an end-side model, which can solve the problems in the existing technology of insufficient dynamic adjustment capabilities in complex working scenarios, working properties, task types, etc., and difficulty in achieving high adaptability based on its own hardware computing resources, resulting in waste of computing power or insufficient performance.

[0005] To solve the above technical problems, an embodiment of the present application provides a mobile robot control system based on an end-side model, including: a data acquisition module: used to acquire multimodal perception data, the multimodal perception data including visual data, lidar data and inertial measurement unit data; a calculation module: used to calculate the scene complexity based on the multimodal perception data; a model selection module, used to adaptively select the end-side workload model according to the scene complexity.

[0006] In one embodiment, the workload model is divided into a lightweight model, a medium-weight model, and a full-weight model according to the type of task performed by the robot and the requirements for navigation accuracy and dynamic obstacle avoidance.

[0007] In one embodiment, the data acquisition module is also used to obtain the task type; the calculation module is also used to calculate the task complexity based on the task type, and calculate the comprehensive complexity based on the task complexity and the scenario complexity; the model selection module is also used to adaptively select the end-side workload model based on the comprehensive complexity.

[0008] In one embodiment, the model selection module includes a prediction submodule and a model selection submodule; the data acquisition module is also used to acquire the motion data of the robot; the calculation module is also used to calculate the time data of the motion data based on time; the prediction submodule is used to predict the comprehensive complexity based on the time data, the task complexity and the scene complexity; the model selection submodule is used to adaptively select the end-side workload model according to the comprehensive complexity.

[0009] In one embodiment, a prediction model is pre-built in the prediction submodule; the prediction submodule predicts the comprehensive complexity by inputting the time data, the task complexity and the scenario complexity into the prediction model.

[0010] An embodiment of the present application also provides a mobile robot control method based on an end-side model, which is applied to the above-mentioned mobile robot control system, including: obtaining multimodal perception data, the multimodal perception data including visual data, lidar data, and inertial measurement unit data; calculating the scene complexity based on the multimodal perception data; and adaptively selecting an end-side workload model based on the scene complexity.

[0011] In one embodiment, the method further includes: obtaining a task type; calculating a task complexity based on the task type; calculating a comprehensive complexity based on the task complexity and the scenario complexity; and adaptively selecting an end-side workload model based on the comprehensive complexity.

[0012] In one embodiment, the method further comprises: acquiring motion data of the robot;

[0013] Calculate time data of the motion data based on time; predict comprehensive complexity based on the time data, the task complexity, and the scenario complexity; and adaptively select an end-side workload model based on the comprehensive complexity.

[0014] In one embodiment, predicting the comprehensive complexity based on the time data, the task complexity data and the scenario complexity data includes: predicting the comprehensive complexity by inputting the time data, the task complexity and the scenario complexity into a pre-built prediction model.

[0015] In one embodiment, predicting the comprehensive complexity by inputting the time data, the task complexity, and the scenario complexity into a pre-built prediction model includes:

[0016] A Transformer encoder is used to encode the three-dimensional data of the time data, task complexity data, and scene complexity data; a multi-head attention mechanism is used to interactively fuse the features of the three dimensions output by the encoder; and based on the fused features, a multi-layer perceptron is used to output the comprehensive complexity in the future period.

[0017] In one embodiment, the workload model is divided into a lightweight model, a medium-weight model, and a full-weight model according to the type of task performed by the robot and the requirements for navigation accuracy and dynamic obstacle avoidance.

[0018] In one embodiment, the calculating of scene complexity based on the multimodal perception data includes: extracting features from the multimodal perception data; and calculating the scene complexity based on the feature-extracted data using the following formula: ,

[0019]

[0020] in, , , represents the weight factor, Indicates the complexity of the visual image, represents the complexity of the lidar, Represents the complexity of the inertial measurement unit.

[0021] In one embodiment, the adaptive selection of the workload model according to the scene complexity includes: when the scene complexity When the scene complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity of the scene is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected.

[0022] In one embodiment, the comprehensive complexity is calculated according to the task complexity and the scene complexity, including: calculating the comprehensive complexity by the following formula :

[0023]

[0024] in, and represents the weight factor, Indicates the scene complexity, Indicates the complexity of the task; the adaptive selection of the workload model according to the comprehensive complexity includes: when the comprehensive complexity When the complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected.

[0025] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned mobile robot control method are implemented.

[0026] The embodiments of the present application acquire multimodal perception data in real time, calculate the scene complexity based on the multimodal perception data, and adaptively select the end-side workload model based on the scene complexity. The end-side workload model can be dynamically matched according to the task to be performed by the mobile robot. When the task to be performed has high requirements for navigation accuracy and dynamic obstacle avoidance, the full-scale model is adopted. When the task to be performed has low requirements for navigation accuracy and dynamic obstacle avoidance, the low-scale model is adopted. This allows the robot to achieve high adaptability based on its own hardware computing resources, avoid wasted computing power or insufficient performance, and reduce power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0028] Figure 1 is a structural diagram of a mobile robot control system according to the first embodiment of the present application;

[0029] Figure 2 is a structural diagram of a mobile robot control system according to a third embodiment of the present application;

[0030] Figure 3 is a schematic diagram of a framework of a prediction model according to the third embodiment of the present application;

[0031] Figure 4 This is a schematic diagram of a framework for selecting a workload model according to the third embodiment of the present application;

[0032] Figure 5 is a flowchart of a mobile robot control method according to a fourth embodiment of the present application;

[0033] Figure 6 is a flowchart of a mobile robot control method according to a fifth embodiment of the present application;

[0034] Figure 7is a flowchart of a mobile robot control method according to a sixth embodiment of the present application;

[0035] Figure 8 This is a flowchart of the method for predicting comprehensive complexity according to the sixth embodiment of the present application. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0037] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0038] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0039] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples, however, one skilled in the art will appreciate that the examples can be practiced without these specific details.

[0040] Currently, existing mobile robots struggle to adapt to diverse tasks and operate in complex, dynamically changing scenarios. It is necessary to establish a control framework and decision-making system on the mobile robot's control side that can adaptively select small-model computing tasks on the side. This will ensure that the robot maintains high real-time performance and appropriate performance based on its own hardware computing resources in different scenarios and task types.

[0041] Based on this, the first embodiment of this application proposes a mobile robot control system based on a terminal-side model, such as Figure 1 As shown, it includes: a data acquisition module 1, used to acquire multimodal perception data, and the multimodal perception data includes visual data, lidar data and inertial measurement unit data; a calculation module 2, used to calculate the scene complexity according to the multimodal perception data; a model selection module 3, used to adaptively select a workload model according to the scene complexity.

[0042] In this embodiment, workload models are categorized into lightweight, medium, and full models based on the type of tasks performed by the robot and the requirements for navigation accuracy and dynamic obstacle avoidance. Specifically, the lightweight model utilizes a highly efficient shallow model and processes only key features. It is suitable for obstacle-free or low-obstacle, spacious mobile areas (including areas with a largely fixed route), or for operations involving only moving loads (transplanting) or inspections. It requires low navigation accuracy, low dynamic obstacle avoidance response, and a small computational workload. The medium-weight model utilizes a hybrid of multiple shallow models to process high-density visual features. It is suitable for narrow, crowded, or obstructed mobile areas, or for secondary positioning operations at the end of a manipulator. It requires high navigation accuracy and dynamic obstacle avoidance response, and a large computational workload. The full-scale model utilizes a deep network model, performs multi-feature data fusion, and performs high-precision 3D modeling. It is suitable for mobile operation areas, docking areas with other equipment, end-of-line manipulator operations, or operations where the mobile robot docks with other equipment. It requires the highest navigation accuracy and dynamic obstacle avoidance response, and a large computational workload. As shown in Table 1:

[0043] Table 1: Workload model and corresponding execution task requirements

[0044]

[0045] In summary, the lightweight model uses a highly efficient shallow model to process only key features, with a small amount of computational tasks; the medium-weight model is a mixture of multiple shallow models to process high-density visual features, with a large amount of computational tasks; the full-scale model uses a deep network model to perform multi-feature data fusion and high-precision 3D modeling, with a large amount of computational tasks. The three workload models in the embodiment of this application can be pre-deployed on the control end side of the mobile robot to adaptively select the workload model based on the acquired multimodal perception data.

[0046] In this embodiment, the data acquisition module 1 acquires multimodal perception data including visual data, lidar data and inertial measurement unit data. The calculation module 2 calculates the scene complexity based on the multimodal perception data. Specifically, the calculation module 2 performs data fusion processing on the complexity of the feature data based on visual perception, the complexity of the lidar feature data, and the complexity of the inertial measurement unit information data. The visual complexity is calculated by graphic entropy. The more complex the texture of the image, the higher the entropy value. For example, the entropy value of a relatively empty scene is lower than the entropy value of objects with a certain density in the scene. Symbolized by Indicates the complexity of the visual image. The point cloud data generated by the laser radar scan is represented by the value of the distribution variance calculated to indicate the complexity of the scene; for example, the larger the variance value of the unstructured scene is, the higher it is than the structured scene, and the symbol Inertial measurement unit data is represented by the rate of change of the value of the calculated data flow per unit time over time. The higher the rate of change over time, the higher the dynamic complexity and the higher the response requirement. The final scene complexity is represented by As shown below:

[0047]

[0048] in, , , Represents the weight factor, which can generally take a relatively fixed value, or its size can be determined by some algorithm; Indicates the complexity of the visual image, represents the complexity of the lidar, Represents the complexity of the inertial measurement unit. Then, the model selection module 3 selects the scene complexity according to the Adaptively select the workload model, specifically the scene complexity can be set The threshold value, when the scene complexity When the workload is higher than the threshold, it can be considered that the conditions for switching the workload model are met. For example, a two-level threshold can be set. When the complexity of the scene is When the scene complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity of the scene is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected. The value of is between [0,1]. The first threshold and the second threshold can be determined according to actual needs. For example, the first threshold can be set to 0.4 and the second threshold can be set to 0.7. Choose a lightweight model when Choose a mid-weight model when Select the full-scale model.

[0049] This embodiment uses data acquisition module 1 to acquire multimodal perception data in real time and adaptively select a workload model. The end-side workload model can be dynamically matched based on the task being performed by the mobile robot. When the task requires high navigation accuracy and dynamic obstacle avoidance, the full-scale model is used; when the task requires low navigation accuracy and dynamic obstacle avoidance, the low-scale model is used. This allows for high adaptability based on the mobile robot's hardware computing resources, avoiding wasted computing power or insufficient performance, and reducing power consumption.

[0050] The second embodiment of the present application relates to a mobile robot control system. This embodiment is improved on the basis of the first embodiment. The improvement is that this embodiment also introduces additional task types for decision-making in the adaptive workload selection model to achieve more accurate computing power allocation. Figure 1 , the data acquisition module 1 in this embodiment is also used to obtain the task type; the calculation module 2 is also used to calculate the task complexity according to the task type, and calculate the comprehensive complexity according to the task complexity and the scene complexity; the model selection module 3 is also used to adaptively select the workload model according to the comprehensive complexity. Specifically, in addition to obtaining multimodal perception data, the data acquisition module 1 in this embodiment also obtains the task type; in addition to calculating the scene complexity according to the multimodal perception data, the calculation module 2 also calculates the task complexity according to the task type. In calculating the task complexity according to the task type, a corresponding relationship between the task complexity and the navigation accuracy and obstacle avoidance dynamic response requirements can be established according to the task type. For example, if the requirements for navigation accuracy and obstacle avoidance dynamic response are low, the task complexity is also low; if the requirements for navigation accuracy and obstacle avoidance dynamic response are high, the task complexity is also high, as shown in Table 2.

[0051] Table 2: Correspondence between task type and task complexity

[0052]

[0053] Furthermore, a certain numerical value can be assigned to the task complexity of the task type in advance. For example, the complexity of mobile navigation tasks can be defined as 0.2; the computational complexity of path planning can be defined as 0.5; the computational complexity of accurate operation of the end manipulator (such as grasping) can be defined as 0.8; emergency and rapid obstacle avoidance can be defined as 0.9, and so on.

[0054] In practical applications, it can also be defined based on other types of tasks that mobile robots often perform. Furthermore, the comprehensive complexity can be calculated according to the following formula :

[0055]

[0056] in, and represents the weight factor, Indicates the scene complexity, Represents the complexity of the task, where the weight factor and You can choose according to actual requirements, such as =0.75, =0.25. Then, the model selection module 3 can be based on the comprehensive complexity Adaptively select the workload model. Specifically, two thresholds can be set. For example, when the comprehensive complexity When the complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected. The value of is between [0,1]. The first threshold and the second threshold can also be determined according to actual needs. For example, the first threshold can be set to 0.4 and the second threshold can be set to 0.7. Choose a lightweight model when Choose a mid-weight model when Select the full-scale model.

[0057] In this embodiment, when implementing the adaptive selection of the terminal-side small model, the data acquisition module 1 not only acquires multimodal perception data in real time and evaluates it according to the scene complexity, but also introduces the task type for decision-making. It is suitable for dynamically responding to key tasks when the mobile robot performs multiple tasks at the same time, avoiding the limitations of scene complexity evaluation, and achieving more accurate computing power allocation (avoiding resource waste or insufficient performance) to achieve long-term resource optimization.

[0058] The third embodiment of the present application relates to a mobile robot control system. This embodiment is improved on the basis of the second embodiment. The improvement is that this embodiment also introduces time data for decision-making in the adaptive workload selection model to achieve a higher level of computing power allocation. Figure 2 As shown in , the model selection module 3 in this embodiment includes a prediction submodule 31 and a model selection submodule 32; the data acquisition module 1 is also used to obtain the motion data of the robot; the calculation module 2 is also used to calculate the time data of the motion data based on time; the prediction submodule 31 is used to predict the comprehensive complexity according to the time data, the task complexity and the scene complexity; the model selection submodule 32 is used to adaptively select the workload model according to the comprehensive complexity. Specifically, a prediction model can be pre-constructed in the prediction submodule 31 of this embodiment; the prediction submodule 31 predicts the comprehensive complexity by inputting the time data, the task complexity and the scene complexity into the prediction model. In this embodiment, the framework structure of the prediction model is as follows Figure 3 As shown in , the prediction model receives three-dimensional data input, including time data Mt (the temporal data dimension), task complexity data Ht (the task data dimension), and scene complexity data Ho (the spatial data dimension). Each dimension of data is encoded using a Transformer encoder, such as a four-layer encoder. Each layer includes multiple attention heads, such as a four-head attention head. Subsequently, three-dimensional cross-attention fusion is performed, and the fusion result enters the prediction layer, which is based on a multi-layer perceptron. The future complexity output layer can be set to output a prediction of the comprehensive complexity over a period of time in the future, such as a prediction for the next three seconds. This provides a basis for selecting the workload model on the robot side.

[0059] The framework structure of the selection process of the workload model in this embodiment is as follows: Figure 4 As shown in the figure, first, the data acquisition module 1 collects multimodal perception data, such as visual data, lidar data, inertial measurement unit (IMU) data, etc., and then the computing module 2 performs feature extraction and fusion calculation on this data. The complexity of the scene Ho is evaluated based on the fusion calculation results. At the same time, the complexity of the task Ht is evaluated based on the type of task to be performed by the robot. Finally, the scene complexity and task complexity are comprehensively considered and based on spatiotemporal perception and robot motion data Mt, the prediction model in the prediction submodule 31 predicts the scene and task of the robot's next step or the next task operation stage. Based on the prediction results, the three different scale models are switched in advance for a certain period of time.

[0060] This embodiment, building on the existing evaluation dimensions of scene complexity and robot task type complexity, also introduces the time dimension of the robot motion data stream. This three-dimensional data, calculated using the established prediction model, can be used to obtain an evaluation result of the comprehensive complexity within a certain period of time in the future, providing a basis for selecting the robot-side workload model. This facilitates rapid pre-loading of models, improving the robot's response speed and ability to predict dynamic scenes, tasks, and motion states. It overcomes the shortcomings of relying solely on scene complexity to select models, and improves the accuracy and effectiveness of the final working model selection.

[0061] The fourth embodiment of the present application relates to a mobile robot control method, the process of which is as follows: Figure 5 As shown, the details are as follows:

[0062] In step 101, multimodal perception data is acquired, including visual data, lidar data, and inertial measurement unit data. In this embodiment, the visual data can be acquired using a visual sensor, and the lidar data can be point cloud data generated by 2D lidar or 3D lidar scanning. The inertial measurement unit data can be motion data such as the robot's speed, acceleration, and attitude angle measured by an accelerometer, gyroscope, magnetometer, or the like.

[0063] In step 102, the scene complexity is calculated based on the multimodal perception data. In this embodiment, the complexity of the multimodal perception data can be first fused. For example, the visual complexity is calculated by the image entropy. The more complex the texture of the image, the higher the entropy value. For example, the entropy value of a relatively empty scene is lower than the entropy value of a scene with a certain density of objects. Indicates the complexity of the visual image. The point cloud data generated by the laser radar scan is represented by the value of the distribution variance calculated. For example, the larger the variance value of the unstructured scene is, the higher it is than the structured scene. The laser radar complexity is represented by the symbol Inertial measurement unit data is represented by the rate of change of the value of the calculated data flow per unit time. The higher the rate of change over time, the higher the dynamic complexity and the higher the response requirement. The complexity of the inertial measurement unit is represented by the symbol The final scene complexity is represented by As shown below:

[0064]

[0065] in, , , Represents the weight factor, which can generally take a relatively fixed value; its size can also be determined by some algorithm; Indicates the complexity of the visual image, represents the complexity of the lidar, Represents the complexity of the inertial measurement unit.

[0066] In step 103, the workload model is adaptively selected according to the complexity of the scene. The workload model in the embodiment of the present application can be divided into: lightweight model, medium weight model and full weight model according to the requirements of navigation accuracy and dynamic obstacle avoidance when the robot performs the task. Specifically, the lightweight model adopts a high-efficiency shallow model, only processes key features, and has a small amount of computational tasks; the medium weight model adopts a mixture of multiple shallow models to process high-density visual features, and has a large amount of computational tasks; the full weight model adopts a deep network model to perform multi-feature data fusion and high-precision 3D modeling, and has a large amount of computational tasks. The three workload models in the embodiment of the present application can be pre-deployed on the control end side of the mobile robot to adaptively select the workload model according to the acquired multimodal perception data. The three workload models in the embodiment of the present application can be pre-deployed on the control end side of the mobile robot to adaptively select the workload model according to the acquired multimodal perception data. Specifically, the scene complexity can be set. The threshold value, when the scene complexity When the workload is higher than the threshold, it can be considered that the conditions for switching the workload model are met. For example, a two-level threshold can be set. When the complexity of the scene is When the scene complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity of the scene is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected. The value of is between [0,1]. The first threshold and the second threshold can be determined according to actual needs. For example, the first threshold can be set to 0.4 and the second threshold can be set to 0.7. Choose a lightweight model when Choose a mid-weight model when Select the full-scale model.

[0067] This embodiment adaptively selects a workload model by acquiring multimodal perception data in real time. The workload model can be dynamically matched to the mobile robot's task. When the task requires high navigation accuracy and dynamic obstacle avoidance, the full-scale model is used; when the task requires low navigation accuracy and dynamic obstacle avoidance, the low-scale model is used. This allows for high adaptability based on the robot's hardware computing resources, avoiding wasted computing power or insufficient performance, and reducing power consumption.

[0068] The fifth embodiment of the present application relates to a mobile robot control method. The fifth embodiment is improved on the basis of the fourth embodiment. The improvement is that this embodiment also introduces additional task types for decision-making in the adaptive workload selection model to achieve more accurate computing power allocation. The process is as follows: Figure 6 As shown, the details are as follows:

[0069] In step 201, multimodal perception data and task types are acquired. In this embodiment, the multimodal perception data includes visual data, lidar data, and inertial measurement unit data. Robot task types may include inspection, transplanting, docking and positioning the robot with other equipment, and end-of-line manipulator operation.

[0070] In step 202, scene complexity is calculated based on the multimodal perception data, and task complexity is calculated based on the task type. The method for calculating scene complexity based on multimodal perception data in this step can be referenced to step 102 in the fourth embodiment and will not be repeated here. When calculating task complexity based on task type, a corresponding relationship between task complexity Ht and navigation accuracy and obstacle avoidance dynamic response requirements can be established based on the task type, as shown in Table 2.

[0071] In step 203, the comprehensive complexity is calculated according to the task complexity and the scene complexity. Specifically, the comprehensive complexity can be calculated according to the following formula: ;

[0072]

[0073] in, and represents the weight factor, is the scene complexity, The complexity of the task.

[0074] In step 204, the workload model is adaptively selected according to the comprehensive complexity. Specifically, two thresholds can also be set, for example: when the comprehensive complexity When the complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected. The value of is between [0,1]. The first threshold and the second threshold can also be determined according to actual needs. For example, the first threshold can be set to 0.4 and the second threshold can be set to 0.7. Choose a lightweight model when Choose a mid-weight model when Select the full-scale model.

[0075] When implementing the adaptive selection of small models on the terminal side, this embodiment not only evaluates based on multimodal perception data acquired in real time, but also introduces task types for decision-making. This is suitable for dynamically responding to key tasks when mobile robots perform multiple tasks simultaneously, avoiding the limitations of scene complexity assessment, and achieving more accurate computing power allocation (avoiding resource waste or insufficient performance) to achieve long-term resource optimization.

[0076] The sixth embodiment of the present application relates to a mobile robot control method. The sixth embodiment is improved on the basis of the fifth embodiment. The improvement is that this embodiment also introduces time data for decision-making in the adaptive workload selection model to achieve a higher level of computing power allocation. The process is as follows Figure 7 As shown, the details are as follows:

[0077] In step 301, multimodal perception data, the task type, and the robot's motion data are acquired. In this embodiment, the multimodal perception data includes visual data, lidar data, and inertial measurement unit data. Robot tasks may include inspection, transplanting, docking and positioning between robots and other equipment, and end-of-line manipulator operation. The robot's motion data may include speed, acceleration, and attitude angle data for the previous few seconds.

[0078] In step 302, the scene complexity is calculated based on the multimodal perception data, the task complexity is calculated based on the task type, and the time data of the motion data based on time is calculated; the method of calculating the scene complexity based on the multimodal perception data and the task complexity based on the task type in this step can refer to step 202 of the fifth embodiment and will not be repeated here. In calculating the time data of the motion data based on time, the motion data can be processed in time series to extract the feature data of the time dimension. .

[0079] In step 303, the comprehensive complexity is predicted based on the time data, the task complexity and the scene complexity; in this embodiment, the time data can be used to , the complexity of the task And the complexity of the scene Input pre-built prediction models to predict comprehensive complexity .

[0080] In this embodiment, the method of using the prediction model to predict the comprehensive complexity is as follows: Figure 8 As shown, the following steps are included:

[0081] In step 3031, a Transformer encoder is used to encode the three-dimensional data of the time data (time data dimension), the task complexity data (task data dimension), and the scene complexity data (spatial data dimension). In this step, each dimension of data is encoded using the Transformer encoder separately.

[0082] In step 3032, a multi-head attention mechanism is used to interactively fuse the features of the three dimensions output by the encoder.

[0083] In step 3033, based on the fused features, the multi-layer perceptron is used to output the predicted comprehensive complexity for a period of time in the future.

[0084] In step 304, a workload model is adaptively selected based on the comprehensive complexity. This step can refer to step 204 in the fifth embodiment and will not be described in detail here.

[0085] This embodiment, building on the existing data streams that only assess scene complexity and robot task type complexity, also introduces a time-dimensional stream of robot motion data. This three-dimensional data is calculated using an established prediction model to obtain an assessment of the overall complexity for a certain period of time in the future (e.g., within the next three seconds), and preloads the corresponding workload model. If the overall complexity exceeds a certain value, it may indicate that the scene and task operations are about to enter a stage of high demand, high dynamics, and high complexity, requiring the preload of the full model. Alternatively, if the predicted overall complexity is low, a higher-level model (e.g., currently at a medium or full level) is released in advance, and a lightweight model is preloaded. This embodiment facilitates rapid preload of models, improving the robot's response speed and predictive ability to dynamic scenes, tasks, and motion states, further enhancing the accuracy and effectiveness of final model selection.

[0086] The seventh embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the integrated circuit process deviation sensitivity analysis method described in any embodiment of the present application are implemented.

[0087] It should be noted that the computer storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof. In a possible embodiment, the present invention may also provide a method of implementing data processing in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to perform several steps of the method described in any of the aforementioned embodiments.

[0088] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A mobile robot control system based on a device-side model, characterized in that: include: Data acquisition module: used to acquire multimodal perception data and task types, wherein the multimodal perception data includes visual data, lidar data, and inertial measurement unit data; Calculation module: used to calculate the scene complexity according to the multimodal perception data; calculate the task complexity according to the task type, and calculate the comprehensive complexity according to the task complexity and the scene complexity; A model selection module, configured to adaptively select a client-side workload model based on the comprehensive complexity; Among them, the task types include at least: inspection, transplanting navigation, docking and positioning of robots and other equipment, beat time variation tolerance, visual secondary positioning, and end-manipulator operation; according to the task type's requirements for navigation accuracy and obstacle avoidance dynamic response, a correspondence between the task type and the task complexity is established, and the task complexity is calculated based on the correspondence.

2. The mobile robot control system according to claim 1, characterized in that: The workload model is divided into a lightweight model, a medium-weight model and a full-scale model according to the type of task performed by the robot and the requirements for navigation accuracy and dynamic obstacle avoidance.

3. The mobile robot control system according to claim 1, characterized in that: The model selection module includes a prediction submodule and a model selection submodule; The data acquisition module is also used to acquire motion data of the robot; The calculation module is further used to calculate time data of the motion data based on time; The prediction submodule is used to predict the comprehensive complexity based on the time data, the task complexity and the scene complexity; The model selection submodule is further configured to adaptively select an end-side workload model according to the comprehensive complexity.

4. The mobile robot control system according to claim 3, characterized in that: A prediction model is pre-built in the prediction submodule; The prediction submodule predicts the comprehensive complexity by inputting the time data, the task complexity and the scene complexity into the prediction model.

5. A mobile robot control method based on a device-side model, applied to the mobile robot control system according to any one of claims 1 to 4, characterized in that: include: Acquire multimodal perception data and task types; The multimodal perception data includes visual data, lidar data, and inertial measurement unit data; Calculating scene complexity based on the multimodal perception data; calculating task complexity based on the task type; Calculating comprehensive complexity according to the task complexity and the scenario complexity; Adaptively selecting a workload model on the end side according to the comprehensive complexity; Among them, the task types include at least: inspection, transplanting navigation, docking and positioning of robots and other equipment, beat time variation tolerance, visual secondary positioning, and end-manipulator operation; according to the task type's requirements for navigation accuracy and obstacle avoidance dynamic response, a correspondence between the task type and the task complexity is established, and the task complexity is calculated based on the correspondence.

6. The mobile robot control method according to claim 5, characterized in that: The method further comprises: Acquiring motion data of the robot; Calculating the motion data based on time data; Predicting comprehensive complexity based on the time data, the task complexity, and the scenario complexity; An end-side workload model is adaptively selected according to the comprehensive complexity.

7. The mobile robot control method according to claim 6, characterized in that: The predicting of comprehensive complexity according to the time data, the task complexity data, and the scenario complexity data includes: The comprehensive complexity is predicted by inputting the time data, the task complexity, and the scenario complexity into a pre-built prediction model.

8. The mobile robot control method according to claim 7, characterized in that: The predicting of the comprehensive complexity by inputting the time data, the task complexity, and the scenario complexity into a pre-built prediction model includes: Encoding the three-dimensional data of the time data, the task complexity data, and the scene complexity data using a Transformer encoder; Use a multi-head attention mechanism to interactively fuse the features of the three dimensions output by the encoder; Based on the fused features, the comprehensive complexity in the future period is output through the multi-layer perceptron.

9. The mobile robot control method according to claim 5, characterized in that: The workload model is divided into a lightweight model, a medium-weight model and a full-scale model according to the type of task performed by the robot and the requirements for navigation accuracy and dynamic obstacle avoidance.

10. The mobile robot control method according to claim 5, characterized in that: The calculating the scene complexity according to the multimodal perception data includes: Performing feature extraction on the multimodal perception data; The scene complexity is calculated based on the data extracted from the features using the following formula: , in, , , represents the weight factor, Indicates the complexity of the visual image, represents the complexity of the lidar, Represents the complexity of the inertial measurement unit.

11. The mobile robot control method according to claim 10, characterized in that: Adaptively selecting a workload model according to the complexity of the scenario includes: When the scene complexity When the scene complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity of the scene is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected.

12. The mobile robot control method according to claim 5, characterized in that: Calculating the comprehensive complexity based on the task complexity and the scene complexity includes: calculating the comprehensive complexity by the following formula : in, and represents the weight factor, Indicates the scene complexity, Indicates the complexity of the task.

13. The mobile robot control method according to claim 12, characterized in that: The adaptively selecting a workload model according to the comprehensive complexity includes: When the comprehensive complexity When the complexity is less than or equal to the first threshold, the lightweight model is selected. When the complexity is greater than the first threshold and less than the second threshold, the medium-sized model is selected. When it is greater than or equal to the second threshold, the full-scale model is selected.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the mobile robot control method according to any one of claims 5 to 13 are implemented.

Images