Robot motion control method, platform, electronic device and storage medium combined with image processing

Abstract

The present invention discloses a robot motion control method, platform, electronic device and storage medium combined with image processing, and relates to the field of mechanical control technology. The method includes: establishing interaction between ROS2 and a planning open source library through a software interface, building a layered modular architecture, and realizing communication at each layer with a communication protocol; receiving planning tasks and controlling the front-end image sensor to collect field of view images, and transmitting them back to the architecture for layered analysis and motion planning: the hardware abstraction layer converts the format of image data; the planning layer formulates control strategies based on image data and constraints; the execution layer converts the strategies into control instructions, and performs real-time motion control according to the characteristics of the actuator. The present invention solves the technical problems of response delay, insufficient uncertainty processing capability, and low task allocation efficiency in traditional robot systems in the prior art, and achieves the technical effect of improving the robot's adaptability in a dynamic environment, task coordination efficiency, and accuracy of task execution.

Description

Technical Field

[0001] The present invention relates to the field of mechanical control technology, and in particular to a robot motion control method, platform, electronic device and storage medium combined with image processing. Background Art

[0002] Traditional robot control systems usually rely on preset path planning and simple sensor feedback, which makes it difficult to adapt to complex and dynamic working environments. Although robot operating systems (such as ROS2) provide an infrastructure for robot control, with sensor data collection, hardware abstraction and communication management functions, they still have limitations in intelligent task planning and multi-robot collaboration.

[0003] In addition, existing motion control systems lack the ability to effectively handle uncertainties in the environment, especially in multi-robot collaborative work scenarios. Due to dynamic changes in the environment (such as obstacle movement or task priority changes), a single robot or fixed path planning is difficult to deal with these uncertainties in real time. At the same time, the current mainstream motion control methods lack flexibility in the decomposition, task allocation and real-time optimization of complex tasks, especially task planning algorithms often cannot flexibly handle the balance problems of multiple objectives and multiple constraints. Summary of the invention

[0004] The present application provides a robot motion control method, platform, electronic device and storage medium combined with image processing, which are used to solve the technical problems in the prior art that traditional robot systems have response delays, insufficient uncertainty processing capabilities, and low task allocation efficiency in dealing with environmental changes and multi-robot collaboration.

[0005] In a first aspect of the present application, a robot motion control method combined with image processing is provided, the method comprising: establishing interaction between ROS2 and a planning open source library based on a software interface, and constructing an interactive architecture, wherein ROS2 is a robot operating system, and the interactive architecture is a hierarchical modular architecture, and interactive communication is performed using a communication protocol; receiving a planning task, controlling a front-end image sensor, collecting a field of view image of the robot, transmitting the field of view image back to the interactive architecture, performing hierarchical analysis and motion planning, and performing robot motion control in response to an actuator; wherein performing hierarchical analysis and motion planning comprises: performing format conversion on the field of view image based on a hardware abstraction layer of the interactive architecture, and determining image conversion data, wherein the format conversion is guided by a computing processing feature based on the planning open source library, and the hardware abstraction layer has a direct interaction with the sensor and the actuator; performing control decision-making on the image conversion data under planning objectives and constraints based on the planning layer of the interactive architecture, and determining a control planning strategy, wherein the planning layer is integrated with the planning open source library; converting the control planning strategy into a control instruction for an execution objective based on the execution layer of the interactive architecture, and responding to the actuator, using the recognition feature of the actuator as the basis for conversion.

[0006] The second aspect of the present application provides a robot motion control platform combined with image processing, the platform comprising: an interactive architecture building module, the interactive architecture building module is used to establish the interaction between ROS2 and the planning open source library based on the software interface, and build an interactive architecture, wherein ROS2 is a robot operating system, and the interactive architecture is a layered modular architecture, which uses a communication protocol for interactive communication; a robot motion control module, the robot motion control module is used to receive planning tasks, control the front-end image sensor, collect the robot's field of view image, transmit the field of view image back to the interactive architecture, perform layered analysis and motion planning, and perform robot motion control in response to the actuator; wherein the robot motion control module also includes: an image format conversion unit, the image format conversion unit The image format conversion unit is used to perform format conversion on the field of view image based on the hardware abstraction layer of the interactive architecture, and determine the image conversion data, wherein the format conversion is guided by the computing processing characteristics based on the planning open source library, and the hardware abstraction layer establishes direct interaction with sensors and actuators; the control decision unit is used to perform control decisions on the image conversion data under planning objectives and constraints based on the planning layer of the interactive architecture, and determine the control planning strategy, and the planning layer is integrated with the planning open source library; the control execution unit is used to convert the control planning strategy into a control instruction of the execution target based on the execution layer of the interactive architecture, and respond to the actuator, with the recognition characteristics of the actuator as the basis for conversion.

[0007] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, the processor being coupled to a memory, the memory being used to store a program, and when the program is executed by the processor, the system is enabled to execute the method described in the first aspect.

[0008] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] The robot motion control method, platform, electronic device and storage medium combined with image processing provided by the present application relate to the field of mechanical control technology, by constructing a hierarchical modular interactive architecture between ROS2 and the planning open source library, and realizing data transmission through a communication protocol, receiving planning tasks, collecting image data, converting through the hardware abstraction layer, the planning layer formulates a control strategy, and the execution layer converts the strategy into a control instruction and drives the actuator, so as to realize the precise motion control of the robot in a complex environment. The technical problems of response delay, insufficient uncertainty processing ability and low efficiency of task allocation in the traditional robot system in the prior art in dealing with environmental changes and multi-robot collaboration are solved, and a hierarchical modular architecture is constructed by deeply integrating image processing technology with ROS2 and the planning open source library, so that the robot can not only perceive and analyze the environmental information in the visual field, but also adjust the motion control strategy in real time based on image data. This design can optimize the robot motion path, realize intelligent task allocation and dynamic collaborative planning, and improve the technical effects of adaptability, task collaboration efficiency and task execution accuracy in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram of a robot motion control method combined with image processing provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of the structure of a robot motion control platform combined with image processing provided in an embodiment of the present application;

[0014] Figure 3 A structural schematic diagram of an electronic device is provided for the present application.

[0015] Explanation of the reference numerals: interactive architecture building module 10, robot motion control module 20, image format conversion unit 21, control decision unit 22, control execution unit 23, electronic device 300, memory 301, processor 302, communication interface 303, bus architecture 304. DETAILED DESCRIPTION

[0016] The present application provides a robot motion control method, platform, electronic device and storage medium combined with image processing, which are used to solve the technical problems in the prior art that traditional robot systems have response delays, insufficient uncertainty processing capabilities, and low task allocation efficiency in dealing with environmental changes and multi-robot collaboration.

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0018] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices.

[0019] Embodiment 1, as Figure 1 As shown, the present application provides a robot motion control method combined with image processing, the method comprising:

[0020] P10: Based on the software interface, establish the interaction between ROS2 and the planning open source library, and build an interactive architecture, where ROS2 is the robot operating system, and the interactive architecture is a layered modular architecture that uses a communication protocol for interactive communication.

[0021] It should be understood that by designing a software interface, deep interaction between ROS2 and the planning open source library is achieved, and a layered modular architecture suitable for robot task execution is constructed. Among them, ROS2 (Robot Operating System 2) is the robot operating system, which is responsible for providing robot hardware abstraction, device drivers, underlying communications and various tool support, while the planning open source library is mainly responsible for the core algorithms of motion planning and task planning, realizing the intelligent path and task optimization of the robot.

[0022] The interactive architecture adopts a layered modular architecture design, which enables the system to flexibly respond to different task requirements and complex environments. Specifically, the entire system is divided into three levels: hardware abstraction layer, planning layer, and execution layer. This layered design is not only easy to maintain and expand, but also enhances the independence of each module. The layered architecture completes the data flow and instruction transmission between layers through a set of efficient communication protocols, making the data transmission and processing more stable and real-time.

[0023] In order to ensure efficient interaction between ROS2 and the planning open source library, this solution introduces a communication protocol to support fast and reliable data transmission. The communication protocol refers to a set of rules used to define the data exchange format and transmission method, and the protocol is used to achieve accurate data transmission between different layers. For example, the sensor data (such as images, distances, etc.) and control instructions (such as robot speed, direction, etc.) transmitted between ROS2 and the planning library must comply with the standards of the communication protocol to ensure compatibility between the two in data flow.

[0024] In data interaction, the software interface plays the role of a bridge. The software interface is a technical method that establishes a logical connection with ROS2 and the planning library through code. The interface is responsible for converting the hardware information collected by ROS2 (such as image data or lidar data) into a format that can be processed by the planning library, forming an adaptive format conversion function. The implementation of the format conversion function is crucial to the real-time processing of data, and determines the smooth flow of information and data compatibility between ROS2 and the planning library.

[0025] The hierarchical modular architecture design brings high flexibility and scalability to the system. When the robot tasks are increased or the hardware is updated, there is no need to make major changes to the overall architecture. Only modules need to be added or adjusted in the corresponding layer. In terms of implementation, this architecture relies on communication protocols for interaction between modules. Modules at different layers are effectively functionally decoupled through interface standards. Each module is independent and can achieve parallel processing. This design can optimize the system response speed, reduce resource consumption, and improve the reliability and efficiency of task execution.

[0026] P20: Receive planning tasks, control the front-end image sensor, collect the robot's field of view image, transmit the field of view image back to the interactive architecture, perform hierarchical analysis and motion planning, and respond to the actuator to control the robot's motion.

[0027] Among them, hierarchical analysis and motion planning include:

[0028] P21: Based on the hardware abstraction layer of the interactive architecture, the field of view image is format converted to determine the image conversion data, wherein the format conversion is guided by the computational processing characteristics based on the planning open source library, and the hardware abstraction layer establishes direct interaction with sensors and actuators; P22: Based on the planning layer of the interactive architecture, the image conversion data is subjected to control decisions under planning objectives and constraints to determine the control planning strategy, and the planning layer is integrated with the planning open source library; P23: Based on the execution layer of the interactive architecture, the control planning strategy is converted into control instructions for the execution objectives, and the recognition characteristics of the actuator are responded to and used as the basis for conversion.

[0029] Optionally, by receiving task requirements, the front-end image sensor is controlled to collect the field of view image in real time, that is, the environment image within the robot's current field of view. The image data is transmitted back to the interactive architecture and enters the hierarchical processing flow, thereby achieving efficient hierarchical analysis and motion planning. The field of view image is a key source of information for the robot to perform motion control. It not only provides visual information such as obstacles and target objects in the environment, but also supports subsequent intelligent planning and decision-making.

[0030] First, at the hardware abstraction layer, the format of the field of view image is converted. Format conversion refers to converting the image data from the original data format of the sensor into a standardized format that can be directly processed by the planning layer. The conversion process refers to the computing and processing features of the planning open source library to ensure that the image data is seamlessly compatible with the computing requirements of the planning library. For example, the image may be converted into a grid map format to meet the path planning algorithm's understanding of the environment. The hardware abstraction layer is not only responsible for data processing of sensors (such as image sensors), but also establishes a direct interactive relationship with the actuator, thereby achieving real-time and accuracy of image acquisition and data processing.

[0031] Next, in the planning layer, the converted image data is task planned. The planning layer integrates the planning open source library, which can make reasonable control decisions based on planning goals and constraints, and generate control planning strategies suitable for the current environment. Planning goals include the final requirements of the task (such as reaching a specified location), while constraints may involve obstacles in the environment, robot motion restrictions, etc. The planning layer quickly generates optimized paths or task instructions based on image data analysis by calling the advanced algorithms of the open source library.

[0032] Finally, the execution layer further converts the planning strategy into specific control instructions for the robot. Relying on the interactive architecture, the execution layer is directly connected to the actuator, interpreting the control strategy into a signal format that the actuator can understand (such as speed, direction, etc.), and accurately converting it based on the identification characteristics of the actuator to ensure the accuracy of the instruction transmission. The identification characteristics of the actuator refer to the characteristic parameters of different actuators (such as the rotation angle range of the robot arm, the speed control of the motor, etc.). The execution layer uses these characteristics to generate specific control signals to achieve real-time movement of the robot.

[0033] Through the layer-by-layer processing of the interactive architecture, a closed loop from image data acquisition to motion control is achieved. In this step, the acquisition of visual images and hierarchical planning processing ensure the robot's ability to quickly respond to tasks in complex environments, thereby achieving intelligent and high-precision motion control.

[0034] Furthermore, before making control decisions under planning objectives and constraints on the image conversion data, step P22 of the embodiment of the present application further includes:

[0035] P22-1: The planning layer includes multifunctional modules, wherein the planning layer is expandable, the multifunctional modules are relatively independent, and each functional module is configured with an independent interface; P22-2: Identify planning tasks, traverse the multifunctional modules, and match target functional modules; P22-3: Based on the target functional module, make control decisions on the image conversion data under planning objectives and constraints.

[0036] It should be understood that the planning layer further enhances its modular design before making control decisions under planning objectives and constraints on the image transformation data, allowing the system to flexibly adapt to different task requirements and efficiently generate control strategies.

[0037] Specifically, the planning layer contains multiple functional modules, each of which focuses on a specific type of planning task, such as path planning, obstacle avoidance planning or speed control. The key to module design lies in its scalability and relative independence. Scalability refers to the ability to add or modify modules as needed to adapt to new task requirements, while relative independence ensures the decoupling between modules, facilitating separate development and debugging. At the same time, each functional module is configured with independent interfaces, which are bridges for communication between the modules and other parts of the planning layer, ensuring smooth data transmission and correct execution of task instructions. For example, the interface of the path planning module can receive environmental image data and output path information, while the obstacle avoidance module outputs decisions to adjust the path after receiving obstacle data.

[0038] During task execution, the planning task is first identified to clarify the specific requirements of the current task. Then, by traversing the functional modules in the planning layer, the module that best matches the task requirements is found and used as the target functional module. For example, when the task requires the robot to avoid specific obstacles and reach the target location, the system will identify the obstacle avoidance module and the path planning module as the priority matching modules. Module matching ensures that the system can quickly call the appropriate algorithms and resources, avoid unnecessary module calls, and improve the system's operating efficiency.

[0039] Once the target functional module is determined, the image conversion data is analyzed in depth based on the module to make precise control decisions based on the planning objectives and constraints. The planning objectives are the specific execution requirements of the task (such as the shortest path or the minimum energy consumption), while the constraints may involve the task environment (such as obstacle location, robot load, etc.) and robot performance limitations (such as maximum speed, turning radius). The target module generates control strategies based on these input data to ensure that the robot can complete the task efficiently. For example, in the path planning module, the control decision will dynamically adjust the robot path based on the planning objectives (such as the shortest path length) and obstacle constraints to form the optimal motion plan.

[0040] Through the above modular design, the system not only has high flexibility and adaptability, but also realizes the efficient calling of multifunctional modules, so that the robot can accurately perform tasks in complex environments. This modular design provides the robot with intelligent task planning and real-time response capabilities, greatly improving the overall performance of the system and the reliability of task completion.

[0041] Further, based on the target function module, the image conversion data is subjected to control decisions under planning objectives and constraints. Step P22-3 of the embodiment of the present application further includes:

[0042] P22-31: Based on the planning task, determine the planning objectives and constraints, wherein the planning objectives and the constraints are at least one item, and the planning objectives correspond to an objective function; P22-32: Based on the planning task, determine the motion control variables, and the motion control variables are divided into discrete variables and continuous variables; P22-33: Based on the planning objectives and the constraints, and with an internal competitive balance between the planning objectives, perform heuristic search optimization on the motion control variables to determine the control planning strategy.

[0043] Optionally, the control decision generation process can be further refined to ensure that the robot can make optimized motion plans in complex tasks and environments.

[0044] First, the planning objectives and constraints are clearly defined based on the planning task. The planning objective is the final effect required to be achieved by the task, such as the shortest path, minimum energy consumption, or minimum collision risk. Each planning objective usually corresponds to an objective function, which is a mathematical expression used to measure the effectiveness of task completion. For example, if the goal is the shortest path, the objective function may be the cumulative value of the distance. Constraints are used to limit the execution scope or limiting factors of the task, such as the robot's travel speed, obstacle location, etc. Constraints are conditions that cannot be violated in the task and must be strictly followed in control decisions. Through comprehensive consideration of objectives and constraints, the system achieves effective decomposition of tasks under multiple task requirements.

[0045] Furthermore, after identifying the planning objectives and constraints, the motion control variables, that is, the specific adjustable parameters that affect the robot's motion, are further confirmed. Motion control variables are divided into discrete variables and continuous variables. Discrete variables refer to variables with a clear set of choices, such as whether the robot turns or starts or stops, while continuous variables include parameters such as speed and angle that can change continuously within a specific range. The purpose of dividing discrete and continuous variables is to flexibly handle different types of control requirements and effectively reduce the complexity of the algorithm. For example, when selecting a path, the turning direction can be regarded as a discrete variable, while the speed adjustment is a continuous variable. This division enables the system to apply different optimization strategies more accurately.

[0046] After all planning objectives and constraints are defined, the motion control variables are optimized through heuristic search optimization to determine the final control planning strategy. Heuristic search is a method that guides the search path by experience or rule priority. It can quickly find a near-optimal solution in a multi-objective scenario. The system finds an optimal solution in an internal competitive balance between multiple planning objectives, that is, it weighs the objective functions to ensure that all objectives are met to some extent. For example, when the task requires the shortest path and the minimum energy consumption, the system will balance the two so that the control strategy is close to the shortest path without consuming too much energy. The final generated control planning strategy contains the optimal combination of motion control variables, enabling the robot to complete efficient and smooth task execution under the planning task and environmental constraints.

[0047] Through the above steps, the task requirements are decomposed into multiple levels of objectives and control variables, and the heuristic optimization strategy is used to balance the planning objectives, thereby generating a control planning strategy that meets the multi-objective requirements. This process greatly improves the system's adaptability and decision-making efficiency in complex task environments, enabling the robot to respond to changes quickly and intelligently and achieve efficient task execution.

[0048] Furthermore, after determining the control planning strategy, step P22 of the embodiment of the present application further includes:

[0049] P22-4: Call historical control data and environmental data, mine uncertainties, and supervise the training of probability distribution models by introducing probability distributions; P22-5: Identify the planning tasks and the image conversion data, perform task uncertainty analysis based on the probability distribution model, and output uncertainties probabilities; P22-6: Compensate for the control planning strategy based on the uncertainties probabilities.

[0050] Specifically, after determining the control planning strategy, the ability to cope with uncertainties in a dynamic environment can be further enhanced by introducing analysis of historical and environmental data.

[0051] First, historical control data and environmental data are called to explore possible uncertainties in task execution. Historical control data is the execution record of the robot in previous tasks, including sensor data, execution errors and other information; environmental data contains the dynamic characteristics of the task environment, such as the frequency of obstacles, changes in path accessibility, etc. By analyzing these data, the system can identify uncertainties that may cause planning deviations during the task.

[0052] Subsequently, a supervised training method is used to construct a probability distribution model, that is, based on the statistical characteristics of uncertain factors, a mathematical model that can reflect the distribution law of these factors is established. For example, the probability of a robot encountering an obstacle in different environments may vary depending on the complexity of the environment. Through the probability distribution model, the system can quantify the possibility of the occurrence of uncertain factors, thereby providing data basis for subsequent compensation strategies.

[0053] After formulating the control planning strategy, the current planning task and image conversion data are identified and input into the probability distribution model for task uncertainty analysis. Task uncertainty analysis is a process of evaluating potential risks in the execution of a task. Through this analysis, it is possible to identify which uncertainties may appear in the current task. Based on the calculation of the probability distribution model, the probability of uncertainty factors is output, that is, the probability of each uncertain factor occurring in the current task environment. These probabilities provide a quantitative risk assessment for subsequent decision-making, which can help the robot system to take precautions or dynamically adjust high-risk uncertainties in advance. For example, in path planning, if the probability of obstacles appearing in a certain section of the path is high, the robot system can mark the path as a high-risk area.

[0054] Finally, the determined control planning strategy is compensated based on the probability of uncertain factors. Compensation refers to adding adjustment measures to deal with uncertainty on the basis of the original control strategy. Specifically, compensation methods can include path fine-tuning, task priority adjustment, etc. For example, if there is a high probability of dynamic obstacles appearing on the robot's path, the system can adjust the path or reduce the travel speed to ensure the smooth completion of the task. The implementation of the compensation strategy enables the control planning strategy to adapt more flexibly to dynamic changes in the actual environment, enhances the robustness of the planning strategy, and thus improves the execution accuracy and safety of the robot in an uncertain environment.

[0055] Through the above steps, using uncertainty analysis and compensation mechanisms, the control strategy can dynamically adapt to environmental changes, improving the stability and reliability of task execution. After the strategy is formulated, the uncertainties are quantified and compensated, so that the robot can respond to emergencies more intelligently and flexibly in complex environments and achieve robust task execution.

[0056] Further, based on the execution layer of the interactive architecture, the control planning strategy is converted into a control instruction for executing the target, and after responding to the actuator, step P23 of the embodiment of the present application further includes:

[0057] P23-1a: Execute the control instructions, monitor the execution process of the robot, and determine the execution tracking data; P23-2a: Feedback the execution tracking data to the planning layer, determine the motion control data by matching functional modules and making adjustment decisions, and generate adjustment instructions based on the execution layer conversion to perform feedback control of the robot motion control.

[0058] Optionally, the execution layer based on the interactive architecture converts the generated control planning strategy into specific control instructions, and performs real-time monitoring and feedback control during the execution process. Through continuous feedback loops and intelligent adjustment mechanisms, the accuracy and reliability of robot motion control are ensured.

[0059] Specifically, after the control instruction is generated, the instruction is sent to the robot's actuator through the execution layer, so that the robot starts to move according to the planned path and behavior pattern. At this time, the system monitors the robot's movement process in real time to ensure that the execution process meets the requirements of the planning strategy. During the monitoring process, the robot's execution tracking data is collected and analyzed. The data contains the robot's actual position information, movement speed, posture angle and other operating states, as well as deviation information from the expected control strategy. The real-time acquisition and processing of execution tracking data enables the system to maintain high-precision tracking of the robot's state during task execution.

[0060] Next, the collected execution tracking data is fed back to the planning layer, where functional module matching and adjustment decisions are made. Functional module matching means that the system identifies the most suitable planning module for handling the current situation based on the feedback content of the tracking data. For example, when it is found that the robot deviates from the preset path, the system may prioritize matching the path planning module for adjustment.

[0061] After the functional modules are matched, motion control data is generated to guide the dynamic adjustment of the control strategy. The motion control data contains compensation parameters and correction information, which can help the execution layer to further refine the control instructions, so that the robot movement is closer to the ideal state. The motion control data is converted by the execution layer to generate adjustment instructions, which are precisely regulated based on the real-time status of the robot.

[0062] Through this feedback control process, a closed-loop mechanism is formed, that is, continuous monitoring, feedback and adjustment during the execution of instructions. This closed-loop mechanism ensures that the robot can respond quickly and make adjustments when encountering external interference or task deviation. For example, when the robot detects an obstacle or path deviation during movement, the system will immediately adjust the control parameters during the feedback control process and update the instructions to bypass the obstacle or return to the preset path.

[0063] This feedback control mechanism greatly improves the reliability and flexibility of the robot's task execution, enabling the system to quickly respond to environmental changes and autonomously correct deviations, thereby achieving high-precision motion control. This process significantly improves the stability and adaptability of the system while ensuring task completion.

[0064] Furthermore, the embodiment of the present application further includes step P30, and step P30 further includes:

[0065] P31: If the planned task is a designated task, a motion control decision is made based on the image conversion data, wherein the execution robot is taken as the designated target and the source of the image conversion data is the execution robot; P32: If the planned task is an undesignated task, a two-step motion decision is executed, which includes a one-step decision based on task decomposition and allocation and a two-step decision based on robot motion control decision. Under the undesignated task, the source of the image conversion data is a cluster robot.

[0066] In a possible embodiment of the present application, the robot's decision-making process in designated tasks and undesignated task scenarios can be further refined to ensure that the system can adapt to the needs of different task types and complete tasks efficiently and accurately.

[0067] First, when the planned task is identified as a designated task, the task goal is clearly directed to a specific execution robot. A designated task means that the task requirements are highly concentrated on the execution capabilities of a single robot, such as delicate assembly, precise material handling, and other tasks that require a single robot to complete.

[0068] In a specified task scenario, the image conversion data of the executing robot is directly collected through the front-end image sensor, that is, the visual image information specifically for the robot. The specificity of this data source enables the image conversion data to more directly reflect the working status and environmental characteristics of the executing robot, providing accurate input data for motion control decisions. Based on this data, the system can quickly make motion control decisions that meet the task requirements, so that the executing robot can complete the task according to the set goals.

[0069] Secondly, when the task type is identified as a non-specified task, more extensive planning and allocation of tasks are required. Non-specified tasks refer to task requirements that are not concentrated on a single robot, and usually require multiple robots to work together to complete. For example, tasks such as moving large quantities of items in a warehouse or conducting large-scale search and rescue at a disaster site are all non-specified tasks. In this scenario, preliminary task allocation and planning are first performed based on the image conversion data collected by the cluster robot. The image conversion data brings together the environmental perception information of multiple robots in a wide area, which facilitates the system to evaluate task requirements from a global perspective. Subsequently, the system performs a two-step motion decision to ensure the reasonable decomposition and efficient execution of tasks:

[0070] Among them, the first step is decision-making: first, the task is decomposed and allocated, that is, based on the task requirements and the status of each robot, the task is divided into multiple subtasks, and these subtasks are allocated to the most suitable robot. For example, for large-scale material handling tasks, the system will reasonably allocate tasks to multiple robots based on the different load capacities of each robot to achieve optimal load balance.

[0071] Second-step decision-making: After the task assignment is completed, specific motion control decisions are made based on the current motion state of each robot and the task requirements. This control decision enables each robot to complete the assigned task while complying with the global path planning and obstacle avoidance strategy to avoid mutual interference and ensure collaborative efficiency. The motion control decision takes into account the real-time motion state of the robot under a specific task, and realizes the refined management of the task completion process.

[0072] Through the above-mentioned designated and non-designated task processing flow, task type identification and multi-level decision-making mechanisms can be realized, making it highly adaptable and flexible in both task division and execution. Whether it is a task led by a single robot or a task coordinated by multiple robots, the solution can reasonably call image data and task allocation strategies to ensure the efficiency and reliability of task execution.

[0073] Furthermore, the two-step motion decision is performed, and step P32 of the embodiment of the present application further includes:

[0074] P32-1: If it is a non-specified task, the robot information is determined by analyzing the image conversion data, wherein the image conversion data identifies the robot's unique code, and the robot information includes real-time status and specification capabilities; P32-2: Based on the robot information, combined with the planning layer, task decomposition and collaborative task planning are performed to determine the task allocation result, and the task allocation result includes a robot code-decomposition task with a mapping relationship; P32-3: Execute motion control decisions based on the task allocation result.

[0075] In a possible embodiment of the present application, the specific process of the two-step motion decision is further refined to efficiently manage the collaborative work of multiple robots in a non-specified task environment, making task allocation more reasonable and execution more precise.

[0076] In the scenario of non-specified tasks, the current robot information of each robot is first extracted by analyzing the image conversion data. The image conversion data contains real-time image data of the robot's environment, and each robot has a unique code as the identifier of the robot. This unique code helps the system accurately locate and identify each robot in the cluster. Through this data analysis, the real-time status (such as current position, movement speed) and specification capacity (such as load capacity, battery power, etc.) of the robot can be obtained. This information is an important basis for task allocation and motion decision-making, because the status and capabilities of each robot directly affect its suitable task type and execution method.

[0077] Furthermore, based on the acquired robot information, the tasks are decomposed and collaboratively planned in combination with the planning layer to generate task allocation results. Task decomposition refers to splitting complex non-specified tasks into subtasks suitable for collaborative execution by multiple robots. Through collaborative task planning, the optimal task allocation scheme is determined based on the capabilities and current status of each robot. During the allocation process, the task allocation result matches the task with the unique code of the robot to form a mapping relationship, that is, each subtask corresponds to a specific robot. This mapping relationship ensures the reasonable allocation of tasks and avoids task duplication or waste of resources. For example, for a large handling task, heavy object handling can be assigned to a robot with a stronger load capacity, while light work can be assigned to a robot with a lower load capacity, ensuring that all robots work efficiently within their capabilities.

[0078] Finally, the corresponding motion control decision is executed for each robot based on the task assignment results. Motion control decision refers to the system generating specific motion instructions (such as speed, path, obstacle avoidance, etc.) for each robot's assigned task to guide the robot to complete the task. The control decision takes into account each robot's current position, target position, and relative position to other robots, so as to ensure that the entire cluster is coordinated and consistent during execution without interfering with each other. For example, when multiple robots need to work together in a narrow channel, the motion control decision will adjust their movement order and path to avoid conflicts and ensure traffic efficiency.

[0079] Through the above detailed steps, the system can accurately obtain robot information in a non-specified task environment, reasonably allocate tasks and make motion control decisions, and achieve high efficiency and accuracy of multi-robot collaborative work. The entire process ensures that the system has a high degree of adaptability and flexibility in multi-target tasks, so that the robot can flexibly respond to complex task requirements and significantly improve the efficiency and quality of task completion.

[0080] In summary, the embodiments of the present application have at least the following technical effects:

[0081] This application builds a layered modular interactive architecture between ROS2 and the planning open source library, and realizes data transmission through a communication protocol. After receiving planning tasks and collecting image data, it is converted by the hardware abstraction layer. The planning layer formulates control strategies, and the execution layer converts the strategies into control instructions and drives the actuators, thereby realizing precise motion control of the robot in complex environments.

[0082] The technical effect of improving the robot's adaptability, task coordination efficiency and accuracy in task execution in dynamic environments by building a hierarchical modular architecture based on ROS2 and planning open source libraries and combining image processing with motion control has been achieved.

[0083] Embodiment 2, based on the same inventive concept as the robot motion control method combined with image processing in the above embodiment, Figure 2 As shown, the present application provides a robot motion control platform combined with image processing, and the platform and method embodiments in the present application embodiments are based on the same inventive concept. Among them, the platform includes:

[0084] The interactive architecture building module 10 is used to establish the interaction between ROS2 and the planned open source library based on the software interface and build an interactive architecture, wherein ROS2 is a robot operating system, and the interactive architecture is a layered modular architecture, which uses a communication protocol for interactive communication.

[0085] The robot motion control module 20 is used to receive planning tasks, control the front-end image sensor, collect the robot's field of view image, transmit the field of view image back to the interactive architecture, perform hierarchical analysis and motion planning, and perform robot motion control in response to the actuator.

[0086] Wherein, the robot motion control module 20 also includes:

[0087] An image format conversion unit 21, wherein the image format conversion unit 21 is used to perform format conversion on the field of view image based on the hardware abstraction layer of the interactive architecture, and determine image conversion data, wherein the format conversion is guided by the computing processing characteristics based on the planned open source library, and the hardware abstraction layer establishes direct interaction with sensors and actuators.

[0088] The control decision unit 22 is used to make control decisions on the image conversion data under planning objectives and constraints based on the planning layer of the interactive architecture, and determine the control planning strategy. The planning layer is integrated with the planning open source library.

[0089] The control execution unit 23 is used to convert the control planning strategy into a control instruction for executing a target based on the execution layer of the interactive architecture, and respond to the actuator, taking the identification characteristics of the actuator as the basis for conversion.

[0090] Furthermore, the control decision unit 22 is also used to perform the following steps:

[0091] The planning layer includes multifunctional modules, wherein the planning layer is expandable, the multifunctional modules are relatively independent, and each functional module is configured with an independent interface; the planning task is identified, the multifunctional modules are traversed, and the target functional modules are matched; based on the target functional module, the image conversion data is controlled and decided under planning objectives and constraints.

[0092] Furthermore, the control decision unit 22 is also used to perform the following steps:

[0093] Based on the planning task, the planning goal and the constraint conditions are determined, wherein the planning goal and the constraint conditions are at least one item, and the planning goal corresponds to an objective function; based on the planning task, the motion control variables are determined, and the motion control variables are divided into discrete variables and continuous variables; based on the planning goal and the constraint conditions, and with an internal competitive balance between the planning goals, the motion control variables are heuristically searched and optimized to determine the control planning strategy.

[0094] Furthermore, the control decision unit 22 is also used to perform the following steps:

[0095] Call historical control data and environmental data, mine uncertain factors, and supervise the training of probability distribution models by introducing probability distribution; identify the planning task and the image conversion data, perform task uncertainty analysis based on the probability distribution model, and output the probability of uncertain factors; based on the probability of uncertain factors, compensate the control planning strategy.

[0096] Furthermore, the control execution unit 23 is also used to execute the following steps:

[0097] Execute the control instructions, monitor the execution process of the robot, and determine the execution tracking data; feed back the execution tracking data to the planning layer, determine the motion control data by matching the functional modules and making adjustment decisions, and generate adjustment instructions based on the execution layer conversion to perform feedback control of the robot motion control.

[0098] Furthermore, the platform also includes a task classification module, and the task classification module is further used to perform the following steps:

[0099] If the planned task is a designated task, motion control decisions are made based on the image conversion data, wherein the execution robot is taken as the designated target and the source of the image conversion data is the execution robot; if the planned task is an undesignated task, a two-step motion decision is executed, which includes a one-step decision based on task decomposition and allocation and a two-step decision based on robot motion control decision. Under the undesignated task, the source of the image conversion data is the cluster robot.

[0100] Among them, if it is a non-specified task, the robot information is determined by analyzing the image conversion data, wherein the image conversion data identifies the robot's unique code, and the robot information includes real-time status and specification capabilities; based on the robot information, combined with the planning layer, task decomposition and collaborative task planning are performed to determine the task allocation result, and the task allocation result includes a robot code-decomposition task with a mapping relationship; based on the task allocation result, motion control decisions are executed.

[0101] Embodiment three, based on the same inventive concept as the robot motion control method combined with image processing in the aforementioned embodiment, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method in embodiment one is implemented.

[0102] Through the above detailed description of the robot motion control method combined with image processing, those skilled in the art can clearly understand the robot motion control method and system combined with image processing in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0103] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0104] Exemplary electronic devices, referenced below Figure 3 To describe the electronic device of the embodiment of the present application.

[0105] Based on the same inventive concept as the robot motion control method combined with image processing in the aforementioned embodiment, the present application also provides a robot motion control system combined with image processing, including: a processor, the processor is coupled to a memory, the memory is used to store a program, when the program is executed by the processor, the system executes the steps of the method described in Example 1.

[0106] The electronic device 300 includes: a processor 302, a communication interface 303, and a memory 301. Optionally, the electronic device 300 may also include a bus architecture 304. The communication interface 303, the processor 302, and the memory 301 may be interconnected via the bus architecture 304; the bus architecture 304 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus architecture 304 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0107] The processor 302 may be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0108] The communication interface 303 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), wired access networks, etc.

[0109] The memory 301 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via the bus architecture 304. The memory may also be integrated with the processor.

[0110] The memory 301 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 302. The processor 302 is used to execute the computer-executable instructions stored in the memory 301, thereby realizing the robot motion control method combined with image processing provided in the above embodiment of the present application.

[0111] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0113] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A robot motion control method combined with image processing, characterized in that: include: Based on the software interface, establish the interaction between ROS2 and the planning open source library, and build an interactive architecture, where ROS2 is the robot operating system, and the interactive architecture is a layered modular architecture, using the communication protocol for interactive communication; Receive planning tasks, control front-end image sensors, collect visual images of the robot, transmit the visual images back to the interactive architecture, perform hierarchical analysis and motion planning, and perform robot motion control in response to actuators; Among them, hierarchical analysis and motion planning include: Based on the hardware abstraction layer of the interactive architecture, the visual image is format converted to determine image conversion data, wherein the format conversion is guided by the computing processing characteristics based on the planned open source library, and the hardware abstraction layer establishes direct interaction with sensors and actuators; Based on the planning layer of the interactive architecture, control decisions are made on the image conversion data under planning objectives and constraints to determine a control planning strategy, wherein the planning layer is integrated with the planning open source library; Based on the execution layer of the interactive architecture, the control planning strategy is converted into control instructions for executing the target, and in response to the actuator, the identification characteristics of the actuator are used as the basis for conversion; The method further comprises: If the planned task is a designated task, a motion control decision is made based on the image conversion data, wherein the execution robot is the designated target and the acquisition source of the image conversion data is the execution robot; If the planned task is a non-specified task, a two-step motion decision is performed, including a one-step decision based on task decomposition and allocation and a two-step decision based on robot motion control decision. Under the non-specified task, the acquisition source of the image conversion data is a cluster robot; The performing of the two-step motion decision includes: If it is a non-specified task, the robot information is determined by analyzing the image conversion data, wherein the image conversion data is identified with a unique robot code, and the robot information includes real-time status, load capacity and battery power; Based on the robot information and in combination with the planning layer, task decomposition and collaborative task planning are performed to determine task allocation results, wherein the task allocation results include robot codes-decomposition tasks with a mapping relationship; According to the task allocation result, motion control decision is executed.

2. The robot motion control method combined with image processing as claimed in claim 1, characterized in that: Before making a control decision under planning objectives and constraints on the image conversion data, it includes: The planning layer includes multifunctional modules, wherein the planning layer is expandable, the multifunctional modules are relatively independent, and each functional module is configured with an independent interface; Identify the planning task, traverse the multifunctional modules, and match the target functional module; Based on the target function module, control decisions are made on the image conversion data under planning targets and constraints.

3. The robot motion control method combined with image processing as claimed in claim 2, characterized in that: Based on the target function module, the image conversion data is controlled and decided under planning targets and constraints, including: Based on the planning task, determining a planning goal and a constraint condition, wherein the planning goal and the constraint condition are at least one item, and the planning goal corresponds to an objective function; Based on the planning task, determining motion control variables, wherein the motion control variables are divided into discrete variables and continuous variables; Based on the planning objectives and the constraints, and on an internal competitive balance between the planning objectives, the motion control variables are heuristically searched and optimized to determine the control planning strategy.

4. The robot motion control method combined with image processing as claimed in claim 3, characterized in that: After determining the control planning strategy, it includes: Call historical control data and environmental data to explore uncertain factors, introduce probability distribution, and supervise the training of probability distribution models; Identify the planned task and the image conversion data, perform task uncertainty analysis based on the probability distribution model, and output uncertainty factor probabilities; The control planning strategy is compensated based on the uncertainty factor probability.

5. The robot motion control method combined with image processing as claimed in claim 1, characterized in that: In response to the actuator, the method comprises: Execute the control instructions, monitor the execution process of the robot, and determine the execution tracking data; The execution tracking data is fed back to the planning layer, and by matching functional modules and making adjustment decisions, the motion control data is determined and adjustment instructions are generated based on the execution layer conversion to perform feedback control of the robot motion control.

6. A robot motion control platform combined with image processing, characterized in that: For implementing the robot motion control method combined with image processing according to any one of claims 1 to 5, the platform comprises: An interactive architecture building module, wherein the interactive architecture building module is used to establish the interaction between ROS2 and the planning open source library based on the software interface and build an interactive architecture, wherein ROS2 is a robot operating system, and the interactive architecture is a layered modular architecture, and interactive communication is performed using a communication protocol; A robot motion control module, the robot motion control module is used to receive the planning task, control the front-end image sensor, collect the robot's visual field image, transmit the visual field image back to the interactive architecture, perform hierarchical analysis and motion planning, and perform robot motion control in response to the actuator; Wherein, the robot motion control module also includes: An image format conversion unit, the image format conversion unit is used to perform format conversion on the visual image based on the hardware abstraction layer of the interactive architecture, and determine image conversion data, wherein the format conversion is guided by the computing processing characteristics based on the planned open source library, and the hardware abstraction layer establishes direct interaction with sensors and actuators; A control decision unit, the control decision unit is used to make control decisions under planning objectives and constraints on the image conversion data based on the planning layer of the interactive architecture, and determine a control planning strategy, the planning layer is integrated with the planning open source library; A control execution unit is used to convert the control planning strategy into a control instruction for executing a target based on the execution layer of the interactive architecture, and respond to the actuator, taking the identification characteristics of the actuator as the basis for conversion.

7. An electronic device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store a program, when the program is executed by the processor, the system is enabled to perform the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which implements the steps of the method according to any one of claims 1 to 5 when executed by a processor.

Images