Robot control method and device, equipment, storage medium and product

Abstract

The invention discloses a robot control method and device, equipment, a storage medium and a product, and relates to the technical field of artificial intelligence, and the robot control method comprises the steps that when a control instruction is received, grabbing target information is determined according to the control instruction; moving to a target position according to the captured target information, and collecting target environment information; determining a grabbing path based on the target environment information and the grabbing target information; and grabbing a grabbing target corresponding to the control instruction according to the grabbing path. According to the method, the target position is moved according to the captured target information, and the target environment information is collected; determining a grabbing path based on the target environment information and the grabbing target information; and grabbing a grabbing target corresponding to the control instruction according to the grabbing path. And more universal and autonomous machine intelligence is achieved, and the grabbing efficiency of the robot is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to robot control methods, devices, equipment, storage media and products. Background Art

[0002] The mobile operating robot system based on embodied intelligence (referred to as "embodied intelligent mobile operating robot") aims to build a robot system with autonomous environmental perception, full understanding of cognition, smooth human-computer interaction, reliable intelligent decision-making and natural motion operation planning. Relying on a cross-domain, multi-scenario, and multi-functional autonomous embodied intelligent platform, it empowers the upgrade of traditional mobile operating robots and leads the industry development of mobile operating robots in the future. With a brain-like structure that can perceive, understand and make decisions, the embodied intelligent mobile operating robot can autonomously understand and complete high-level instructions issued by humans, realizing true general intelligence. Compared with traditional mobile robots, embodied intelligent mobile operating robots can complete some complex tasks that usually require human wisdom to complete. As its technology continues to develop and mature, it will surely bring revolutionary changes to human society. Embodied intelligent mobile operating robots have broad application prospects in civil fields such as services, catering, medical care, smart homes, unmanned delivery, industrial fields such as smart factories and smart manufacturing, and military fields such as single-soldier combat. At present, most of the research and development of embodied intelligent mobile operating robots at home and abroad are still in the laboratory testing stage, and there are certain defects in understanding and completing the instructions issued by humans. The overall technology is not mature. Therefore, how to improve the control efficiency of embodied intelligent mobile operating robots so that embodied intelligent mobile operating robots can understand human instructions and efficiently perform corresponding grasping and other tasks has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of this application is to provide a robot control method, device, equipment, storage medium and product, aiming to solve the technical problem of low execution efficiency of existing embodied intelligent mobile operation robots when performing grasping tasks.

[0004] To achieve the above objectives, the present application proposes a robot control method, which includes: When receiving a control instruction, determining the grab target information according to the control instruction; Move to the target location according to the captured target information and collect target environment information; Determine a grasping path based on the target environment information and the grasping target information; The grabbing target corresponding to the control instruction is grabbed according to the grabbing path.

[0005] Optionally, when receiving the control instruction, the step of determining to capture target information according to the control instruction includes: When receiving a control instruction, determining user demand information according to the control instruction; Collecting images of the surrounding environment; The user demand information and the surrounding environment image are input into a preset multimodal large model to obtain the grasping target information output by the preset multimodal large model, wherein the grasping target information includes the relative map position and robot posture coordinates of the grasping target corresponding to the control instruction.

[0006] Optionally, when receiving the control instruction, the step of determining the user demand information according to the control instruction includes: When receiving a control instruction, converting the control instruction into text to obtain text information; Determining whether a target wake-up word is detected according to the text information; When the target wake-up word is detected, the user demand information is determined based on the text information.

[0007] Optionally, the step of determining a grasping path based on the target environment information and the grasping target information includes: Determine the position information of the grab target to be grabbed in the target environment information according to the target environment information and the grab target information; Sending the position information and the point cloud map corresponding to the position information to a preset GraspNet model to obtain the grasping posture information output by the preset GraspNet model; A grasping path is determined according to the grasping posture information.

[0008] Optionally, the step of grabbing a grab target corresponding to the control instruction according to the grab path includes: Constructing a Lagrangian dynamics model, wherein the Lagrangian dynamics model is used to predict the stability of the robot when grasping through the grasping path; Inputting the grasping path into the Lagrangian dynamics model, the Lagrangian dynamics model simulates the stability of the robot when grasping, and outputs a prediction result; The grab target corresponding to the control instruction is grabbed according to the prediction result.

[0009] Optionally, after the step of grabbing the grab target corresponding to the control instruction according to the grab path, the method further includes: Determine the target placement position according to the captured target information; The grab target is placed at the target placement position.

[0010] In addition, to achieve the above-mentioned purpose, the present application also proposes a robot control device, the robot control device comprising: A receiving module, configured to determine the grab target information according to the control instruction when receiving the control instruction; A moving module, used to move to a target location according to the captured target information and collect target environment information; A grasping path determination module, used to determine a grasping path based on the target environment information and the grasping target information; The grabbing module is used to grab the grabbing target corresponding to the control instruction according to the grabbing path.

[0011] In addition, to achieve the above-mentioned objectives, the present application also proposes a robot control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the robot control method described above.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the robot control method described above are implemented.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the robot control method described above are implemented.

[0014] When receiving a control instruction, the present application determines the grasping target information according to the control instruction; moves to the target position according to the grasping target information and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; and grasps the grasping target corresponding to the control instruction according to the grasping path. Since the present application moves to the target position according to the grasping target information and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; and grasps the grasping target corresponding to the control instruction according to the grasping path. A more general and autonomous machine intelligence is achieved, and the robot grasping efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the robot control method of the present application; Figure 2 A schematic diagram of the structure of an embodied intelligent mobile operating robot provided in Example 1 of the robot control method of the present application; Figure 3 A schematic diagram of the vehicle chassis structure provided in Example 1 of the robot control method of this application; Figure 4 A schematic diagram of the details of the vehicle chassis structure provided in Example 1 of the robot control method of this application; Figure 5 A schematic diagram of a flow chart provided for the second embodiment of the robot control method of the present application; Figure 6 The overall flow diagram provided for the second embodiment of the robot control method of the present application; Figure 7 This is a schematic diagram of the module structure of the robot control device according to an embodiment of the present application; Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the robot control method in the embodiment of the present application.

[0018] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0020] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of the embodiment of the present application is: upon receiving a control instruction, determine the grasping target information according to the control instruction; move to the target position according to the grasping target information, and collect the target environment information; determine the grasping path based on the target environment information and the grasping target information; grasp the grasping target corresponding to the control instruction according to the grasping path. Since the present application moves to the target position according to the grasping target information, and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; grasps the grasping target corresponding to the control instruction according to the grasping path. A more general and autonomous machine intelligence is achieved, and the robot grasping efficiency is improved.

[0022] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an embodied intelligent mobile operation robot, etc. The following takes the embodied intelligent mobile operation robot (hereinafter referred to as "robot") as an example to illustrate this embodiment and the following embodiments.

[0023] Based on this, the present application embodiment provides a robot control method, referring to Figure 1 , Figure 1 A flowchart diagram of the first embodiment of the robot control method of the present application.

[0024] In this embodiment, the robot control method includes steps S10 to S40: Step S10, when receiving a control instruction, determining the grab target information according to the control instruction; It should be noted that the control instruction may be a control command sent by a user, for example, to let the robot grab an object. Determining the grab target information according to the control instruction may be determining the position of the object to be grabbed, the name of the object or the image of the object, and the target placement point to be placed after the grab according to the control instruction.

[0025] It should be noted that, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of the embodied intelligent mobile operation robot provided in the first embodiment of the robot control method of the present application; the embodied intelligent mobile operation robot includes a mechanical arm and a vehicle chassis, the mechanical arm is used for the grasping of the embodied intelligent mobile operation robot, and the vehicle chassis is used for the movement of the embodied intelligent mobile operation robot. Figure 3 , Figure 3The schematic diagram of the chassis structure of the robot control method embodiment 1 of this application; the chassis includes a Gemini Pro camera, two TOF laser radars, and six evenly distributed ultrasonic sensors. Other cameras, laser radars, and ultrasonic sensors can also be used as substitutes, and this embodiment is not limited here. Figure 4 , Figure 4 A schematic diagram of the details of the vehicle chassis structure provided for Example 1 of the robot control method of the present application; the vehicle chassis includes a power charging port, a hard emergency stop button, a power switch, a router WAN interface, an external power supply interface, a soft emergency stop button, USB, Type C, automatic recharging, and a suspension chassis.

[0026] Further, in order to enable the robot to accurately understand the user's intention and execute the corresponding instruction, the step S10 may include: upon receiving the control instruction, determining the user demand information according to the control instruction; Collecting images of the surrounding environment; The user demand information and the surrounding environment image are input into a preset multimodal large model to obtain the grasping target information output by the preset multimodal large model, wherein the grasping target information includes the relative map position and robot posture coordinates of the grasping target corresponding to the control instruction.

[0027] It should be noted that when a control instruction is received, determining the user demand information according to the control instruction may be to perform semantic analysis on the control instruction to obtain a semantic analysis result, and determining the information of the item that the user wants to grab according to the semantic analysis result, which may include the name of the item and its approximate location, such as, grab the Coke on the refrigerator. The acquisition of the surrounding environment image may be a robot taking pictures of the surrounding environment using a camera on the robot. The preset multimodal large model may be a model such as ChatGPT4, Tongyi Qianwen pro max, etc. The preset multimodal large model predicts the relative map position of the grab target and the robot posture coordinates based on the user demand information, the surrounding environment image, and the pre-collected map information within the robot's activity range.

[0028] Before the step S10, it also includes mapping the range of activity of the robot to obtain the map information. Specifically, gmapping technology is used for mapping. gmapping is a simultaneous localization and mapping (SLAM) technology based on particle filtering. It mainly uses an improved Rao-Blackwellised particle filter to solve the positioning and map construction problems at the same time. The core idea is to use multiple particles, each particle represents a possible robot posture and is associated with a map. Using gmapping technology based on lidar information, the remote-controlled robot moves within the range of activity and simultaneously builds a 2D plane map for autonomous navigation and positioning of the robot.

[0029] Furthermore, in order to improve the service efficiency of the robot, the step of determining the user demand information according to the control instruction when receiving the control instruction includes: When receiving a control instruction, converting the control instruction into text to obtain text information; Determining whether a target wake-up word is detected according to the text information; When the target wake-up word is detected, the user demand information is determined based on the text information.

[0030] It should be noted that the control instruction can be a voice command sent by the user to the robot. When the robot receives the voice command sent by the user, it first converts the control instruction into text to obtain text information. Then, it is determined whether there is a target wake-up word based on the text information. The target wake-up word can be a keyword such as "take, take, put, send, and hand over" that instructs the robot to grab items. When the target wake-up word is detected, the user demand information is determined based on the text information. In order to improve the analysis accuracy of the preset multimodal large model, a prompt word can also be set in this embodiment. After the user's text information is optimized by the prompt word, it is input into the preset multimodal large model, so that the preset multimodal large model outputs more accurate user demand information.

[0031] Step S20, moving to the target position according to the captured target information, and collecting target environment information; It should be noted that the grab target information includes the target position of the grab target to be grabbed, and moving to the target position according to the grab target information can be calling navigation autonomous mobile navigation to move to the target position corresponding to the grab target information, and collecting surrounding environmental information at the target position to obtain target environmental information.

[0032] Step S30, determining a grasping path based on the target environment information and the grasping target information; It should be noted that determining the grasping path based on the target environment information and the grasping target information may be determining the detailed position and posture information of the grasping target in the target environment information according to the target environment information and the grasping target information, and determining the optimal grasping posture of the robot when grasping and the path of the robot from the initial state to the optimal grasping posture according to the detailed position and posture information, that is, the grasping path.

[0033] Step S40: grabbing a grab target corresponding to the control instruction according to the grab path.

[0034] It should be noted that grasping the grasping target corresponding to the control instruction according to the grasping path may be controlling the robot arm to grasp the grasping target corresponding to the control instruction according to the grasping path.

[0035] Furthermore, after the grab target is grabbed, it is also necessary to place the grab target at a specified position according to the control instruction. After the step S40, it also includes: determining the target placement position according to the grab target information; The grab target is placed at the target placement position.

[0036] It should be noted that the grasped target information may also include a target placement position. Placing the grasped target at the target placement position may include determining a movement path of the robot according to the constructed robot activity range map and the target placement position, and moving to the target placement position according to the movement path. Then, the grasped grasped target is placed at the target placement position.

[0037] When receiving a control instruction, this embodiment determines the grasping target information according to the control instruction; moves to the target position according to the grasping target information and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; and grasps the grasping target corresponding to the control instruction according to the grasping path. Since this embodiment moves to the target position according to the grasping target information and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; and grasps the grasping target corresponding to the control instruction according to the grasping path. A more general and autonomous machine intelligence is achieved, and the robot grasping efficiency is improved.

[0038] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 5 , Figure 5The flowchart of the second embodiment of the robot control method of the present application is provided, wherein the step S30 further includes the following steps: Step S301: determining the position information of the grab target to be grabbed in the target environment information according to the target environment information and the grab target information; It should be noted that the determination of the position information of the grasping target in the target environment information according to the target environment information and the grasping target information may be to send the target environment information and the grasping target information to the preset multimodal large model, and the preset multimodal large model determines the position information of the grasping target in the target environment information according to the characteristics of the grasping target in the grasping target information, which may specifically be the coordinate range in the target environment information.

[0039] Step S302: sending the position information and the point cloud map corresponding to the position information to a preset GraspNet model to obtain the grasping posture information output by the preset GraspNet model; It should be noted that GraspNet is a deep learning network for robot grasping, which aims to solve the problem of effective grasping of objects by robots in various environments. The network predicts the best grasping position and posture by analyzing the three-dimensional shape of the object and the surrounding environment. GraspNet not only focuses on the grasping of a single object, but also can handle the grasping task of multiple objects in complex scenes. GraspNet is a small neural network model deployed locally, which cooperates with the above-mentioned preset multimodal large model deployed in the cloud to jointly complete the grasping of objects. Specifically, the depth camera on the robot's mechanical arm can collect data from the target environment, and the target environment can be the environment where the grasping target is located. The data collected by the mechanical arm forms a point cloud map, which provides visual and depth information of the scene and forms the basis for grasping analysis. The point cloud map is a point cloud map formed by the data collected by the depth camera on the mechanical arm from the target environment. The preset GraspNet model can be a deep learning network for robot grasping. The grasping posture information can be the optimal grasping posture output by the preset GraspNet model.

[0040] Step S303: determining a grasping path according to the grasping posture information.

[0041] It should be noted that the grasping path determined according to the grasping posture information may be a path from the initial state of the robot to the optimal grasping posture generated by the robot arm path planning library MoveIt in the robot ROS. The MoveIt is a powerful robot motion planning framework. This embodiment uses the robot arm path planning library MoveIt in ROS to generate a reference trajectory from the initial state to the optimal grasping posture.

[0042] Furthermore, during the robot grasping process, there will be random vibration. In order to improve the grasping efficiency, the step of grasping the grasping target corresponding to the control instruction according to the grasping path includes: Constructing a Lagrangian dynamics model, wherein the Lagrangian dynamics model is used to predict the stability of the robot when grasping through the grasping path; Inputting the grasping path into the Lagrangian dynamics model, the Lagrangian dynamics model simulates the stability of the robot when grasping, and outputs a prediction result; The grab target corresponding to the control instruction is grabbed according to the prediction result.

[0043] It should be noted that this embodiment proposes a tracking control algorithm for a robotic arm system running in a random vibration environment and achieves finite-time stability, and is suitable for situations where the system has unknown dynamics. First, a random Lagrangian dynamic model of the robotic arm in a random vibration environment (i.e., the Lagrangian dynamic model) is constructed. Then, a command filter adaptive backstepping controller is proposed, which not only approximates the unknown dynamics of the robotic arm system, but also avoids the singularity problem of the traditional finite-time backstepping method. In addition, an error compensation mechanism is introduced to compensate for the error of the filter, and an auxiliary system is further introduced to deal with the common input saturation problem in practice. The results prove the actual mean square finite-time stability of the tracking error. Finally, a random robotic arm model is used to verify the effectiveness of the proposed control algorithm. Specifically, the random Lagrangian control system of the robotic arm joint space is considered as follows:

[0044] in, is a state variable, n is used to represent the dimension, yes The first derivative of is the generalized mass (inertia) matrix, is the Coriolis / centrifugal matrix, is the gravity vector, White noise The random excitation force caused by ; is the controlling force acting on the system, and is the input saturation function, representing the controller input saturation; it satisfies:

[0045] in is a known constant, defined as:

[0046] So, there is ,in , And define .

[0047] Assumption 1: Symmetric and positive, and It can be divided into the nominal part and the unknown and uncertain part. satisfy:

[0048] in, , and is a known constant. The unknown part satisfy ,in is an unknown constant.

[0049] According to the assumption, , it can be concluded that ,in Similarly, . Then the robotic arm system can be expressed as:

[0050] definition , we can get the Ito stochastic integral equation of the robot arm as:

[0051] in, , . It is white noise The power spectral density of is a positive definite matrix, is a The standard Wiener process of dimensionality.

[0052] Control algorithm: Defining the Tracking Error Signal Represents the difference between the virtual control signal and the robot system state,

[0053] in, is the reference signal, and its first-order derivative is assumed to exist. is the output of the finite-time command filter, designed as:

[0054] It is the input of virtual control signal.

[0055] Virtual Signal and controller The design is as follows:

[0056] in, and is the gain parameter of the design. yes of estimates, and is defined as A constant, where Represents the bi-norm of the vector The update process is designed as follows:

[0057] is the compensated error signal, defined as:

[0058] is the error compensation mechanism, defined as:

[0059] in, .

[0060] is an auxiliary system, defined as:

[0061] Description: Finite time command filter, with virtual controller As input, we get and its first-order derivative . In fact, the first-order Levant differentiator proposed in the finite-time command filter can not only achieve fast filtering of the virtual control signal, but also ensure stability in finite time. The finite-time command filter is first applied to the finite-time control of a random manipulator system, solving the singularity problem encountered by the traditional finite-time backstepping method. In addition, an auxiliary system is added to the controller design to ensure that the real control input can be designed to counter the influence of input saturation on the control performance. The finite-time stability of this control system has been proven.

[0062] In the specific implementation, please refer to Figure 6 , Figure 6The overall flow diagram provided for the second embodiment of the robot control method of this application, LVM is the preset multimodal large model, first obtain the access rights of the preset multimodal large model and the preset prompt, the prompt is used to optimize the control instructions input to the preset multimodal large model. The audio information can be the control voice sent by the user. After the audio information is converted into text language information, it is determined whether it contains the wake-up word. If it does, the picture in the room is taken, and the pre-established 2D map and the taken picture are sent to LVM. After LVM predicts the path, the robot moves to the target point corresponding to the grasping target according to the path, and then the robot arm takes the environmental picture around the grasping target, and inputs the environmental picture into LVM. The coordinates of the grasping target are predicted by LVM, and the robot arm takes the depth map corresponding to the coordinates, that is, the point cloud map, and then predicts the optimal grasping posture of the grasping through GraspNet, and generates the grasping path from the initial state to the optimal grasping posture through MoveIt. Finally, the robot arm performs the grasping action according to the grasping path and places the grasping target at the specified position.

[0063] This embodiment determines the position information of the grasping target to be grasped in the target environment information according to the target environment information and the grasping target information; sends the position information and the point cloud map corresponding to the position information to the preset GraspNet model to obtain the grasping posture information output by the preset GraspNet model; and determines the grasping path according to the grasping posture information. This embodiment determines the grasping posture information by the preset GraspNet model, and then determines the grasping path according to the grasping posture information. It can improve the grasping success rate.

[0064] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the robot control method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0065] This application also provides a robot control device, please refer to Figure 7 , the robot control device comprises: The receiving module 10 is used to determine the grab target information according to the control instruction when receiving the control instruction; A moving module 20 is used to move to a target location according to the captured target information and collect target environment information; A grasping path determining module 30, configured to determine a grasping path based on the target environment information and the grasping target information; The grabbing module 40 is used to grab the grabbing target corresponding to the control instruction according to the grabbing path.

[0066] When receiving a control instruction, this embodiment determines the grasping target information according to the control instruction; moves to the target position according to the grasping target information and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; and grasps the grasping target corresponding to the control instruction according to the grasping path. Since this embodiment moves to the target position according to the grasping target information and collects the target environment information; determines the grasping path based on the target environment information and the grasping target information; and grasps the grasping target corresponding to the control instruction according to the grasping path. A more general and autonomous machine intelligence is achieved, and the robot grasping efficiency is improved.

[0067] The robot control device provided by the present application adopts the robot control method in the above embodiment, which can solve the technical problem that the existing embodied intelligent mobile operation robot has low execution efficiency when performing grasping tasks. Compared with the prior art, the beneficial effects of the robot control device provided by the present application are the same as the beneficial effects of the robot control method provided by the above embodiment, and the other technical features in the robot control device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0068] The present application provides a robot control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the robot control method in the above-mentioned embodiment one.

[0069] Reference below Figure 8 , which shows a schematic diagram of the structure of a robot control device suitable for implementing the embodiment of the present application. The robot control device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The robot control device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0070] like Figure 8As shown, the robot control device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the robot control device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the robot control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a robot control device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0071] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0072] The robot control device provided by the present application adopts the robot control method in the above embodiment, which can solve the technical problem that the existing embodied intelligent mobile operation robot has low execution efficiency when performing grasping tasks. Compared with the prior art, the beneficial effects of the robot control device provided by the present application are the same as the beneficial effects of the robot control method provided by the above embodiment, and the other technical features in the robot control device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0073] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0074] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0075] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the robot control method in the above-mentioned embodiment.

[0076] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0077] The computer-readable storage medium may be included in the robot control device; or may exist independently without being assembled into the robot control device.

[0078] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the robot control device, the robot control device executes the robot control method.

[0079] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0080] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0081] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0082] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned robot control method, and can solve the technical problem that the existing embodied intelligent mobile operation robot has low execution efficiency when performing grasping tasks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the robot control method provided by the above-mentioned embodiment, and will not be repeated here.

[0083] The present application also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the robot control method as described above are implemented.

[0084] The computer program product provided by the present application can solve the technical problem that the existing embodied intelligent mobile operation robot has low execution efficiency when performing grasping tasks. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the robot control method provided by the above embodiment, which will not be repeated here.

[0085] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A robot control method, characterized in that: The robot control method comprises the following steps: When receiving a control instruction, determining the grab target information according to the control instruction; Move to the target location according to the captured target information and collect target environment information; Determine a grasping path based on the target environment information and the grasping target information; The grabbing target corresponding to the control instruction is grabbed according to the grabbing path.

2. The robot control method according to claim 1, characterized in that: The step of determining the target information to be captured according to the control instruction when the control instruction is received includes: When receiving a control instruction, determining user demand information according to the control instruction; Collecting images of the surrounding environment; The user demand information and the surrounding environment image are input into a preset multimodal large model to obtain the grasping target information output by the preset multimodal large model, wherein the grasping target information includes the relative map position and robot posture coordinates of the grasping target corresponding to the control instruction.

3. The robot control method according to claim 2, characterized in that: The step of determining the user demand information according to the control instruction when the control instruction is received includes: When receiving a control instruction, converting the control instruction into text to obtain text information; Determining whether a target wake-up word is detected according to the text information; When the target wake-up word is detected, the user demand information is determined based on the text information.

4. The robot control method according to claim 1, characterized in that: The step of determining a grasping path based on the target environment information and the grasping target information includes: Determine the position information of the grab target to be grabbed in the target environment information according to the target environment information and the grab target information; Sending the position information and the point cloud map corresponding to the position information to a preset GraspNet model to obtain the grasping posture information output by the preset GraspNet model; A grasping path is determined according to the grasping posture information.

5. The robot control method according to claim 4, characterized in that: The step of grabbing the grab target corresponding to the control instruction according to the grab path includes: Constructing a Lagrangian dynamics model, wherein the Lagrangian dynamics model is used to predict the stability of the robot when grasping through the grasping path; Inputting the grasping path into the Lagrangian dynamics model, the Lagrangian dynamics model simulates the stability of the robot when grasping, and outputs a prediction result; The grab target corresponding to the control instruction is grabbed according to the prediction result.

6. The robot control method according to any one of claims 1 to 5, characterized in that: After the step of grabbing the grab target corresponding to the control instruction according to the grab path, the method further includes: Determine the target placement position according to the captured target information; The grab target is placed at the target placement position.

7. A robot control device, characterized in that: The robot control device comprises: A receiving module, configured to determine the grab target information according to the control instruction when receiving the control instruction; A moving module, used to move to a target location according to the captured target information and collect target environment information; A grasping path determination module, used to determine a grasping path based on the target environment information and the grasping target information; The grabbing module is used to grab the grabbing target corresponding to the control instruction according to the grabbing path.

8. A robot control device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the robot control method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 6 are implemented.

Images