Robot control method, robot control device and robot
Through the action translation model, the control instructions at the operation end are mapped into instructions that can be recognized by the target robot, which solves the problem of low flexibility and functionality of robot execution actions in the prior art, and improves the versatility and user experience of robot control.
Patent Information
- Application Number
- CN202410286911.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2024-11-12
AI Technical Summary
Existing robot control methods rely on strictly predefined programs and instructions, resulting in less flexibility and functionality in robots performing actions, affecting user experience.
Using the action translation model, the control instructions sent by the operation end are mapped into instructions that the target robot can recognize. Through the training action translation model, the control of multiple categories of robots is realized, and the universality and generalization ability of the robot control process is improved, and through Feature extraction and mapping improve translation efficiency.
It realizes the robot to execute complex and flexible operator instructions, enhances the robot's generalization ability and flexibility and functionality in actual application scenarios, and significantly improves the user experience and translation speed.
Smart Images

Figure CN118927232A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of robot control technology, and in particular, relates to a robot control method, a robot control device and a robot. Background Art
[0002] In the development of modern robotics technology, controlling robots and automated operations of robots are key research directions, and efficient and flexible control of robots has become an urgent problem to be solved. In related technologies, robots are mainly controlled through remote control technology, visual recognition systems, and artificial intelligence training methods. However, most of the above-mentioned methods of controlling robots rely on strictly predefined programs and instruction sets to control the robots to perform corresponding actions, so that robots can often only execute relatively simple operating instructions, and the flexibility and functionality of executing actions are low, which affects the user experience. Summary of the invention
[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a robot control method, a robot control device and a robot, which can convert complex instructions into instructions that can be recognized by the robot, so as to remotely control the robot to execute complex and flexible operator instructions, thereby improving the generalization, flexibility and functionality of the robot.
[0004] In a first aspect, the present application provides a robot control method, which is applied to a server side, and the method comprises:
[0005] Receiving a first control instruction sent by an operation terminal;
[0006] Inputting the first control instruction into an action translation model, and obtaining a second control instruction corresponding to the target robot output by the action translation model;
[0007] Sending the second control instruction to the target robot; the second control instruction is used to control the target robot; wherein,
[0008] The action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
[0009] According to the robot control method of the present application, a motion translation model is trained by using sample robot instructions and sample user instructions recognizable by multiple categories of robots. The trained motion translation model maps the first control instruction sent by the receiving operating terminal into a second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction recognizable by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions. In addition, the translation of control instructions through the motion translation model can also improve the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
[0010] According to the robot control method of the present application, when the first control instruction is a video frame, inputting the first control instruction into the action translation model to obtain the second control instruction corresponding to the target robot output by the action translation model includes:
[0011] Extracting features from the video frame to obtain position and posture information corresponding to the first object;
[0012] The posture information is mapped to obtain the second control instruction.
[0013] According to the robot control method of the present application, the step of extracting features from the video frame to obtain posture information corresponding to the first object includes:
[0014] Extracting a foreground image in the video frame;
[0015] Feature extraction is performed on joint points corresponding to the first object in the foreground image to obtain the posture information.
[0016] According to the robot control method of the present application, after sending the second control instruction to the target robot, the method further includes:
[0017] receiving first object feedback information, where the first object feedback information is used to characterize the situation in which the target robot executes the second control instruction; the first object feedback information is sent by the operating end;
[0018] The action translation model is optimized based on the first object feedback information.
[0019] According to the robot control method of the present application, the sending the second control instruction to the target robot includes:
[0020] Pairing and encoding the first control instruction with a second control instruction corresponding to the first control instruction to obtain coding information;
[0021] Encrypting the coded information based on a target encryption algorithm to obtain encrypted information;
[0022] The encrypted information is sent to the target robot.
[0023] In a second aspect, the present application provides a robot control device, which is applied to a server side, and the device includes:
[0024] A first processing module, used for receiving a first control instruction sent by an operation terminal;
[0025] A second processing module, configured to input the first control instruction into an action translation model, and obtain a second control instruction corresponding to the target robot output by the action translation model;
[0026] The third processing module is used to send the second control instruction to the target robot; the second control instruction is used to control the target robot; wherein,
[0027] The action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
[0028] According to the robot control device of the present application, a motion translation model is trained by using sample robot instructions and sample user instructions recognizable by multiple categories of robots. The trained motion translation model maps the first control instruction sent by the receiving operating terminal into the second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction recognizable by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions. In addition, the translation of control instructions through the motion translation model can also improve the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
[0029] In a third aspect, the present application provides a robot control method, applied to an operating end, the method comprising:
[0030] receiving a first input from a first object;
[0031] In response to the first input, a first control instruction is acquired; the first control instruction is used for mapping by the server to be converted into a second control instruction recognizable by the target robot; the second control instruction is used to control the target robot;
[0032] and / or,
[0033] Acquire a video frame corresponding to the first object, and determine the video frame as the first control instruction;
[0034] Send the first control instruction to the server.
[0035] According to the robot control method of the present application, by receiving the first input of the first object of the operating end, in response to the first input, the first input or the collected video frame corresponding to the first object of the operating end is determined as the first control instruction, and the control instruction of the first object is effectively obtained, thereby controlling the robot through the first control instruction.
[0036] According to the robot control method of the present application, after sending the first control instruction to the server, the method further includes:
[0037] First object feedback information is obtained according to the action execution status of the target robot based on the second control instruction; the first object feedback information is used to optimize the action translation model, and the action translation model is used to map the first control instruction to the second control instruction.
[0038] In a fourth aspect, the present application provides a robot control device, applied to an operating end, the device comprising:
[0039] A fourth processing module, configured to receive a first input from a first object;
[0040] A fifth processing module, configured to obtain a first control instruction in response to the first input; the first control instruction is used for mapping by the server to be converted into a second control instruction recognizable by the target robot; the second control instruction is used to control the target robot;
[0041] and / or,
[0042] a sixth processing module, configured to collect a video frame corresponding to the first object, and determine the video frame as the first control instruction;
[0043] A seventh processing module is used to send the first control instruction to the server.
[0044] According to the robot control device of the present application, by receiving the first input of the first object of the operating end, in response to the first input, the first input or the collected video frame corresponding to the first object of the operating end is determined as the first control instruction, thereby effectively acquiring the control instruction of the first object, thereby controlling the robot through the first control instruction.
[0045] In a fifth aspect, the present application provides a robot control method, applied to a robot side, the method comprising:
[0046] Receiving a second control instruction sent by the server, where the second control instruction is generated by the server by mapping the received first control instruction;
[0047] Based on the second control instruction, a target action is executed.
[0048] According to the robot control method of the present application, by receiving the second control instruction sent by the server, the robot executes the first control instruction sent by the operator based on the recognizable second control instruction, effectively executes the target action, thereby realizing the operator controlling the robot, which is suitable for different application scenarios and improves the user experience.
[0049] In a sixth aspect, the present application provides a robot control device, applied to a robot end, the device comprising:
[0050] An eighth processing module, configured to receive a second control instruction sent by the server, where the second control instruction is generated by the server mapping the received first control instruction;
[0051] A ninth processing module is used to execute a target action based on the second control instruction.
[0052] According to the robot control device of the present application, by receiving the second control instruction sent by the server side, the robot side executes the first control instruction sent by the operator side based on the recognizable second control instruction, effectively executes the target action, thereby realizing the operator side controlling the robot side, which is suitable for different application scenarios and improves the user experience.
[0053] In a seventh aspect, the present application provides a robot, comprising:
[0054] at least one robotic arm;
[0055] A mobile device, wherein the at least one mechanical arm is disposed on the mobile device;
[0056] Multiple sensors;
[0057] a processor, the processor being electrically connected to the at least one mechanical arm, the mobile device, and the plurality of sensors, respectively;
[0058] The robot performs actions based on a robot control method.
[0059] In an eighth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot control method as described in the first aspect above is implemented.
[0060] In a ninth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the robot control method as described in the first aspect above.
[0061] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0062] By using sample robot instructions and sample user instructions that are recognizable by robots of various categories to train the motion translation model, the trained motion translation model maps the first control instruction sent by the receiving operating terminal into the second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the robot's generalization ability in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions. In addition, the translation of control instructions through the motion translation model can also increase the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
[0063] Furthermore, by optimizing the AI model in turn through the first object feedback data obtained after controlling the robot to perform related tasks, the motion translation model can be continuously optimized through large-scale first object feedback data, thereby improving the generalization ability of the motion translation model as well as the precision and accuracy of the motion translation model, so that the second control instruction output by the motion translation model is closer to the true intention of the first object, thereby improving the user experience.
[0064] Furthermore, by pairing and encoding the first control instruction with the second control instruction corresponding to the first control instruction, the coded information is obtained, and the coded information is encrypted based on the target encryption algorithm to obtain encrypted information, which is then sent to the target robot to ensure data security and thus protect user privacy.
[0065] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0067] Figure 1 This is one of the flowcharts of the robot control method provided in the embodiment of the present application;
[0068] Figure 2 This is the second flow chart of the robot control method provided in the embodiment of the present application;
[0069] Figure 3 This is the third flow chart of the robot control method provided in the embodiment of the present application;
[0070] Figure 4 is a schematic diagram of a robot control method provided in an embodiment of the present application;
[0071] Figure 5 It is one of the structural schematic diagrams of the robot control device provided in the embodiment of the present application;
[0072] Figure 6 This is the second structural schematic diagram of the robot control device provided in the embodiment of the present application;
[0073] Figure 7 This is the third structural schematic diagram of the robot control device provided in the embodiment of the present application;
[0074] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0075] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0076] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0077] The robot control method, robot control device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0078] Among them, the robot control method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.
[0079] The terminal includes but is not limited to portable communication devices such as mobile phones or tablet computers. It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer.
[0080] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.
[0081] The robot control method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the robot control method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The robot control method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.
[0082] It should be noted that the robot control method of the present application can be applied to the field of robot remote control. During the remote control process, for example, a remote operator can be equipped for each robot through a dedicated recruitment platform, so that the operator can remotely control the robot to perform tasks such as navigation, robotic arm operation and walking.
[0083] When controlling the robot to perform actions, the server side, the operation side and the robot side are involved, among which the server side is the terminal for processing data; the operation side is the terminal used by the operator, and the robot side is the terminal for performing tasks; the server side is connected between the operation side and the robot side, and is used to receive, send and process data.
[0084] It is understandable that the operating end and the robot end may not be in the same area, such as the operating end may be in position A and the robot end may be in position B, etc.
[0085] like Figure 1 As shown, the robot control method is applied to the server side, including: step 110, step 120 and step 130.
[0086] Step 110: The server receives a first control instruction sent by the operator;
[0087] In this step, the operating end is used to obtain a first control instruction input by a first object.
[0088] The first control instruction is an instruction proposed by the first object for controlling the robot to perform an action; the first control instruction may include: a remote control instruction and an on-site control instruction.
[0089] For example, taking remote control instructions as an example, the first object of the operating end is at position A, and the robot is at the home of a second object at position B. After the first object of the operating end at position A inputs the first control instruction, the operating end sends the first control instruction to the server, and the server receives the first control instruction.
[0090] The first object is an operator or a specific user who controls the robot to perform actions.
[0091] The first object can send a first control instruction through the operation terminal.
[0092] The second object may be a general user such as the owner or user of the target robot.
[0093] The target robot can be placed in the scene where the second subject needs to use the robot.
[0094] The first control instruction may be expressed as an action instruction, a voice instruction, a character instruction, or any other instruction.
[0095] The first control instruction can be input through a camera, a microphone, a screen button, etc.
[0096] In the actual execution process, when the operation end sends the first control instruction to the server end, the server end receives the first control instruction.
[0097] In some embodiments, the first control instruction may be obtained based on the following steps:
[0098] The operating end receives a first input of a first object;
[0099] In response to the first input, a first control instruction is obtained; the first control instruction is used for mapping by the server to be converted into a second control instruction that can be recognized by the target robot; the second control instruction is used to control the target robot;
[0100] In this embodiment, the first input is used to receive a control instruction input by a first object.
[0101] The first input may be in at least one of the following ways:
[0102] First, the first input may be a touch operation, including but not limited to a click operation, a slide operation, and a press operation.
[0103] In this implementation, receiving the first input of the first object may be receiving a touch operation of the first object in a display area of the terminal display screen.
[0104] In order to reduce the misoperation rate of the first object, the effective area of the first input can be limited to a specific area, such as the upper middle area of the control command interface; or when the control command interface is displayed, the target control is displayed on the current interface, and the first input can be achieved by touching the target control; or the first input can be set to a continuous multiple tapping operation on the display area within a target time interval.
[0105] Secondly, the first input may be a physical key input.
[0106] In this embodiment, a physical button corresponding to the control instruction is provided on the terminal body, and receiving the first input of the first object can be an operation of receiving the first object pressing the corresponding physical button; the first input can also be a combined operation of pressing multiple physical buttons at the same time.
[0107] Third, the first input may be voice input.
[0108] In this implementation, when receiving a voice such as "grab", the terminal may use "grab" as the first control instruction and send the first control instruction "grab" to the server.
[0109] Of course, in other embodiments, the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present application.
[0110] The first control instruction is used for mapping by the server to be converted into a second control instruction that can be recognized by the target robot.
[0111] The second control instruction is used to control the target robot.
[0112] In the actual execution process, when the operating end receives the first input of the first object at position A, in response to the first input, the first input is determined as the first control instruction.
[0113] According to the robot control method provided in the embodiment of the present application, by receiving the first input from the operating end and determining the first input as the first control instruction, the control instruction issued by the operating end is effectively obtained.
[0114] In some embodiments, the first control instruction may also be obtained based on the following steps:
[0115] A video frame corresponding to the first object is acquired, and the video frame is determined as a first control instruction.
[0116] In this embodiment, the video frame corresponding to the first object is an action video frame of the first object captured by a camera or a depth sensor.
[0117] For example, when the first object at the operating end at position A sends a video frame of a grabbing action, the server receives the video frame and determines the video frame as a first control instruction.
[0118] According to the robot control method provided in the embodiment of the present application, by collecting the video frame corresponding to the first object and determining the video frame as the first control instruction, the first control instruction corresponding to the video frame input by the operating end is effectively obtained.
[0119] Step 120: The server inputs the first control instruction into the action translation model, and obtains the second control instruction corresponding to the target robot output by the action translation model. In this step, the target robot is a robot that executes the action corresponding to the first control instruction.
[0120] The second control instruction is an instruction that can be recognized by the target robot corresponding to the first control instruction.
[0121] The action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
[0122] The action translation model may be an artificial intelligence model.
[0123] The action translation model can map different types of first control instructions into corresponding second control instructions.
[0124] The first control instruction type includes: video frames, voice instructions, and screen input instructions.
[0125] The action translation model can be a neural network model or a machine learning model.
[0126] Of course, the action translation model may also be any other model that can map the first control instruction to the second control instruction.
[0127] During the actual execution process, a dedicated motion translator can be deployed on the server. The motion translation model in the motion translator can map the operator's motion data into the robot's control instructions.
[0128] A dedicated motion translator can map the operator's body movements into the robot's motion instructions.
[0129] The action translator can also automatically adapt to different robot models and forms to ensure the universality of instruction mapping. The sample user instructions are multiple first control instructions sent by the operator end and stored in the server end.
[0130] The sample robot instruction is a second control instruction stored on the server side corresponding to the sample user instruction.
[0131] It is understandable that for the same sample user instruction, the sample robot instructions that can be recognized by robots of different categories may be different.
[0132] For different sample user instructions, the sample robot instructions that can be recognized by different categories of robots may also be different.
[0133] During the actual execution process, the action translation model can be continuously trained and optimized through a large number of sample user instructions and sample robot instructions.
[0134] In the actual execution process, after receiving the first control instruction at the server end, the first control instruction is input into the motion translation model, and the motion translation model outputs a second control instruction that can be recognized by the target robot.
[0135] For example, when the first control instruction is a video frame for grabbing an object, the video frame is mapped to a second control instruction that can be recognized by the target robot.
[0136] When the first control instruction is a voice instruction of "take out article A", the voice instruction is mapped to a second control instruction that can be recognized by the target robot.
[0137] In the case where the first control instruction is a screen input instruction obtained by clicking on the screen or in other ways, the screen input instruction is mapped to a second control instruction that can be recognized by the target robot.
[0138] In the actual execution process, any other feasible method can also be used, for example, by pre-building a mapping relationship table between the first control instruction and the second control instruction, and translating based on the mapping relationship table to translate the first control instruction into a second control instruction that can be recognized by the target robot.
[0139] In other embodiments, the first control instruction may be mapped to a second control instruction that can be recognized by the target robot through an artificial intelligence model that is different from the motion translation model.
[0140] Step 130: The server sends a second control instruction to the target robot.
[0141] In this step, the second control instruction is used to control the target robot.
[0142] During the actual execution process, the second control instruction may be sent to the target robot through a wireless communication module provided in the target robot.
[0143] In some embodiments, the robot performs actions based on the following:
[0144] The robot receives the second control instruction sent by the server;
[0145] Based on the second control instruction, the target action is executed.
[0146] In this embodiment, the robot side can receive the second control instruction sent by the server side through the wireless communication module.
[0147] The target action is the action that the second control instruction requires the robot to perform.
[0148] During the actual execution process, after the first object at the operating end at position A inputs the first control instruction through the operating end, the operating end sends the first control instruction to the server end, and the server end translates the first control instruction into a second control instruction that can be recognized by the target robot through the action translation model, and then sends the second control instruction to the target robot in the home of a second object at position B. After the target robot receives the second control instruction through the wireless communication module, it can collect image data of the surrounding environment through the image sensor, and process it through the image processor to obtain information about the current surrounding environment, so as to execute the target action based on the second control instruction and combined with the information about the current surrounding environment.
[0149] Of course, in some embodiments, the robot can also be navigated and controlled by a navigation sensor, such as enabling the robot to travel along an optimal path or along an edge while executing a target action. According to the robot control method provided in the embodiment of the present application, by receiving the second control instruction sent by the server, the robot executes the first control instruction sent by the operator based on the recognizable second control instruction, effectively executing the target action, thereby enabling the operator to control the robot, which is suitable for different application scenarios and improves the user experience.
[0150] In some embodiments, step 130 may further include:
[0151] Pairing and encoding the first control instruction with the second control instruction corresponding to the first control instruction to obtain coding information;
[0152] Encrypting the coded information based on the target encryption algorithm to obtain encrypted information;
[0153] Send an encrypted message to the target robot.
[0154] In this embodiment, the coded information is information obtained by pairing and coding the first control instruction with the second control instruction corresponding to the first control instruction.
[0155] The specific implementation method of pairing coding can be determined based on actual conditions and is not limited in this application.
[0156] The target encryption algorithm is the algorithm used to encrypt the encoded information.
[0157] The target encryption algorithm can be determined based on actual conditions, and this application does not limit it; for example, the target encryption algorithm can be the Advanced Encryption Standard (AES) 256 encryption algorithm or the RSA encryption algorithm.
[0158] The encrypted information is the information corresponding to the coded information after being encrypted by the target encryption algorithm.
[0159] During the actual execution process, the first control instruction is paired and encoded with the second control instruction corresponding to the first control instruction to obtain the encoded information, and the encoded information is encrypted based on the target encryption algorithm to obtain the encrypted information, and the encrypted information is sent to the target robot through the wireless transmission module.
[0160] It is understandable that by adopting high-performance wireless transmission devices and encryption technology, operators can achieve remote control in various network environments, and data transmission always remains high-speed and secure; this stable connection ensures that remote operators can continuously and seamlessly control the robot without being affected by network fluctuations.
[0161] In the actual execution process, an encryptor may be provided in the wireless transmission module, and the first control instruction and the second control instruction corresponding to the first control instruction are encrypted based on the encryptor.
[0162] In the actual implementation process, in order to ensure the security of data, all transmitted data is encrypted through a built-in encryptor to ensure that it cannot be accessed by unauthorized third parties.
[0163] Additionally, users can choose whether to share their data to further protect privacy.
[0164] For example, when the server receives a first control instruction sent by the operating terminal at location A, it translates the first control instruction into a second control instruction, then encrypts the second control instruction through the encryptor in the wireless transmission module, and sends the encrypted second control instruction to a robot in the home of a second object at location B to ensure data security.
[0165] According to the robot control method provided in the embodiment of the present application, by pairing and encoding a first control instruction with a second control instruction corresponding to the first control instruction, obtaining coded information, encrypting the coded information based on a target encryption algorithm, and obtaining encrypted information, the encrypted information is sent to the target robot, which can improve data security and protect user privacy, thereby ensuring that it is not accessed by unauthorized third parties.
[0166] In the actual execution process, after the first control instruction is mapped to the second control instruction, the first control instruction and the second control instruction corresponding to the first control instruction may be stored in a memory on the server side, and the storage method may be encrypted storage.
[0167] During the research and development process, the inventors found that in the related technologies, robots are mainly controlled through remote control technology, visual recognition systems, and artificial intelligence training methods. However, most of the above robot control methods rely on strictly predefined programs and instruction sets to control the robots to perform corresponding actions, so that the robots can often only execute relatively simple operation instructions, and the flexibility and functionality of the execution actions are low, which affects the user experience.
[0168] The present application sets up an action translation model to map different types of first control instructions received from the operating end into second control instructions that can be recognized by the target robot, effectively converting complex instructions into instructions that can be recognized by the robot, and sending the second control instruction to the target robot, so that the target robot executes the action corresponding to the second control instruction, thereby realizing the needs of remote control of the robot and improving the flexibility of the robot in executing actions.
[0169] According to the robot control method provided in the embodiment of the present application, a motion translation model is trained by using sample robot instructions and sample user instructions recognizable by multiple categories of robots. The trained motion translation model maps the first control instruction sent by the receiving operating terminal into a second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction recognizable by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions. In addition, the translation of control instructions through the motion translation model can also improve the translation speed and significantly improve the translation efficiency, thereby enhancing the user experience.
[0170] In some embodiments, when the first control instruction is a video frame, step 120 may further include:
[0171] Extracting features from the video frame to obtain position and posture information corresponding to the first object;
[0172] The posture information is mapped to obtain a second control instruction.
[0173] In this embodiment, feature extraction is to extract action features of the first object in the video frame.
[0174] The position and posture information is the position and posture information of the first object.
[0175] During the actual execution process, features of the video frames can be extracted through deep learning models and machine learning models, the posture information corresponding to the first object in the video frame can be obtained, and the posture information can be mapped into a second control instruction that can be recognized by the target robot.
[0176] Of course, features of video frames may also be extracted in any other feasible manner.
[0177] During the actual execution process, advanced computer vision algorithms are used to capture and analyze the operator's body movements in real time, and deep learning and other AI technologies are used to improve the accuracy and speed of motion recognition.
[0178] like Figure 4 As shown, for example, when the posture information of the first object is a raised palm with five fingers stretched out, mapping is performed based on the actual situation of the target robot. When the target robot has only four fingers, the posture information of the first object is mapped, and the obtained second control instruction may be an instruction that the target robot can recognize, which is that the target robot raises its robotic arm and stretches out all four fingers.
[0179] According to the robot control method provided in the embodiment of the present application, by performing feature extraction on the video frame corresponding to the first control instruction, the posture information of the operator in the video frame is obtained, and the posture information is mapped to the second control instruction, so that the robot can capture the operator's body movements in real time, reduce delays, improve the real-time and accuracy of the operation, enable the robot to perform the same actions as the operator, improve its functionality, and achieve the effect of real-time translation.
[0180] In some embodiments, extracting features from the video frame to obtain posture information corresponding to the first object includes:
[0181] Extracting foreground images from video frames;
[0182] Feature extraction is performed on the joint points corresponding to the first object in the foreground image to obtain posture information.
[0183] In this embodiment, the foreground image is a body movement image of the first object.
[0184] The joint points corresponding to the first object are positions of the joint points corresponding to the arm, palm and fingers of the first object.
[0185] During the actual execution process, deep learning models and machine learning models can be used to identify joints, obstacles and other objects corresponding to the first object in the video frame, obtain a foreground image in the video frame, segment the foreground image from the background image in the video frame, obtain the joints corresponding to the first object in the foreground image, extract features of the joints corresponding to the first object, determine the action of the first object based on information corresponding to multiple joints, and obtain the posture information corresponding to each joint.
[0186] For example, the operator at position A (ie, the first object) raises his hand, and the camera at the operating end captures the operator's hand-raising action, and the operating end obtains a video frame corresponding to the hand-raising action; the video frame includes: the operator and the environmental background in which the operator is located.
[0187] In this video frame, the foreground image is the operator; the background image is the environment in which the operator is located, which may be a classroom or a living room. The specific joint points of the operator in the foreground image can be identified and extracted, and the corresponding posture information of the operator can be determined based on the specific joint points.
[0188] In the actual implementation process, this method can be used to identify other important elements in the environment, such as obstacles and target objects, to enhance the robot's environmental perception ability.
[0189] According to the robot control method provided in the embodiment of the present application, by extracting the foreground image in the video frame, the foreground image is segmented from the background image in the video frame to reduce the interference of the background image, thereby accurately obtaining the joint points corresponding to the first object in the foreground image, performing feature extraction on the joint points corresponding to the first object, thereby obtaining precise posture information, and performing instruction mapping based on the posture information to improve the accuracy of the second control instruction finally mapped, thereby further improving the precision and accuracy of the robot control; in addition, it can also reduce the amount of data involved in the calculation, increase the speed of action translation, and improve the efficiency of controlling the robot.
[0190] In some embodiments, after sending the second control instruction to the target robot, the method may further include:
[0191] receiving feedback information from the first object;
[0192] The action translation model is optimized based on the feedback information of the first object.
[0193] In this embodiment, the first object feedback information is an evaluation of the first object on the execution result of the second control instruction executed by the target robot.
[0194] The first object feedback information is used to characterize a situation in which the target robot executes the second control instruction.
[0195] The first object feedback information is sent by the operating terminal.
[0196] In some embodiments, the first object feedback information may be obtained in the following manner:
[0197] receiving a second input from the first object;
[0198] In response to the second input, first object feedback information is obtained.
[0199] In this embodiment, the second input is used to receive feedback information of the first object.
[0200] The second input may be touch input, physical key input, voice input, character input, or any other feasible input that is the same as the first input, which will not be elaborated in this application.
[0201] The first object feedback information includes: incentive information and loss information.
[0202] In the actual execution process, the action translation model may be optimized based on the specific content of the feedback information of the first object received by the server.
[0203] For example, when the first object feedback information is motivational information, the action translation model maps the new first control instruction to a second control instruction corresponding to the original first control instruction when receiving a new first control instruction that is identical to the original first control instruction.
[0204] For another example, when the first object feedback information is loss information, when the action translation model receives a new first control instruction that is the same as the original first control instruction, the second control instruction corresponding to the new first control instruction will be different from the second control instruction corresponding to the original first control instruction.
[0205] In the actual execution process, the action translation model can be optimized by setting the activation function and loss function in the large language model.
[0206] Of course, the motion translation model may also be optimized in any other feasible manner, such as optimizing motion control and the motion translation model for remote control mapping translation through proximal policy optimization (PPO).
[0207] In this embodiment, a large number of robots are used to operate in the home of the second object, while collecting control data, original perception data, and feedback information of the first object, so as to optimize the AI model in turn based on the feedback information, and train a more generalized AI model so that it can perform well in a variety of environments and tasks, thereby improving the generalization ability of the motion translation model as well as the precision and accuracy of the motion translation model.
[0208] In the actual execution process, through machine learning technology, the action translator can be continuously optimized to improve the accuracy and efficiency of its mapping.
[0209] According to the robot control method provided in the embodiment of the present application, the AI model is optimized in turn by obtaining the feedback data of the first object after the robot is controlled to perform the relevant task. The motion translation model can be continuously optimized through large-scale first object feedback data, thereby improving the generalization ability of the motion translation model as well as the precision and accuracy of the motion translation model, so that the second control instruction output by the motion translation model is closer to the true intention of the first object, thereby improving the accuracy and efficiency of mapping, achieving precise control of the robot, and thus improving the user experience.
[0210] In some embodiments, the server includes an image processor and a memory, and the step of optimizing the action translation model based on the feedback information of the first object includes:
[0211] Based on the data stored in the memory, the image processor is used to train the reinforcement learning model to optimize the motion translation model.
[0212] In this embodiment, the image processor is a processor used to train a reinforcement learning model.
[0213] The data stored in the memory includes: a pre-trained model, first object feedback information, a first control instruction, and a second control instruction.
[0214] In the actual implementation process, a data storage unit can be set up, and a large-capacity storage device can be equipped on the server to save the robot's operation history and original perception data.
[0215] At the same time, a processing unit can also be set up, and an efficient data processing unit can be set up inside the server to analyze and process the data in real time.
[0216] Among them, the pre-trained models may include: neural network models, machine learning models and other models.
[0217] Image processors can be used to train reinforcement learning models.
[0218] Reinforcement learning models can be constructed through horizontal federated reinforcement learning algorithms in large language models and hierarchical reinforcement learning strategies.
[0219] It is understandable that using reinforcement learning, the system can continuously learn and improve in actual operation.
[0220] In the actual implementation process, a high-performance reinforcement learning (HFRL) solution such as ChatGPT and the PPO optimization strategy can be combined to build an efficient and robust reinforcement learning process.
[0221] In the actual execution process, based on the data stored in the memory, the reinforcement learning model is trained through a large amount of data to optimize the action translation model. Through cloud computing and distributed processing technology, the rapid processing of data and the real-time updating of the model are ensured.
[0222] In the actual execution process, a training device can be configured on the server. For example, a set of dedicated graphics processing units (GPUs) can be provided in the server for reinforcement learning model training.
[0223] Before training, multiple pre-trained neural network models can be stored on the server to form a model library, which can be continuously optimized based on the collected data.
[0224] Understandably, all data and learning models are stored on the server, which makes system upgrades, maintenance, and data analysis simpler and more centralized.
[0225] In addition, since the robot's instructions and data are synchronized to the server in real time, fault diagnosis and remote maintenance can be achieved.
[0226] According to the robot control method provided in the embodiment of the present application, the reinforcement learning model is trained through the large amount of data stored on the server side, the generalization degree of the reinforcement learning model is improved, and different application scenarios and different tasks are effectively handled. Based on the continuously updated data, the reinforcement learning model is effectively trained and the accuracy of the reinforcement learning model is improved, so that the action translation model can not only translate the operator's actions in real time, but also has a learning function. As time goes by, it can automatically identify and adapt to the operator's operating habits, thereby further improving the efficiency and accuracy of the operation, thereby optimizing the action translation model and improving the generalization ability of the model.
[0227] The robot control method provided in the embodiment of the present application can be executed by a robot control device. In the embodiment of the present application, the robot control device provided in the embodiment of the present application is described by taking the robot control method executed by the robot control device as an example.
[0228] The embodiment of the present application also provides a robot control device, which is applied to a server side.
[0229] like Figure 5 As shown, the robot control device includes: a first processing module 510 , a second processing module 520 and a third processing module 530 .
[0230] A first processing module 510 is used to receive a first control instruction sent by an operation terminal;
[0231] The second processing module 520 is used to input the first control instruction into the motion translation model, and obtain the second control instruction corresponding to the target robot output by the motion translation model;
[0232] The third processing module 530 is used to send a second control instruction to the target robot; the second control instruction is used to control the target robot; wherein,
[0233] The action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
[0234] According to the robot control device provided in the embodiment of the present application, a motion translation model is trained by using sample robot instructions and sample user instructions that are recognizable by multiple categories of robots. The trained motion translation model maps the first control instruction sent by the receiving operating terminal into a second control instruction, so that in the subsequent application process, the first control instruction is input, and the motion translation model can output the second control instruction that can be recognized by the robot currently to be controlled. Therefore, it can be applied to the control of multiple categories of robots, improve the versatility of the robot control process, send the second control instruction to the target robot, and remotely control the robot to execute complex and flexible operator instructions, enhance the generalization ability of the robot in actual application scenarios, and improve the flexibility and functionality of the robot's execution of actions. In addition, by translating the control instructions through the motion translation model, the translation speed can be improved, the translation efficiency can be significantly improved, and the user experience can be improved.
[0235] In some embodiments, the second processing module 520 may also be used to:
[0236] Extracting features from the video frame to obtain position and posture information corresponding to the first object;
[0237] The posture information is mapped to obtain a second control instruction.
[0238] In some embodiments, the apparatus may further include a tenth processing module, configured to:
[0239] Extracting foreground images from video frames;
[0240] Feature extraction is performed on the joint points corresponding to the first object in the foreground image to obtain posture information.
[0241] In some embodiments, the apparatus may further include an eleventh processing module, configured to:
[0242] receiving first object feedback information, where the first object feedback information is used to characterize the situation in which the target robot executes the second control instruction; the first object feedback information is sent by the operating terminal;
[0243] The action translation model is optimized based on the feedback information of the first object.
[0244] In some embodiments, the apparatus may further include a twelfth processing module, configured to:
[0245] Pairing and encoding the first control instruction with the second control instruction corresponding to the first control instruction to obtain coding information;
[0246] Encrypting the coded information based on the target encryption algorithm to obtain encrypted information;
[0247] Send an encrypted message to the target robot.
[0248] like Figure 2 As shown, the robot control method is applied to the operating end, including: step 210, step 220, step 230 and step 240.
[0249] The first control instruction acquired in step 210 and step 220 and the first control instruction determined in step 230 are two parallel execution modes.
[0250] Step 210: The operating terminal receives a first input of a first object;
[0251] In this step, the first object is an operator or a specific user who controls the robot to perform actions.
[0252] The first input is used to receive an instruction input by a first object.
[0253] The first input may be in at least one of the following ways:
[0254] First, the first input may be a touch operation, including but not limited to a click operation, a slide operation, and a press operation.
[0255] In this implementation, receiving the first input of the first object may be receiving a touch operation of the first object in a display area of the terminal display screen.
[0256] In order to reduce the misoperation rate of the first object, the effective area of the first input can be limited to a specific area, such as the upper middle area of the control command interface; or when the control command interface is displayed, the target control is displayed on the current interface, and the first input can be achieved by touching the target control; or the first input can be set to a continuous multiple tapping operation on the display area within a target time interval.
[0257] Secondly, the first input may be a physical key input.
[0258] In this embodiment, a physical button corresponding to the control instruction is provided on the terminal body, and receiving the first input of the first object can be an operation of receiving the first object pressing the corresponding physical button; the first input can also be a combined operation of pressing multiple physical buttons at the same time.
[0259] Third, the first input may be voice input.
[0260] In this implementation, when receiving a voice such as "grab", the terminal may use "grab" as the first control instruction and send the first control instruction "grab" to the server.
[0261] Of course, in other embodiments, the first input may also be in other forms, including but not limited to character input, etc., which can be determined according to actual needs and is not limited in this embodiment of the present application.
[0262] In the case where operator control is by verbal command, this may be accomplished through a miniature microphone or other sound capture device.
[0263] The operator can also operate directly via the touch screen, for example, by swiping or clicking to give instructions.
[0264] During the actual implementation, a user console can be set up, and the operator can remotely operate the robot through a dedicated user console.
[0265] This console can be a computer or mobile device equipped with a camera and microphone.
[0266] It is understandable that the user-friendly and intuitive design of the console, coupled with tactile and auditory feedback, provides operators with a more immersive and intuitive operating experience; this not only enhances the operator's operating confidence, but also greatly improves user satisfaction.
[0267] Step 220: the operating terminal obtains a first control instruction in response to the first input;
[0268] In this step, the first control instruction is used for mapping by the server to be converted into a second control instruction that can be recognized by the target robot.
[0269] The second control instruction is used to control the target robot.
[0270] In the actual execution process, when the operating end receives the first input of the first object, in response to the first input, the first input is determined as the first control instruction.
[0271] Step 230: the operating terminal collects a video frame corresponding to the first object, and determines the video frame as a first control instruction;
[0272] In this step, the video frame corresponding to the first object is an action video frame of the first object captured by a camera or a depth sensor.
[0273] The first control instruction is an instruction corresponding to the information in the video frame.
[0274] During the actual execution process, a video frame corresponding to the action of the first object may be collected through a camera or a depth sensor, and the video frame may be determined as the first control instruction.
[0275] Step 240: The operation end sends a first control instruction to the server end.
[0276] In this step, during the actual execution process, the operating end sends the acquired first control instruction to the server end through a wireless communication method.
[0277] In the actual implementation process, a modular general control framework can be designed, which can be easily integrated with different robot models and task types. The framework supports multiple input sources, such as visual recognition, user voice commands, etc., to ensure flexibility and adaptability.
[0278] According to the robot control method provided in the embodiment of the present application, a plurality of different input methods are provided for the first object of the operating end to input the first control instruction, which has high flexibility, so that the first object can select the best input method based on the actual situation, thereby broadening the application scenarios, and can ensure the efficient transmission of control instructions, further improving the user experience.
[0279] In some embodiments, after step 240, the method may further include:
[0280] The operating end obtains the first object feedback information according to the action execution status of the target robot performing the action based on the second control instruction.
[0281] In this embodiment, the action execution status refers to the degree of matching between the result of the target robot executing the action and the first control instruction.
[0282] For example, when the matching degree is high, the first object feedback information may be motivational information.
[0283] For another example, when the matching degree is low, the first object feedback information may be loss information.
[0284] The feedback information is used to optimize the action translation model.
[0285] The action translation model may be used to map the first control instruction into the second control instruction.
[0286] During the actual execution process, a feedback system can be set up on a dedicated console, through which the operator can provide comments and suggestions on the robot operation.
[0287] It is understandable that feedback information, as a key input, helps the system understand its performance in actual applications and optimizes accordingly. According to the robot control method applied for by itself, by obtaining the evaluation of the target robot by the first object according to the action execution of the target robot based on the second control instruction, that is, the first object feedback information, the robot can be optimized based on the first object feedback information to improve the accuracy of control.
[0288] In the robot control method provided in the embodiment of the present application, the execution subject may be a robot control device. In the embodiment of the present application, the robot control device provided in the embodiment of the present application is described by taking the robot control method executed by the robot control device as an example.
[0289] An embodiment of the present application also provides a robot control device, which is applied to an operating end.
[0290] like Figure 6 As shown, the robot control device includes: a fourth processing module 610 , a fifth processing module 620 , a sixth processing module 630 and a seventh processing module 640 .
[0291] A fourth processing module 610 is configured to receive a first input from a first object;
[0292] A fifth processing module 620 is used to obtain a first control instruction in response to the first input; the first control instruction is used for mapping by the server to be converted into a second control instruction that can be recognized by the target robot; the second control instruction is used to control the target robot;
[0293] and / or,
[0294] A sixth processing module 630, configured to collect a video frame corresponding to the first object, and determine the video frame as a first control instruction;
[0295] The seventh processing module 640 is used to send a first control instruction to the server.
[0296] According to the robot control device provided in the embodiment of the present application, by receiving the first input of the first object of the operating end, in response to the first input, the first input or the collected video frame corresponding to the first object of the operating end is determined as the first control instruction, thereby effectively acquiring the control instruction of the first object, thereby controlling the robot through the first control instruction.
[0297] In some embodiments, the apparatus may further include a thirteenth processing module, configured to:
[0298] The first object feedback information is obtained according to the action execution status of the target robot performing the action based on the second control instruction.
[0299] like Figure 3 As shown, the robot control method is applied to the robot side, including: step 310 and step 320.
[0300] The robot side includes different categories of robots.
[0301] The robot is provided with multiple joints, a robotic arm and a terminal effector.
[0302] The robotic arm mimics the structure of the human arm.
[0303] The end effector is a gripping tool similar to the palm and fingers of a hand.
[0304] The bottom of the robot is provided with a plurality of motion wheels.
[0305] The robot is also equipped with navigation sensors.
[0306] The top and front of the robot are equipped with high-resolution image sensors and image processors.
[0307] In actual implementation, the image sensor can be a camera; high-resolution cameras are configured on the top and front of the robot to capture the environment and the operator's movements in real time.
[0308] The image processor is used to analyze and process the data collected by the image sensor.
[0309] In actual implementation, a powerful graphics processor can be integrated to quickly analyze the data captured by the camera and perform real-time action recognition through a pre-trained deep learning model.
[0310] The robot can be equipped with a wireless communication module.
[0311] The wireless communication module can be used to communicate with the server and the operation end.
[0312] In actual implementation, the low power consumption design of the wireless module ensures that the robot can work continuously for a long time without frequent charging due to excessive power consumption. This continuous working ability further enhances the robot's work efficiency and use value.
[0313] In the actual implementation process, a high-speed wireless communication module can be configured inside the robot to ensure real-time communication between the robot and the remote server and operator.
[0314] An encryptor may be provided in the wireless communication module.
[0315] The cipher can be used to encrypt the transmitted data.
[0316] During the actual execution process, all transmitted data is encrypted through the built-in encryptor to ensure data security.
[0317] Step 310: The robot receives a second control instruction sent by the server.
[0318] In this step, the second control instruction is generated by the server side by mapping the received first control instruction.
[0319] The second control instruction may be an encrypted control instruction.
[0320] During the actual execution process, the robot side can receive the second control instruction sent by the server side through the wireless communication module.
[0321] Step 320: The robot performs the target action based on the second control instruction.
[0322] In this step, the target action is the action that the second control instruction requires the robot to perform.
[0323] During the actual execution process, after the first object at the operating end at position A inputs the first control instruction through the operating end, the operating end sends the first control instruction to the server end, and the server end translates the first control instruction into a second control instruction that can be recognized by the target robot through the action translation model, and then sends the second control instruction to the target robot in the home of a second object at position B. After the target robot receives the second control instruction through the wireless communication module, it can collect image data of the surrounding environment through the image sensor, and process it through the image processor to obtain information about the current surrounding environment, so as to execute the target action based on the second control instruction.
[0324] In the actual implementation process, Figure 4As shown, the camera collects the video frames of the operating end to obtain the joint point information of the first object action in the video frame. The server side performs detection and fusion based on the information in the video frame and repositions based on the joint information to generate instructions that can be recognized by the robot side, so that the simulator or robot on the robot side performs the corresponding action.
[0325] In the actual implementation process, the robots can be mass-produced and placed in the home of the second object, and the first object at the operating end can control the robots through the control strategy.
[0326] It is understandable that as the server continuously collects and processes the robot's operating data and environmental information, the robot's autonomous control algorithm is continuously optimized. This enables the robot to autonomously complete some basic tasks and adapt to different operating environments without a remote operator.
[0327] According to the robot control method provided in the embodiment of the present application, by receiving the second control instruction sent by the server, the robot executes the first control instruction sent by the operator based on the recognizable second control instruction, effectively executes the target action, thereby enabling the operator to control the robot, which is suitable for different application scenarios and improves the user experience.
[0328] In the robot control method provided in the embodiment of the present application, the execution subject may be a robot control device. In the embodiment of the present application, the robot control device provided in the embodiment of the present application is described by taking the robot control method executed by the robot control device as an example.
[0329] An embodiment of the present application also provides a robot control device, which is applied to a robot end.
[0330] like Figure 7 As shown, the robot control device includes: an eighth processing module 710 and a ninth processing module 720 .
[0331] An eighth processing module, configured to receive a second control instruction sent by the server, where the second control instruction is generated by the server mapping the received first control instruction;
[0332] The ninth processing module is used to execute the target action based on the second control instruction.
[0333] According to the robot control device provided in the embodiment of the present application, by receiving the second control instruction sent by the server, the robot side executes the first control instruction sent by the operator side based on the recognizable second control instruction, effectively executes the target action, thereby enabling the operator side to control the robot side, which is suitable for different application scenarios and improves the user experience.
[0334] The robot control device in the embodiment of the present application can be a robot, or an electronic device that can communicate with the robot, or a component in the robot or electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0335] The robot control device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0336] The robot control device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0337] An embodiment of the present application also provides a robot, comprising: at least one mechanical arm, a mobile device, a processor and multiple sensors.
[0338] In this embodiment, at least one of the robotic arms includes a plurality of joints.
[0339] The robotic arm is similar in structure to the human arm.
[0340] In some embodiments, the robot arm is provided with a terminal effector, which may be a structure similar to a palm and a finger.
[0341] The terminal effector can be rotated flexibly.
[0342] According to the robot provided in the present application, when used in conjunction with an advanced visual recognition system to capture the operator's body movements, this structure ensures that the translation of movements from humans to machines is more accurate and natural. This natural movement response not only enhances the smoothness of the operation, but also reduces the possibility of misoperation.
[0343] The mechanical arm is arranged on the mobile device.
[0344] The moving means may comprise at least one scroll wheel.
[0345] The moving device can be arranged at the bottom of the robot.
[0346] The robot can move freely in complex environments through the mobile device.
[0347] Sensors are used to collect data required by the target robot.
[0348] In some embodiments, the plurality of sensors includes: an image sensor and a navigation sensor.
[0349] Among them, the image sensor is used to collect environmental image information of the location of the target robot.
[0350] Navigation sensors are used to locate the target robot and assist in path planning, etc.
[0351] The processor is electrically connected to the robot arm, the sensor and the moving device respectively.
[0352] The processor is used to receive and process the data collected by each sensor and control the robot arm or mobile device to execute relevant instructions.
[0353] It is understandable that, during the actual execution process, the target robot can integrate the environmental image information and positioning information, and perform task strategy planning based on the second control instruction to better perform the target action.
[0354] In some embodiments, the processor may include an image processor that quickly analyzes data captured by the camera and performs real-time motion recognition through a pre-trained deep learning model.
[0355] In some embodiments, the processor may further include a robot control device as described in any of the above embodiments.
[0356] The robot is used to execute the robot control method described in any of the above embodiments.
[0357] According to the robot provided in the embodiment of the present application, by receiving the second control instruction sent by the server, the robot executes the first control instruction sent by the operation end based on the recognizable second control instruction, effectively executes the target action, thereby realizing remote control, which is suitable for different application scenarios and improves the user experience.
[0358] In some embodiments, the robot further comprises: an encryptor.
[0359] In this embodiment, the encryptor is used to encrypt the transmitted data, and the specific implementation method has been described in the above embodiment and will not be repeated here.
[0360] According to the robot provided in the embodiment of the present application, by setting an encryptor, the security of data transmission can be effectively improved and the privacy of the user can be protected.
[0361] In some embodiments, Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, each process of the above-mentioned robot control method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0362] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0363] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned robot control method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0364] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0365] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned robot control method when executed by a processor.
[0366] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0367] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned robot control method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0368] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0369] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0370] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0371] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0372] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0373] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A robot control method, characterized in that: Applied to the server side, the method includes: Receiving a first control instruction sent by an operation terminal; Inputting the first control instruction into the motion translation model, and obtaining a second control instruction corresponding to the target robot output by the motion translation model; Sending the second control instruction to the target robot; the second control instruction is used to control the target robot; wherein, The action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
2. The robot control method according to claim 1, characterized in that: In the case where the first control instruction is a video frame, inputting the first control instruction into the motion translation model to obtain a second control instruction corresponding to the target robot output by the motion translation model includes: Extracting features from the video frame to obtain position and posture information corresponding to the first object; The posture information is mapped to obtain the second control instruction.
3. The robot control method according to claim 2, characterized in that: The extracting features from the video frame to obtain the posture information corresponding to the first object includes: Extracting a foreground image in the video frame; Feature extraction is performed on joint points corresponding to the first object in the foreground image to obtain the posture information.
4. The robot control method according to any one of claims 1 to 3, characterized in that: After sending the second control instruction to the target robot, the method further includes: receiving first object feedback information, where the first object feedback information is used to characterize the situation in which the target robot executes the second control instruction; the first object feedback information is sent by the operating end; The action translation model is optimized based on the first object feedback information.
5. The robot control method according to any one of claims 1 to 3, characterized in that: The sending the second control instruction to the target robot includes: Pairing and encoding the first control instruction with a second control instruction corresponding to the first control instruction to obtain coding information; Encrypting the coded information based on a target encryption algorithm to obtain encrypted information; The encrypted information is sent to the target robot.
6. A robot control method, characterized in that: Applied to the operating end, the method includes: receiving a first input from a first object; In response to the first input, a first control instruction is acquired; the first control instruction is used for mapping by the server to be converted into a second control instruction recognizable by the target robot; the second control instruction is used to control the target robot; and / or, Acquire a video frame corresponding to the first object, and determine the video frame as the first control instruction; Send the first control instruction to the server.
7. The robot control method according to claim 6, characterized in that: After sending the first control instruction to the server, the method further includes: First object feedback information is obtained according to the action execution status of the target robot based on the second control instruction; the first object feedback information is used to optimize the action translation model, and the action translation model is used to map the first control instruction to the second control instruction.
8. A robot control method, characterized in that: Applied to the robot side, the method includes: Receiving a second control instruction sent by the server, where the second control instruction is generated by the server by mapping the received first control instruction; Based on the second control instruction, a target action is executed.
9. A robot control device, characterized in that: Applied to the server side, the device comprises: A first processing module, used for receiving a first control instruction sent by an operation terminal; A second processing module, used for inputting the first control instruction into the motion translation model, and obtaining a second control instruction corresponding to the target robot output by the motion translation model; The third processing module is used to send the second control instruction to the target robot; the second control instruction is used to control the target robot; wherein, The action translation model is trained using sample user instructions as samples and sample robot instructions that can be recognized by multiple categories of robots corresponding to the sample user instructions as sample labels.
10. A robot control device, characterized in that: Applied to the operating end, the device comprises: A fourth processing module, configured to receive a first input from a first object; A fifth processing module, configured to obtain a first control instruction in response to the first input; the first control instruction is used for mapping by the server to be converted into a second control instruction recognizable by the target robot; the second control instruction is used to control the target robot; and / or, a sixth processing module, configured to collect a video frame corresponding to the first object, and determine the video frame as the first control instruction; A seventh processing module is used to send the first control instruction to the server.
11. A robot control device, characterized in that: Applied to the robot side, the device comprises: An eighth processing module, configured to receive a second control instruction sent by the server, where the second control instruction is generated by the server mapping the received first control instruction; A ninth processing module is used to execute a target action based on the second control instruction.
12. A robot, characterized in that: include: at least one robotic arm; Mobile devices; The at least one mechanical arm is disposed on the mobile device; Multiple sensors; a processor, the processor being electrically connected to the at least one mechanical arm, the mobile device, and the plurality of sensors, respectively; The robot performs actions based on the robot control method according to any one of claims 1 to 8.
13. The robot according to claim 12, characterized in that: The plurality of sensors include: Image sensor; Navigation sensor.
14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot control method according to any one of claims 1 to 8 is implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the robot control method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Interactive method and interactive system of old-age service robot
CN105141899A
Robot control method and device
CN116098536A
Robot instruction operation method and system based on natural language and medium
CN116690616A
Model adjustment method and device, electronic equipment and readable storage medium
CN117532607A
Appointment robot
US20160155094A1
Cited By
Robot control method
WO2025190181A1
