Robot control method and device, robot and readable storage medium
By combining data enhancement technology and target encoding network to extract visual features and use target instruction prediction networks for instruction prediction, the problem of poor robot motion control accuracy under complex lighting conditions is solved, and higher adaptability and accuracy are achieved.
Patent Information
- Application Number
- CN202411803648.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-03-07
AI Technical Summary
The prior art is difficult to maintain high accuracy under complex lighting conditions in visual motion control, resulting in poor robot motion control accuracy.
The target encoding network trained by combining data augmentation technology extracts visual features, and uses the target instruction prediction network to predict based on visual features and joint position information, thereby controlling the robot motion.
It enhances the adaptability of the robot to complex lighting conditions, improves the robot's motion control accuracy, and reduces the impact of lighting conditions on control accuracy.
Smart Images
Figure CN119567258A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robot technology, and in particular to a robot control method, a robot control device, a robot and a readable storage medium. Background Art
[0002] With the rapid development of robotics technology, visual motion control has become an important part of autonomous navigation and decision-making of intelligent robots. In visual motion control, it is necessary to determine the control instructions of the robot based on the collected images and then control the robot. Changes in lighting conditions may cause different motion control instructions determined based on images in the same scene, that is, poor motion control accuracy. Therefore, how to improve the accuracy of visual motion control is a technical problem that technicians in this field need to solve urgently. Summary of the invention
[0003] The embodiments of the present application provide a robot control method, device, robot and readable storage medium, which can use a target encoding network trained in combination with data enhancement technology to extract visual features in an image, and then use a target command prediction network to predict commands based on visual features and current joint position information, and move the robot based on the predicted commands, thereby enhancing adaptability to complex lighting conditions and improving the robot's motion control accuracy.
[0004] The embodiments of the present application can be implemented as follows:
[0005] In a first aspect, an embodiment of the present application provides a method for controlling a robot, the method comprising:
[0006] Obtaining a first visual image and first joint position information of the robot at a first moment, wherein the first joint position information is used to indicate a joint position state of the robot at the first moment;
[0007] Obtaining a first visual feature based on the first visual image through a target encoding network, wherein the target encoding network is a network trained based on the image after data enhancement processing;
[0008] Obtaining an initial joint position control instruction based on the first visual feature and the first joint position information through a target instruction prediction network, wherein the target instruction prediction network is trained according to the corresponding second visual feature, the second joint position information and the joint position control instruction;
[0009] The robot movement is controlled according to the initial joint position control instruction.
[0010] In a second aspect, an embodiment of the present application provides a control device for a robot, the device comprising:
[0011] An information acquisition module, used to obtain a first visual image and first joint position information of the robot at a first moment, wherein the first joint position information is used to indicate a joint position state of the robot at the first moment;
[0012] A feature acquisition module, configured to acquire a first visual feature based on the first visual image through a target encoding network, wherein the target encoding network is a network trained based on an image after data enhancement processing;
[0013] An analysis module, configured to obtain an initial joint position control instruction based on the first visual feature and the first joint position information through a target instruction prediction network, wherein the target instruction prediction network is trained according to the corresponding second visual feature, the second joint position information and the joint position control instruction;
[0014] A motion control module is used to control the motion of the robot according to the initial joint position control instruction.
[0015] In a third aspect, an embodiment of the present application provides a robot, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the robot control method described in the aforementioned embodiment.
[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the robot control method as described in the aforementioned embodiment.
[0017] The control method, device, robot and readable storage medium of the robot provided in the embodiment of the present application, when obtaining the first visual image of the robot at the first moment, obtain the first visual feature of the first visual image through the target encoding network obtained by training the image after data enhancement processing, and obtain the first joint position information of the robot at the first moment, and then use the target instruction prediction network to obtain the initial joint position control instruction based on the above-mentioned first visual feature and the first joint position information, and finally control the robot movement according to the initial joint position control instruction. Wherein, the above-mentioned first joint position information is used to indicate the joint position state of the robot at the first moment, and the target instruction prediction network is trained according to the corresponding second visual feature, second joint position information and joint position control instruction. In this way, the adaptability to complex lighting conditions can be enhanced, the robot motion control accuracy can be improved, and the influence of complex lighting conditions on the robot motion control accuracy can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 A schematic block diagram of a robot provided in an embodiment of the present application;
[0020] Figure 2 One of the flowcharts of the robot control method provided in the embodiment of the present application;
[0021] Figure 3 A schematic diagram of the principle of the robot control method provided in the embodiment of the present application;
[0022] Figure 4 A second flowchart of the robot control method provided in an embodiment of the present application;
[0023] Figure 5 One of the block diagrams of the control device of the robot provided in the embodiment of the present application;
[0024] Figure 6 A schematic diagram of a control device for a robot provided in an embodiment of the present application Figure 2 .
[0025] Icons: 20 - robot; 21 - memory; 22 - processor; 23 - communication unit; 300 - robot control device; 301 - model training module; 310 - information acquisition module; 320 - feature acquisition module; 330 - analysis module; 340 - motion control module. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0027] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0028] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0030] Please refer to Figure 1 , Figure 1 The block diagram of the robot 20 provided in the embodiment of the present application. The robot 20 may include a memory 21, a processor 22 and a communication unit 23. The memory 21, the processor 22 and the communication unit 23 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.
[0031] The memory 21 is used to store programs or data. The memory 21 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.
[0032] The processor 22 is used to read / write data or programs stored in the memory 21 and execute corresponding functions. For example, the memory 21 stores a robot control device 300, and the robot control device 300 includes at least one software function module that can be stored in the memory 21 in the form of software or firmware. The processor 22 executes various functional applications and data processing by running software programs and modules stored in the memory 21, such as the robot control device 300 in the embodiment of the present application, that is, realizing the robot control method in the embodiment of the present application.
[0033] The communication unit 23 is used to establish a communication connection between the robot 20 and other communication terminals through the network, and to send and receive data through the network.
[0034] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the robot 20. The robot 20 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0035] Please refer to Figure 2 , Figure 2 This is one of the flow diagrams of the robot control method provided in the embodiment of the present application. The method can be applied to the above-mentioned robot. The specific flow of the robot control method is described in detail below. In this embodiment, the method may include steps S110 to S140.
[0036] Step S110, obtaining a first visual image and first joint position information of the robot at a first moment.
[0037] In this embodiment, the robot may be provided with a camera for capturing images. The image captured by the camera at the first moment may be obtained as a first visual image, and the first joint position information of the robot at the first moment may be obtained. The first joint position information is used to indicate the joint position state of the robot at the first moment, and can also be used to indicate the angle of the motor of each joint of the robot. The first moment is the current moment. The first visual image and the first joint position information are used to describe the environmental state and joint state of the robot at the current moment.
[0038] Step S120: obtaining a first visual feature based on the first visual image through a target encoding network.
[0039] In this embodiment, the target encoding network can be obtained in advance based on the image training after data enhancement processing. When the first visual image is obtained, the target encoder network can be used to obtain the first visual feature based on the first visual image. That is, the first visual feature is an image feature obtained based on the first visual image.
[0040] Step S130, obtaining an initial joint position control instruction based on the first visual feature and the first joint position information through a target instruction prediction network.
[0041] A target instruction prediction network may be obtained in advance. The target instruction prediction network is trained according to the corresponding second visual feature, second joint position information and joint position control instruction. When the first visual feature and the first joint position information are obtained, the first visual feature and the first joint position information may be input into the target instruction prediction network, and the initial joint position control instruction may be obtained from the output result of the target instruction prediction network.
[0042] Step S140, controlling the movement of the robot according to the initial joint position control instruction.
[0043] After obtaining the initial joint position control instruction, the movement of the robot can be controlled according to the initial joint position control instruction. Optionally, the movement of the robot can be directly controlled according to the initial joint position control instruction, so that the joint position state of the robot at the second moment is the joint position state indicated by the initial joint position control instruction. The second moment is the next moment of the first moment. The initial joint position control instruction can also be processed, and then the movement of the robot can be controlled according to the initial joint position control instruction after processing. In this case, the joint position state of the robot at the second moment is the joint position state indicated by the initial joint position control instruction after processing.
[0044] In this embodiment, a target encoding network trained in combination with data enhancement technology is used to extract visual features in the image, and then a target command prediction network is used to predict commands based on the visual features and current joint position information, and the robot is moved based on the predicted commands, thereby enhancing the adaptability to complex lighting conditions and improving the robot's motion control accuracy.
[0045] In this embodiment, the number of cameras on the robot and their specific setting positions can be determined in combination with actual needs. As a possible implementation method, the robot's wrist and / or head are both provided with cameras, and the image obtained by the camera of the robot's wrist and / or head at the first moment can be used as the first visual image. Among them, the camera of the wrist can obtain an image only from one perspective, or can obtain an image from two different perspectives respectively (that is, the wrist camera can obtain two images at the same time). It can be understood that the type of image used when training the target instruction prediction network is the same as the type of the first visual image at the first moment, for example, both are images obtained by the camera of the head, or both include images obtained by the camera of the wrist and the camera of the head. In this way, it is convenient to improve the upper limb movement control ability of the robot.
[0046] The position of the joint of the robot at the first moment can also be obtained in any manner to obtain the first joint position information. The specific manner of obtaining the first joint position information can be determined in combination with actual needs and is not specifically limited here.
[0047] In this embodiment, the target encoding network may be a ResNet18 convolutional network, a DenseNet network, etc., which may be determined in combination with actual needs. As a possible implementation, Figure 3 As shown, the target encoding network is a ResNet18 convolutional network.
[0048] When the first visual image is obtained, the first visual image can be input into a target encoding network for feature extraction to obtain the first visual feature.
[0049] In this embodiment, the target instruction prediction network includes an encoder subnetwork and a decoder subnetwork. The encoder subnetwork can obtain a latent variable based on the first visual feature and the first joint position information, and then the decoder subnetwork can obtain the initial joint position control instruction based on the obtained latent variable.
[0050] As a possible implementation, Figure 3 As shown, the target instruction prediction network is a conditional variational autoencoder (CVAE), including a CVAE encoder (i.e. Figure 3 CVAE (encoder) in CVAE decoder (i.e. Figure 3 The first visual feature U and the first joint position information (i.e. Figure 3Joint(P))y in is input into CVAE (encoder) to generate a latent variable z. This latent variable captures the joint representation of visual features and joint position state, and the formula is: z~q(z|U,y)=N(μ(U,y),σ 2 (U,y)), where μ(U,y) represents the mean calculated by the encoder (i.e., the mean of the latent variable), σ 2 (U, y) represents the variance calculated by the encoder (i.e., the variance of the latent variable), μ(U, y) and σ 2 (U, y) reflects the potential distribution of visual features and joint position states. Next, the above latent variable z is input into CVAE (decoder), and CVAE (decoder) decodes the latent variable z to generate a preliminary joint control signal (i.e., the initial joint position control instruction) CVAE (encoder) and CVAE (decoder) are both transform networks.
[0051] As a possible implementation, when the initial joint position control instruction is obtained, the target joint position control instruction corresponding to the initial joint position control instruction can be obtained by smoothing the historical joint position control instructions used by the robot before the first moment and the initial joint position control instruction, and then the robot movement is controlled according to the target joint position control instruction. The target joint position control instruction is used to indicate the joint position of the robot at the second moment after the first moment, that is, after executing the target joint position control instruction, the joint position of the robot at the second moment is the position indicated by the target joint position control instruction. In this way, the jitter caused by directly following the initial joint position control instruction can be avoided.
[0052] Alternatively, if Figure 3 As shown, the moving average filter includes a preset number of historical joint position control instructions and an initial joint position control instruction. The moving average filter calculates the mean of the preset number of historical joint position control instructions and the initial joint position control instruction, and uses the mean as the target joint position control instruction corresponding to the initial joint position control instruction.
[0053] Please refer to Figure 4 Figure 4 The second flowchart of the robot control method provided in the embodiment of the present application. In this embodiment, before step S110, the method may further include steps S101 to S103.
[0054] Step S101, obtaining a sample set.
[0055] Each sample in the sample set includes a second visual image, second joint position information and a joint position control instruction corresponding to the same moment.
[0056] Step S102, performing data enhancement processing on the second visual image in the sample to obtain a target sample, and performing training based on the target sample to obtain a control model.
[0057] Among them, the target sample includes a target visual image, second joint position information and joint position control instructions, and the control model includes the target encoding network and the target instruction prediction network.
[0058] In this embodiment, a large number of visual images obtained by the camera of the robot and their corresponding joint position data (i.e., joint point position information) and joint position control instructions can be collected. The image data undergoes data enhancement processing during the training process to generate diversified training sample images to enhance the adaptability of the control model to different lighting conditions. That is, by performing data enhancement processing on the second visual image in the sample set, the target sample set is obtained.
[0059] Optionally, the second visual image may be subjected to at least one of brightness adjustment, rotation, translation, and scaling to obtain a target visual image (i.e., a training sample image). The brightness adjustment amount, rotation angle, translation method, and scaling scale may be preset values or random values, which may be determined in combination with actual needs. In this embodiment, the image brightness may be randomly changed in brightness adjustment, the image may be randomly rotated in rotation processing, the image may be randomly translated in translation processing, and the image may be randomly scaled in scaling processing.
[0060] That is, assuming that the second visual image is x, the brightness adjustment (Brightness Adjustment) of the second visual image x is performed in the following manner: by randomly changing the image brightness, simulating different lighting conditions, the enhanced image is expressed as x'=x+Δb, Δb is a brightness increment randomly selected within a predetermined range. The second visual image x can also be rotated (Rotation) in the following manner: randomly rotating the image by a certain angle to enhance the adaptability of the model to angle changes, the rotated image is expressed as: x'=R(θ)·x, R(θ) represents the rotation matrix, and θ represents the randomly selected rotation angle. The second visual image x can also be translated (Translation) in the following manner: randomly translating the image horizontally or vertically to simulate the changes of objects in different positions, the translated image is expressed as: x'=T(dx,dy)·x, T(dx,dy) represents the translation matrix, dx represents the translation distance in the horizontal direction, and dy represents the translation distance in the vertical direction. The second visual image x can also be scaled (Scaling) in the following manner: by randomly scaling the image size to simulate the change in the object's distance from the camera, the scaled image is represented by: x'=S(s)·x, S(s) represents the scaling matrix, and s represents the randomly selected scaling factor.
[0061] The target encoding network can be trained by supervised learning so that the target encoding network can extract joint features in the image. During the training process, a cross entropy loss function can be used to optimize the network weights to ensure that the model can correctly extract visual features.
[0062] Alternatively, if Figure 3 As shown, the second visual image (Image data) is data enhanced to obtain a target visual image, and then the target visual image can be input into a ResNet encoder as a target encoding network for feature extraction. The ResNet network processes the target visual image through multiple residual blocks to generate a visual feature vector U, which is expressed as: U = f ResNet (x'), f ResNet ResNet represents the feature extraction function of ResNet, and U represents the deep feature representation of the image, that is, U is the second visual feature used in model training. ResNet18 alleviates the gradient vanishing problem in deep network training through residual connections and enhances the expressiveness of image features.
[0063] Optionally, the target instruction prediction network is a CVAE model. The CVAE model is trained by maximizing likelihood estimation, with the goal of minimizing reconstruction loss and KL divergence loss:
[0064] Γ CVAE =E q(z|U,y) [logp(y|z)]-DKL ([q(z|U,y)||p(z)])
[0065] The reconstruction loss is used to minimize the difference between the joint position control instructions generated by the decoder and the true joint position control instructions, and the KL divergence loss is used to ensure the consistency between the distribution of the latent variable z and the prior distribution p(z).
[0066] It is understood that in the above process, Figure 3 As shown, when the sample set is obtained, samples are obtained from the sample set, and then data enhancement processing is performed on the images in the samples, and then the initial control model is trained; thereafter, the above processing is repeated until a trained control model is obtained; or data enhancement processing can be performed on the images in the complete sample set first, and then model training is performed.
[0067] If visual features are directly extracted from the images captured by the camera, and then combined with the joint position information for command prediction, the image feature extraction may be unstable due to changes in lighting conditions, thereby affecting the robot's motion control accuracy. In order to avoid the above situation, in this embodiment, data enhancement technology is used as an important means to improve the robustness of the model; by performing a series of enhancement operations (such as brightness adjustment, rotation, translation, etc.) before the image data is input, the control model's adaptability to complex lighting conditions can be effectively enhanced. At the same time, the CVAE model can generate a more accurate motion control strategy by jointly modeling visual data and joint states.
[0068] In the above method provided in the present embodiment, visual feature extraction is performed based on data enhancement technology and ResNet18 convolutional network architecture, and model training is performed based on the CVAE framework, so as to obtain a control model for generating a visual motion control strategy. The above method introduces data enhancement technology in the visual data input stage in order to improve the influence of illumination changes on image feature extraction. The enhanced image data is encoded through the ResNet18 network to extract deep visual features, and is input into the CVAE model together with the robot joint position state. The CVAE encoder encodes the visual features and joint state information into a latent variable z, and then trains the latent variable z, the joint position and the image features processed by ResNet18 together through the CVAE encoder and decoder to obtain a visual motion control model. Finally, the above control model can be used to predict the initial joint position control instruction, and the robot movement can be controlled based on the initial joint position control instruction. The above method combines data augmentation technology, ResNet network and CVAE model to significantly improve the robot's visual feature extraction ability under complex lighting conditions, generate more accurate motion control signals, and is suitable for robot control tasks in various complex environments. It is expected to improve the generalization ability and stability of robot visual motion control under a wider range of environmental conditions.
[0069] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing a robot control device 300 is given below. Optionally, the robot control device 300 can adopt the above Figure 1 The device structure of the robot 20 shown in FIG. Figure 5 , Figure 5 This is one of the block diagrams of the robot control device 300 provided in the embodiment of the present application. It should be noted that the basic principle and technical effects of the robot control device 300 provided in this embodiment are the same as those in the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiment. In this embodiment, the robot control device 300 may include: an information acquisition module 310, a feature acquisition module 320, an analysis module 330 and a motion control module 340.
[0070] The information acquisition module 310 is used to obtain a first visual image and first joint position information of the robot at a first moment, wherein the first joint position information is used to indicate the joint position state of the robot at the first moment.
[0071] The feature acquisition module 320 is used to acquire a first visual feature based on the first visual image through a target encoding network, wherein the target encoding network is a network trained based on an image after data enhancement processing.
[0072] The analysis module 330 is used to obtain an initial joint position control instruction based on the first visual feature and the first joint position information through a target instruction prediction network, wherein the target instruction prediction network is trained according to the corresponding second visual feature, the second joint position information and the joint position control instruction.
[0073] The motion control module 340 is used as a motion control module, and is used to control the movement of the robot according to the initial joint position control instruction.
[0074] Please refer to Figure 6 Figure 6 A schematic diagram of a robot control device 300 provided in an embodiment of the present application Figure 2 . In this embodiment, the control device 300 of the robot may further include a model training module 301. The model training module 301 is used to: obtain a sample set, wherein each sample in the sample set includes a second visual image, second joint position information and joint position control instructions corresponding to the same moment; perform data enhancement processing on the second visual image in the sample to obtain a target sample, and perform training based on the target sample to obtain a control model, wherein the control model includes the target encoding network and the target instruction prediction network.
[0075] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 21 shown in the figure may be fixed in the operating system (OS) of the robot 20 and may be Figure 1 The processor 22 in the processor 22 is executed. At the same time, the data, program codes, etc. required to execute the above modules can be stored in the memory 21.
[0076] An embodiment of the present application also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot control method is implemented.
[0077] In summary, the embodiments of the present application provide a control method, device, robot and readable storage medium of a robot. In the case of obtaining a first visual image of the robot at a first moment, the first visual feature of the first visual image is obtained by a target encoding network obtained by image training after data enhancement processing, and the first joint position information of the robot at the first moment is obtained, and then the target instruction prediction network is used to obtain the initial joint position control instruction based on the above-mentioned first visual feature and the first joint position information, and finally the robot movement is controlled according to the initial joint position control instruction. Among them, the above-mentioned first joint position information is used to indicate the joint position state of the robot at the first moment, and the target instruction prediction network is trained according to the corresponding second visual feature, the second joint position information and the joint position control instruction. In this way, the adaptability to complex lighting conditions can be enhanced, the robot motion control accuracy can be improved, and the influence of complex lighting conditions on the robot motion control accuracy can be reduced.
[0078] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0079] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0080] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0081] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A robot control method, characterized in that: The method comprises: Obtaining a first visual image and first joint position information of the robot at a first moment, wherein the first joint position information is used to indicate a joint position state of the robot at the first moment; Obtaining a first visual feature based on the first visual image through a target encoding network, wherein the target encoding network is a network trained based on the image after data enhancement processing; Obtaining an initial joint position control instruction based on the first visual feature and the first joint position information through a target instruction prediction network, wherein the target instruction prediction network is trained according to the corresponding second visual feature, the second joint position information and the joint position control instruction; The robot movement is controlled according to the initial joint position control instruction.
2. The method according to claim 1, characterized in that The target instruction prediction network includes an encoder subnetwork and a decoder subnetwork, and obtaining an initial joint position control instruction based on the first visual feature and the first joint position information through the target instruction prediction network includes: Obtaining a latent variable based on the first visual feature and the first joint position information through the encoder sub-network; The initial joint position control instruction is obtained through the decoder sub-network based on the obtained latent variables.
3. The method according to any one of claims 1 to 2, characterized in that: The controlling the robot movement according to the initial joint position control instruction comprises: Obtaining a target joint position control instruction by performing smoothing processing according to the historical joint position control instructions used by the robot before the first moment and the initial joint position control instructions; The movement of the robot is controlled according to the target joint position control instruction, wherein the target joint position control instruction is used to indicate the joint position of the robot at a second moment after the first moment.
4. The method according to any one of claims 1 to 2, characterized in that: The method comprises: Obtaining a sample set, wherein each sample in the sample set includes a second visual image, second joint position information, and a joint position control instruction corresponding to the same moment; The second visual image in the sample is subjected to data enhancement processing to obtain a target sample, and training is performed based on the target sample to obtain a control model, wherein the control model includes the target encoding network and the target instruction prediction network.
5. The method according to claim 4, characterized in that The performing data enhancement processing on the second visual image in the sample to obtain a target sample includes: The second visual image in the sample is subjected to at least one of brightness adjustment, rotation, translation, and scaling to obtain a target visual image, wherein the target sample includes the target visual image, second joint position information, and joint position control instructions.
6. The method according to claim 5, characterized in that The brightness of the image is randomly changed in the brightness adjustment, the image is randomly rotated in the rotation processing, the image is randomly translated in the translation processing, and the image is randomly scaled in the scaling processing.
7. The method according to claim 4, characterized in that The target encoding network is a ResNet18 convolutional network, and the target instruction prediction network is a conditional variational autoencoder.
8. A robot control device, characterized in that: The device comprises: An information acquisition module, used to obtain a first visual image and first joint position information of the robot at a first moment, wherein the first joint position information is used to indicate a joint position state of the robot at the first moment; A feature acquisition module, configured to acquire a first visual feature based on the first visual image through a target encoding network, wherein the target encoding network is a network trained based on an image after data enhancement processing; An analysis module, configured to obtain an initial joint position control instruction based on the first visual feature and the first joint position information through a target instruction prediction network, wherein the target instruction prediction network is trained according to the corresponding second visual feature, the second joint position information and the joint position control instruction; A motion control module is used to control the motion of the robot according to the initial joint position control instruction.
9. A robot, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the robot control method described in any one of claims 1-7.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot control method according to any one of claims 1 to 7 is implemented.