Neural network training method and device for controlling mechanical arm, equipment and medium

By using a neural network training method, a dataset is generated using cameras from two perspectives and information from the robotic arm joints. The neural network model is trained in real time to generate a sequence of robotic arm movements, which solves the problems of cumbersome and poor adaptability of traditional robotic arm control methods and realizes intelligent and adaptive control of the robotic arm.

CN118906053APending Publication Date: 2024-11-08AUTOMOTIVE WALKING TECHNOLOGY (BEIJING) CO LTD

Patent Information

Application Number
CN202411013631.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional robotic arm control methods require cumbersome manual programming or complex sensor feedback, resulting in poor adaptability.

Method used

A dataset is generated using cameras from two perspectives and information from the robotic arm joints. This dataset is then input into a neural network model for real-time training to generate a sequence of robotic arm movements, which in turn controls the robotic arm to perform actions.

Benefits of technology

It improves the flexibility and efficiency of the robotic arm, reduces reliance on manual programming, and enables the robotic arm to have intelligent sensing and learning capabilities, making it more adaptable.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118906053A_ABST
    Figure CN118906053A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a neural network training method and device for controlling a mechanical arm, equipment and a medium, and the method comprises the steps: obtaining an image shot by a camera and the joint information of the mechanical arm, and generating a data set; wherein the camera comprises a first visual angle camera and a second visual angle camera; inputting the data set to a neural network model, and training the neural network model in real time to obtain a mechanical arm action sequence; based on the mechanical arm action sequence, the mechanical arm is controlled to execute actions; according to the method, images are obtained only through cameras with two visual angles, meanwhile, the joint information of the mechanical arm is obtained, tedious manual programming or complex sensor feedback does not need to be adopted, the working flexibility and efficiency of the mechanical arm are improved, dependence of manual programming is reduced, and through real-time adjustment of the neural network, the real-time adjustment precision of the mechanical arm is improved. The intelligent sensing and learning ability of the mechanical arm is achieved, and the adaptability is higher.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robotic arm control technology, and in particular to a neural network training method, device, equipment and medium for controlling a robotic arm. Background Art

[0002] The robotic arm is the most widely used automated mechanical device in the field of robotics. It can be seen in industrial manufacturing, medical treatment, entertainment services, military, semiconductor manufacturing, space exploration and other fields. Although they have different forms, they all have one common feature, which is the ability to receive instructions and accurately locate a point in three-dimensional (or two-dimensional) space to perform operations. Robotic arms are widely used in the field of automation, while traditional robotic arm control methods often require cumbersome manual programming or complex sensor feedback, and have poor adaptability. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a neural network training method, device, equipment and medium for controlling a robotic arm, so as to solve the problem that traditional robotic arm control methods often require cumbersome manual programming or complex sensor feedback and have poor adaptability.

[0004] In a first aspect, an embodiment of the present application provides a neural network training method for controlling a robotic arm, comprising:

[0005] Acquire images taken by a camera and joint information of a robotic arm to generate a data set; wherein the camera includes a first-view camera and a second-view camera;

[0006] Input the data set into the neural network model and train the neural network model in real time to obtain the robot arm action sequence;

[0007] Based on the robot arm action sequence, control the robot arm to perform actions.

[0008] In the above implementation process, images taken by the camera and joint information of the robotic arm are obtained to generate a data set; wherein the camera includes a first-view camera and a second-view camera; the data set is input into a neural network model, and the neural network model is trained in real time to obtain a robotic arm motion sequence; based on the robotic arm motion sequence, the robotic arm is controlled to perform actions; only two cameras of different views are needed to acquire images, and at the same time, the joint information of the robotic arm itself is acquired, without the need for cumbersome manual programming or complex sensor feedback, thereby improving the working flexibility and efficiency of the robotic arm, reducing the reliance on manual programming, and realizing the intelligent perception and learning ability of the robotic arm through real-time adjustment of the neural network, with stronger adaptability.

[0009] Furthermore, the input data set is input to the neural network model, and the neural network model is trained in real time to obtain the robot arm action sequence, including:

[0010] Obtain image information and joint position information from the dataset;

[0011] Real-time training and adjustment of neural network models based on data sets;

[0012] Output the robot arm action sequence executed within the set time.

[0013] In the above implementation process, the neural network model is trained through image information and joint position information, and a robotic arm action sequence executed within a set time is generated. By training the neural network, the intelligent and adaptive control of the robotic arm is realized, thereby improving the working flexibility and efficiency of the robotic arm.

[0014] Furthermore, the real-time training and adjusting of the neural network model according to the data set includes:

[0015] Read image sample information and joint position sample information from the dataset;

[0016] According to the image sample information and the joint position sample information, the style variable parameters are obtained by using the encoder trained by the style variable training;

[0017] The feature extractor of the pre-trained convolutional neural network extracts the image information into feature information, outputs the feature information and style variable parameters to the decoder, and predicts the robot arm movements for several time steps;

[0018] Multiple iterations of the style variable parameters were performed to verify and adjust the robot arm motion.

[0019] In the above implementation process, the intelligent and adaptive control of the robotic arm is realized by training the neural network, thereby improving the working flexibility and efficiency of the robotic arm.

[0020] Furthermore, the style variable parameters are iterated multiple times to verify and adjust the robot arm motion, including:

[0021] Calculate the reconstruction loss function based on the mean square error between the current action and the predicted action;

[0022] Calculate the regularization loss function to constrain the style variable parameters output by the encoder;

[0023] Update the parameters of the encoder and decoder, and update the loss function with the reconstruction loss function and the regularization loss function:

[0024] L=L reconst +βL reg ;

[0025] Among them, Lreconst To reconstruct the loss function, L reg is the regularized loss function, and β is the regularized loss weight.

[0026] In the above implementation process, the style variable parameters are iterated multiple times to adjust the robot arm movements, thereby realizing the intelligent perception and learning capabilities of the robot arm, making it more adaptable.

[0027] Furthermore, before the input data set is input to the neural network model and the neural network model is trained in real time to obtain the robot arm action sequence, the method further includes:

[0028] Initialize the encoder and decoder.

[0029] In the above implementation process, before training the neural network model, it is necessary to initialize the encoder and decoder to ensure the accuracy of the trained model.

[0030] Furthermore, the input data set to the neural network model includes:

[0031] Initialize training parameters;

[0032] Set the action parameters and observation parameters at a time step, where the action parameters represent the image information and joint position information obtained by the camera, and the observation parameters represent the joint position information in the time series.

[0033] In the above implementation process, the parameters of the image information and key information are initialized to facilitate model training.

[0034] In a second aspect, an embodiment of the present application provides a neural network training device for controlling a robotic arm, comprising:

[0035] A data acquisition module, used to acquire image information captured by a camera and joint information of a robotic arm, and generate a data set; wherein the camera includes a first-view camera and a second-view camera;

[0036] The model generation module is used to input the data set into the neural network model and train the neural network model in real time to obtain the robot arm action sequence;

[0037] The action execution module is used to control the robot arm to perform actions based on the robot arm action sequence.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0039] A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the neural network training method for controlling a robotic arm as described above.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a server, the neural network training method for controlling a robotic arm as described above is implemented.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product comprises instructions, and when the instructions are executed by a computer, the computer implements the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a neural network training method for controlling a robotic arm provided in an embodiment of the present application;

[0044] Figure 2 A schematic flow chart of another neural network training method for controlling a robotic arm provided in an embodiment of the present application;

[0045] Figure 3 A schematic flow chart of another neural network training method for controlling a robotic arm provided in an embodiment of the present application;

[0046] Figure 4 A schematic flow chart of another neural network training method for controlling a robotic arm provided in an embodiment of the present application;

[0047] Figure 5 A data sampling schematic diagram of a neural network training method for controlling a robotic arm provided in an embodiment of the present application;

[0048] Figure 6 is a structural schematic diagram of a neural network training device for controlling a robotic arm provided in an embodiment of the present application;

[0049] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0051] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0052] Please see Figure 1 , Figure 1 A flowchart of a neural network training method for controlling a robotic arm provided in an embodiment of the present application. The neural network training method for controlling a robotic arm includes:

[0053] 100. Acquire images taken by a camera and joint information of a robotic arm to generate a data set; wherein the camera includes a first-view camera and a second-view camera.

[0054] Optionally, cameras with two perspectives are used. Exemplarily, only two image information, a camera on a robotic arm and a camera from a third-party perspective, and position information feedback from motors of each joint are used to generate a data set as input to a neural network model. The output is joint position information over a period of time. This information can guide each joint to complete a task according to specified actions, and the action of each operation will change in real time with the on-site situation. Among them, only cameras with two perspectives and joint position information of the robotic arm are used, which reduces the amount of data collection and improves the speed of training and calculation.

[0055] For example, please refer to Figure 5 , which is a schematic diagram of data sampling. Sampling is performed from the data set to obtain image information and joint position information of two-view cameras for analysis, training and prediction, and a set of robot arm action sequences are obtained.

[0056] 200. Input the data set into the neural network model, and train the neural network model in real time to obtain the robot arm action sequence.

[0057] Specifically, the data set is input into the neural network model, and the style parameters are trained and corrected in real time, and the robot arm action sequence is inferred through the model.

[0058] Specifically, the training parameters are initialized; the action parameters and observation parameters at a time step are set, wherein the action parameters represent the image information and joint position information acquired by the camera, and the observation parameters represent the joint position information in the time series; thereby, the parameters of the image information and key information are initialized to facilitate model training.

[0059] Optionally, after initializing the training parameters and setting the parameters, the encoder and decoder need to be initialized; thus, before training the neural network model, the encoder and decoder need to be initialized to ensure the accuracy of the trained model.

[0060] Specifically, Figure 2 A flowchart of another neural network training method for controlling a robotic arm provided in an embodiment of the present application is given. The neural network training method for controlling a robotic arm is a specific embodiment of the neural network training method for controlling a robotic arm. Figure 2 , the input data set is input to the neural network model, and the neural network model is trained in real time to obtain the robot arm action sequence, including:

[0061] 210. Obtain image information and joint position information from the dataset.

[0062] Optionally, the image information includes images captured by a first-view camera and images captured by a second-view camera, and the joint position information includes position information fed back by motors of each joint at intervals.

[0063] 220. Real-time training and adjustment of neural network models based on data sets.

[0064] 230. Output the robot arm action sequence executed within the set time.

[0065] Optionally, the image information and joint position information are input into a neural network model for training, and a sequence of robotic arm actions executed within a set time is output, i.e., the robotic arm is controlled to perform one or more robotic arm actions within a set time, wherein the set time is set according to demand, for example, the time for lifting an object with a robotic arm is 2s, the transportation time is 10s, the prevention time is 5s, etc.

[0066] Therefore, the neural network model is trained through image information and joint position information, and a robotic arm action sequence executed within a set time is generated. By training the neural network, the intelligent and adaptive control of the robotic arm is realized, thereby improving the working flexibility and efficiency of the robotic arm.

[0067] Optional, Figure 3 A flowchart of another neural network training method for controlling a robotic arm provided in an embodiment of the present application is given. The neural network training method for controlling a robotic arm is a specific embodiment of the neural network training method for controlling a robotic arm. Figure 3 , the real-time training and adjusting of the neural network model according to the data set includes:

[0068] 310. Read image sample information and joint position sample information from the data set.

[0069] Optionally, the data set includes image information collected from two perspectives and collected robotic arm joint position information, as well as image sample information and joint position sample information, so that sample parameters can be calculated based on the image sample information and joint position sample information to train the neural network model.

[0070] 320. According to the image sample information and the joint position sample information, the style variable parameters are obtained by using the encoder trained by the style variable training.

[0071] Specifically, according to the image sample information and the joint position sample information, the encoder trained by style variable training is used to obtain the style variable parameters, and then the neural network model is trained according to the style variable parameters to realize the prediction of the image information and the joint position information.

[0072] 330. The image information is extracted into feature information through the feature extractor of the pre-trained convolutional neural network, and the feature information and style variable parameters are output to the decoder to predict the robot arm movements for several time steps.

[0073] 340. Perform multiple iterations on the style variable parameters to verify and adjust the robot arm movements.

[0074] Therefore, by training the neural network, the intelligent and adaptive control of the robotic arm can be achieved, thereby improving the working flexibility and efficiency of the robotic arm.

[0075] In some embodiments, Figure 4 A flowchart of another neural network training method for controlling a robotic arm provided in an embodiment of the present application is given. The neural network training method for controlling a robotic arm is a specific embodiment of the neural network training method for controlling a robotic arm. Figure 4 , the style variable parameters are iterated multiple times to verify and adjust the robot arm action, including:

[0076] 410. Calculate the reconstruction loss function according to the mean square error between the current action and the predicted action.

[0077] 420. Calculate the regularization loss function to constrain the style variable parameters output by the encoder.

[0078] 430. Update the parameters of the encoder and decoder, and update the loss function using the reconstruction loss function and the regularization loss function:

[0079] L=L reconst +βL reg ;

[0080] Among them, L reconst To reconstruct the loss function, L regis the regularized loss function, and β is the regularized loss weight.

[0081] In the above implementation process, the style variable parameters are iterated multiple times to adjust the robot arm movements, thereby realizing the intelligent perception and learning capabilities of the robot arm, making it more adaptable.

[0082] Exemplarily, the training process of the neural network model is as follows:

[0083] 1. Given a demonstration dataset D, set the time block size k for periodically acquiring joint position information and the regularized loss weight β.

[0084] 2. Let a t , o t Represent the action parameters and observation parameters at time step t respectively, let o represents no image observation parameters t , o t It represents the image information and joint position information obtained by the camera with two perspectives, a t It represents the joint position information in time series.

[0085] 3. Initialize the encoder and decoder.

[0086] 4. Read sample o from dataset D t , a t:t+k .

[0087] 5. Encoder for training style variable parameters Q The style variable parameter Q is obtained through training.

[0088] 6. Use the pre-trained convolutional neural network ResNet18 as a feature extractor to extract feature information from the input observation image data, and pass the style variable parameter Q and the extracted feature information into the decoder In the example, the action of k time steps is predicted

[0089] 7. Calculate the reconstruction loss function L based on the mean square error between the current action and the predicted action reconst .

[0090] 8. The regularized loss function L is calculated by DKL reg , the regularization loss function is used to constrain the style variable parameter Q output by the encoder so that the style variable parameter Q is represented within the standard normal distribution.

[0091] 9. Update the parameters θ, φ of the encoder and decoder using the ADAM algorithm, and update the loss function using the reconstruction loss function and the regularization loss function:

[0092] L=Lreconst +βL reg

[0093] 10. Repeat steps 4-9 until the iteration ends.

[0094] 300. Based on the robot arm action sequence, control the robot arm to perform actions.

[0095] Optionally, a multi-frame synchronous action sequence uses an exponential decay function to synthesize multi-frame action values ​​to obtain a smooth motion execution action for controlling the robot arm.

[0096] Specifically, a robot arm motion sequence within a set time is obtained through neural network model training, and according to the robot arm motion sequence, the robot arm is controlled to perform one or more actions within the set time. Optionally, the robot arm motion sequence can be an action executed for a certain time, or a series of actions, each action having a corresponding execution time.

[0097] The embodiment of the present application only uses two image information, namely a camera on a robotic arm and a camera from a third-party perspective, as well as the position information fed back by the motors of each joint as input, and outputs the joint position information within a period of time. This information can guide each joint to complete the task according to the specified action, and the action of each operation will change in real time with the on-site situation; the amount of data collection is reduced, and the speed of training and calculation is improved; the data set is collected from actual working conditions, and the content is a series of joint position information when the robotic arm operated by a person completes the target action.

[0098] In addition, the style variable parameters were introduced and modified so that the variable can be used to distinguish different situations that occur during the operation of the robotic arm, provide diversity for the action, and reduce compound errors, making the generated action sequence coherent and accurate. The overall structure of the training adopts a neural network solution, using the pre-trained convolutional neural network ResNet18 to provide the Transformers encoder with image feature information, and then the Transformers encoder and decoder obtain the style variables and predicted action sequences respectively.

[0099] At the same time, based on data feedback, a certain number of iterations is set, and training is terminated early when the performance on the validation set no longer improves to avoid invalid iterations.

[0100] As described above, the embodiment of the present application obtains images taken by a camera and joint information of a robotic arm to generate a data set; wherein the camera includes a first-perspective camera and a second-perspective camera; the data set is input into a neural network model, and the neural network model is trained in real time to obtain a robotic arm motion sequence; based on the robotic arm motion sequence, the robotic arm is controlled to perform actions; only two cameras of different perspectives are needed to acquire images, and at the same time, the joint information of the robotic arm itself is acquired, without the need for cumbersome manual programming or complex sensor feedback, thereby improving the working flexibility and efficiency of the robotic arm, reducing dependence on manual programming, and realizing intelligent perception and learning capabilities of the robotic arm through real-time adjustment of the neural network, with stronger adaptability.

[0101] The above steps are not to be performed in a strict order as described in the numbers, but should be understood as an overall solution.

[0102] In the second aspect, based on the above embodiments, Figure 6 A schematic diagram of the structure of a neural network training device for controlling a robotic arm provided in an embodiment of the present application. Figure 6 The neural network training device for controlling a robotic arm provided in this embodiment specifically includes: a data acquisition module 601, a model generation module 602 and an action execution module 603.

[0103] Among them, the data acquisition module 601 is used to obtain image information taken by the camera and the joint information of the robotic arm to generate a data set; wherein the camera includes a first-view camera and a second-view camera; the model generation module 602 is used to input the data set into the neural network model, and train the neural network model in real time to obtain the robotic arm action sequence; the action execution module 603 is used to control the robotic arm to perform actions based on the robotic arm action sequence.

[0104] The embodiment of the present application only uses two image information, namely a camera on a robotic arm and a camera from a third-party perspective, as well as the position information fed back by the motors of each joint as input, and outputs the joint position information within a period of time. This information can guide each joint to complete the task according to the specified action, and the action of each operation will change in real time with the on-site situation; the amount of data collection is reduced, and the speed of training and calculation is improved; the data set is collected from actual working conditions, and the content is a series of joint position information when the robotic arm operated by a person completes the target action.

[0105] In addition, the style variable parameters were introduced and modified so that the variable can be used to distinguish different situations that occur during the operation of the robotic arm, provide diversity for the action, and reduce compound errors, making the generated action sequence coherent and accurate. The overall structure of the training adopts a neural network solution, using the pre-trained convolutional neural network ResNet18 to provide the Transformers encoder with image feature information, and then the Transformers encoder and decoder obtain the style variables and predicted action sequences respectively.

[0106] At the same time, based on data feedback, a certain number of iterations is set, and training is terminated early when the performance on the validation set no longer improves to avoid invalid iterations.

[0107] As described above, the embodiment of the present application obtains images taken by a camera and joint information of a robotic arm to generate a data set; wherein the camera includes a first-perspective camera and a second-perspective camera; the data set is input into a neural network model, and the neural network model is trained in real time to obtain a robotic arm motion sequence; based on the robotic arm motion sequence, the robotic arm is controlled to perform actions; only two cameras of different perspectives are needed to obtain images, and at the same time, the joint information of the robotic arm itself is obtained, without the need for cumbersome manual programming or complex sensor feedback, thereby improving the working flexibility and efficiency of the robotic arm, reducing the reliance on manual programming, and realizing the intelligent perception and learning ability of the robotic arm through real-time adjustment of the neural network, with stronger adaptability.

[0108] The neural network training device for controlling a robotic arm provided in an embodiment of the present application can be used to execute the neural network training method for controlling a robotic arm provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0109] In a third aspect, an embodiment of the present application further provides an electronic device that can integrate the neural network training device for controlling a robotic arm provided in an embodiment of the present application. Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 The electronic device includes: an input device 73, an output device 74, a memory 72 and one or more processors 71; the memory 72 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 71 implement the neural network training method for controlling the robot arm as provided in the above embodiment. The input device 73, the output device 74, the memory 72 and the processor 71 can be connected by a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0110] The processor 71 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 72, that is, implements the above-mentioned neural network training method for controlling the robotic arm.

[0111] The electronic device provided above can be used to execute the neural network training method for controlling a robotic arm provided in the above embodiment, and has corresponding functions and beneficial effects.

[0112] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the neural network training method for controlling a robotic arm as described above, and can achieve the same beneficial effects.

[0113] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present application, whose computer executable instructions are not limited to the neural network training method for controlling a robotic arm as described above, can also execute related operations in the neural network training method for controlling a robotic arm provided in any embodiment of the present application.

[0114] In a fifth aspect, the embodiments of the present application also provide a computer program product, and the methods described in the various embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the various embodiments of the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model) or other programmable device.

[0115] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0116] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0118] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0119] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0120] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0121] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A neural network training method for controlling a robotic arm, It is characterized in that include: Acquire image information captured by a camera and joint information of a robotic arm to generate a data set; wherein the camera includes a first-view camera and a second-view camera; Input the data set into the neural network model and train the neural network model in real time to obtain the robot arm action sequence; Based on the robot arm action sequence, control the robot arm to perform actions.

2. The neural network training method for controlling a robotic arm according to claim 1, It is characterized in that The input data set is fed into the neural network model, and the neural network model is trained in real time to obtain the robot arm action sequence, including: Obtain image information and joint position information from the dataset; Real-time training and adjustment of neural network models based on data sets; Output the robot arm action sequence executed within the set time.

3. The neural network training method for controlling a robotic arm according to claim 2, It is characterized in that The real-time training and adjusting of the neural network model according to the data set includes: Read image sample information and joint position sample information from the dataset; According to the image sample information and the joint position sample information, the style variable parameters are obtained by using the encoder trained by the style variable training; The feature extractor of the pre-trained convolutional neural network extracts the image information into feature information, outputs the feature information and style variable parameters to the decoder, and predicts the robot arm movements for several time steps; Multiple iterations of the style variable parameters were performed to verify and adjust the robot arm motion.

4. The neural network training method for controlling a robotic arm according to claim 3, It is characterized in that The style variable parameters are iterated multiple times to verify and adjust the robot arm motion, including: Calculate the reconstruction loss function based on the mean square error between the current action and the predicted action; Calculate the regularization loss function to constrain the style variable parameters output by the encoder; Update the parameters of the encoder and decoder, and update the loss function with the reconstruction loss function and the regularization loss function: L=L reconst +βL reg ; Among them, L reconst To reconstruct the loss function, L reg is the regularized loss function, and β is the regularized loss weight.

5. The neural network training method for controlling a robotic arm according to claim 1, It is characterized in that The input data set is input to the neural network model, and the neural network model is trained in real time to obtain the robot arm action sequence, and the following steps are also included: Initialize the encoder and decoder.

6. The neural network training method for controlling a robotic arm according to claim 1, It is characterized in that The input data set to the neural network model includes: Initialize training parameters; Set the action parameters and observation parameters at a time step, where the action parameters represent the image information and joint position information obtained by the camera, and the observation parameters represent the joint position information in the time series.

7. A neural network training device for controlling a robotic arm, It is characterized in that include: A data acquisition module, used to acquire image information captured by a camera and joint information of a robotic arm, and generate a data set; wherein the camera includes a first-view camera and a second-view camera; The model generation module is used to input the data set into the neural network model and train the neural network model in real time to obtain the robot arm action sequence; The action execution module is used to control the robot arm to perform actions based on the robot arm action sequence.

8. An electronic device, It is characterized in that include: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions, which, when executed by the processor, are used to implement the neural network training method for controlling a robotic arm as described in any one of claims 1 to 6.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a server, implements a neural network training method for controlling a robotic arm as described in any one of claims 1-6.

10. A computer program product, It is characterized in that The computer program product comprises instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1-6.

Citation Information

Patent Citations

  • Robot remote teaching system based on virtual reality

    CN107263449A

  • Mechanical arm controlling method and device and neural network model training method and device

    CN110293549A

  • Mechanical arm control method and device, control device and storage medium

    CN110293552A

  • Hardware self-adaptive mechanical arm control calibration method based on neural network

    CN117047770A

  • Super-redundant snakelike mechanical arm control method and device based on reinforcement learning

    CN117601131A

Cited By

  • Track smooth control method, device and equipment for mechanical arm and storage medium

    CN121893288A

  • Trajectory smoothing control method, device and equipment of mechanical arm and storage medium

    CN121893288B