A visual servoing method, device, equipment and medium based on a diffusion model

Abstract

This invention relates to the fields of robotics and information technology, and discloses a visual servoing method, apparatus, device, and medium based on a diffusion model. The method includes: forming a training sample by combining a preset tool image, a preset teaching image corresponding to the preset tool image, and a real desired image corresponding to the preset tool image; training a diffusion model based on the training sample; stopping the training of the diffusion model when the total loss value is less than a preset value, thus obtaining a trained diffusion model; fusing the feature vectors of the current tool image and the current teaching image to obtain a second fused vector; processing the second fused vector through a decoder in the trained diffusion model to obtain the current desired image; transmitting the current desired image to the robot's visual servoing system, and controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image. This invention can provide the current desired image to the robot's visual servoing system.

Description

Technical Field

[0001] This invention relates to the fields of robotics and information technology, and in particular to a visual servoing method, apparatus, device, and medium based on a diffusion model. Background Technology

[0002] The desired image represents the visual state or ideal appearance of the target object that the visual servoing system hopes to achieve. The desired image plays a crucial role in a visual servoing system.

[0003] However, current technologies primarily rely on manual acquisition to obtain the desired image. This manual acquisition method increases the time required and hinders efficiency. Therefore, how to acquire the desired image is a pressing technical problem. Furthermore, without a desired image, the robot's visual servoing system will blindly attempt various movements, trying to find the correct motion path through trial and error, a method prone to errors. Therefore, providing the desired image to the robot's visual servoing system is also a crucial technical challenge. Summary of the Invention

[0004] This invention provides a visual servoing method, apparatus, computer device, and storage medium based on a diffusion model to solve the technical problem of how to acquire the current desired image and how to provide the current desired image to the visual servoing system of a robot.

[0005] Firstly, a visual servoing method based on a diffusion model is provided, including:

[0006] Acquire images obtained by the camera from the preset tool, select the images obtained by the camera from the preset tool as the preset tool images, and combine the preset tool images, the preset teaching images corresponding to the preset tool images, and the real expected images corresponding to the preset tool images into a training sample;

[0007] The diffusion model is trained based on the training samples, and the feature vectors of the preset tool image and the preset teaching image are fused to obtain the first fused vector;

[0008] The first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image;

[0009] Obtain the total loss value between the predicted expected image and the true expected image;

[0010] When the total loss value is less than the preset value, stop training the diffusion model and obtain the trained diffusion model.

[0011] The system acquires an image of the current tool captured by the camera, selects the image captured by the current tool as the current tool image, fuses the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, processes the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmits the current desired image to the robot's visual servoing system to control the visual servoing system to adjust the position and orientation of the current tool according to the current desired image.

[0012] Furthermore, the step of training a diffusion model based on training samples and fusing the feature vectors of a preset tool image and a preset teaching image to obtain a first fused vector includes:

[0013] The diffusion model is trained based on the training samples, and the preset tool images and the preset teaching images corresponding to the preset tool images in the training samples are input into the variational autoencoder.

[0014] The feature vector of the preset tool image is obtained by extracting features from the preset tool image through a variational autoencoder. The feature vector of the preset teaching image is obtained by extracting features from the preset teaching image through a variational autoencoder. The feature vectors of the preset tool image and the preset teaching image are fused through a mutual attention mechanism to obtain the first fused vector.

[0015] Further, the step of processing the first fusion vector through the decoder in the diffusion model to obtain the predicted desired image includes:

[0016] Obtain the diffusion model and input the first fusion vector into the decoder of the diffusion model;

[0017] The first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image.

[0018] Further, obtaining the total loss value between the predicted expected image and the true expected image includes:

[0019] The first loss value between the predicted expected image and the true expected image is calculated using the KL divergence loss function, and the second loss value between the predicted expected image and the true expected image is calculated using the mean square error loss function.

[0020] The first loss value and the second loss value are added together to obtain the total loss value between the predicted expected image and the true expected image.

[0021] Further, the step of stopping the training of the diffusion model when the total loss value is less than a preset value, and obtaining the trained diffusion model, includes:

[0022] When the total loss value is less than the preset value, stop training the diffusion model and save the model parameters.

[0023] The model parameters are loaded into the model structure, and the model structure with the loaded model parameters is selected as the trained diffusion model.

[0024] Further, the process of acquiring an image of the current tool captured by a camera, selecting the image captured by the current tool as the current tool image, fusing the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmitting the current desired image to the robot's visual servoing system to control the visual servoing system to adjust the position and orientation of the current tool according to the current desired image includes:

[0025] The system acquires an image of the current tool captured by the camera, selects the image captured by the current tool as the current tool image, inputs the current tool image and the corresponding current teaching image into the variational autoencoder, extracts features from the current tool image through the variational autoencoder to obtain the feature vector of the current tool image, and extracts features from the current teaching image through the variational autoencoder to obtain the feature vector of the current teaching image.

[0026] The feature vectors of the current tool image and the current teaching image are fused through a mutual attention mechanism to obtain a second fused vector. The second fused vector is then processed by the decoder in the trained diffusion model to obtain the current desired image. A transmission command is then obtained and executed to transmit the current desired image to the robot's visual servo system. The visual servo system is then controlled to adjust the position and orientation of the current tool according to the current desired image.

[0027] Further, after acquiring an image of the current tool captured by the camera, selecting the image captured by the current tool as the current tool image, fusing the feature vector of the current tool image and the feature vector of the current teaching image to obtain a second fused vector, processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, transmitting the current desired image to the robot's visual servoing system, and controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image, the visual servoing method includes:

[0028] Obtain the adjustment results returned by the robot, create a display window, and display the adjustment results through the display window.

[0029] Secondly, a visual servoing device based on a diffusion model is provided, comprising:

[0030] The first acquisition module is used to acquire images obtained by the camera shooting the preset tool, select the images obtained by shooting the preset tool as the preset tool images, and combine the preset tool images, the preset teaching images corresponding to the preset tool images and the real expected images corresponding to the preset tool images into a training sample;

[0031] The fusion module is used to train a diffusion model based on training samples, and to fuse the feature vectors of the preset tool image and the feature vectors of the preset teaching image to obtain the first fusion vector;

[0032] The processing module is used to process the first fusion vector through the decoder in the diffusion model to obtain the predicted desired image;

[0033] The second acquisition module is used to acquire the total loss value between the predicted expected image and the true expected image;

[0034] The stop module is used to stop training the diffusion model when the total loss value is less than a preset value, thus obtaining the trained diffusion model.

[0035] The control module is used to acquire images of the current tool captured by the camera, select the image captured by the current tool as the current tool image, fuse the feature vector of the current tool image and the feature vector of the current teaching image to obtain a second fused vector, process the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmit the current desired image to the robot's visual servo system to control the visual servo system to adjust the position and orientation of the current tool according to the current desired image.

[0036] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described visual servoing method.

[0037] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described visual servoing method.

[0038] This application provides a visual servoing method, apparatus, computer device, and storage medium based on a diffusion model. The advantages are twofold: First, it acquires an image of the current tool captured by a camera, selects this image as the current tool image, and fuses the feature vectors of the current tool image and the current teaching image to obtain a second fused vector. The decoder in the trained diffusion model processes this second fused vector to obtain the desired current image. This solves the technical problem of how to acquire the desired current image. Since manual acquisition of the desired current image is unnecessary, the acquisition time is reduced, thus improving the acquisition efficiency. Second, it transmits the desired current image to the robot's visual servoing system, controlling the system to adjust the position and orientation of the current tool based on the desired current image. This solves the technical problem of how to provide the desired current image to the robot's visual servoing system. Since the desired current image is automatically transmitted to the robot's visual servoing system, it is not affected by manual intervention, thus improving the reliability of the desired current image. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of an application environment for a visual servoing method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic flowchart of a visual servoing method provided in an embodiment of the present invention;

[0042] Figure 3 yes Figure 2 A flowchart illustrating a specific implementation of step S23;

[0043] Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S25;

[0044] Figure 5 yes Figure 2 A schematic diagram of a specific implementation method for step S26;

[0045] Figure 6 This is a schematic diagram of a visual servo device in one embodiment of the present invention;

[0046] Figure 7This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment for the visual servoing method according to an embodiment of the present invention. The visual servoing method provided in this embodiment of the present invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network.

[0049] The server obtains images of preset tools by the camera through the client, selects the images obtained by the preset tools as preset tool images, and combines the preset tool images, the preset teaching images corresponding to the preset tool images, and the real expected images corresponding to the preset tool images into a training sample;

[0050] The diffusion model is trained based on the training samples, and the feature vectors of the preset tool image and the preset teaching image are fused to obtain the first fused vector;

[0051] The first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image;

[0052] Obtain the total loss value between the predicted expected image and the true expected image;

[0053] When the total loss value is less than the preset value, stop training the diffusion model and obtain the trained diffusion model.

[0054] The system acquires an image of the current tool captured by the camera, selects the image captured by the current tool as the current tool image, fuses the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, processes the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmits the current desired image to the robot's visual servoing system to control the visual servoing system to adjust the position and orientation of the current tool according to the current desired image.

[0055] The beneficial effects of the above-mentioned visual servoing methods, devices, equipment, and media are twofold. Firstly, by acquiring an image of the current tool captured by a camera, selecting the image as the current tool image, and fusing the feature vectors of the current tool image and the current teaching image to obtain a second fused vector, the second fused vector is processed by the decoder in the trained diffusion model to obtain the current desired image. This solves the technical problem of how to acquire the current desired image. Since there is no need for manual acquisition of the current desired image, the acquisition time is reduced, which is beneficial to improving the acquisition efficiency of the current desired image. Secondly, by transmitting the current desired image to the robot's visual servoing system and controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image, this solves the technical problem of how to provide the current desired image to the robot's visual servoing system. Since the current desired image is automatically transmitted to the robot's visual servoing system, it is not affected by manual intervention, which is beneficial to improving the reliability of the current desired image.

[0056] The device running the client is referred to as the client device.

[0057] Among them, the equipment that runs the server is referred to as: server equipment.

[0058] Client devices include, but are not limited to, smartphones, personal computers, vehicle networking terminals, tablets, and portable wearable devices.

[0059] The server-side equipment can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments. Please refer to... Figure 2 , Figure 2 A flowchart illustrating a visual servoing method according to an embodiment of the present invention includes the following steps:

[0060] S21, acquire the image obtained by the camera shooting the preset tool, select the image obtained by shooting the preset tool as the preset tool image, and combine the preset tool image, the preset teaching image corresponding to the preset tool image and the real expected image corresponding to the preset tool image into a training sample;

[0061] Among them, the preset tool image is a preset tool image.

[0062] Among them, the preset teaching image is a preset teaching image.

[0063] Among them, the true expected image is the true expected image.

[0064] The predicted desired image is the desired image predicted by the diffusion model during training.

[0065] Here, the current desired image is the desired image predicted by the trained diffusion model.

[0066] S22, train the diffusion model based on the training samples, and fuse the feature vectors of the preset tool image and the feature vectors of the preset teaching image to obtain the first fused vector;

[0067] The step of training a diffusion model based on training samples, fusing the feature vectors of a preset tool image and a preset teaching image to obtain a first fused vector, includes:

[0068] The diffusion model is trained based on the training samples, and the preset tool images and the preset teaching images corresponding to the preset tool images in the training samples are input into the variational autoencoder.

[0069] The feature vector of the preset tool image is obtained by extracting features from the preset tool image through a variational autoencoder. The feature vector of the preset teaching image is obtained by extracting features from the preset teaching image through a variational autoencoder. The feature vectors of the preset tool image and the preset teaching image are fused through a mutual attention mechanism to obtain the first fused vector.

[0070] The first fusion vector not only contains key information about the feature vectors of the preset tool image and the preset teaching image, but also enhances the interaction between the feature vectors of the preset tool image and the preset teaching image through a mutual attention mechanism, enabling the diffusion model to better understand the preset tool image and the preset teaching image.

[0071] Among them, the variational autoencoder can map the high-dimensional data of the preset tool image and the preset teaching image to a low-dimensional latent space, which is beneficial for processing the high-dimensional data of the preset tool image and the preset teaching image.

[0072] For example, a diffusion model is trained based on training samples, and a preset tool image and a preset teaching image corresponding to the preset tool image are input into a variational autoencoder, including:

[0073] The diffusion model is trained using training samples, and the number of training rounds is obtained during the training process.

[0074] When the number of training rounds is less than the preset number of rounds, the preset tool image and the preset teaching image corresponding to the preset tool image in the training samples are input into the variational autoencoder.

[0075] S23, the first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image;

[0076] S24, obtain the total loss value between the predicted expected image and the true expected image;

[0077] The step of obtaining the total loss value between the predicted expected image and the true expected image includes:

[0078] The first loss value between the predicted expected image and the true expected image is calculated using the KL divergence loss function, and the second loss value between the predicted expected image and the true expected image is calculated using the mean square error loss function.

[0079] The first loss value and the second loss value are added together to obtain the total loss value between the predicted expected image and the true expected image.

[0080] The first loss value and the second loss value are added together to obtain the total loss value between the predicted expected image and the true expected image, including:

[0081] Using a pre-defined total loss value generation model, the first loss value and the second loss value are added together to obtain the total loss value between the predicted expected image and the true expected image;

[0082] The total loss value generation model is as follows: Where L is the total loss value, As the first weighting coefficient, The first loss value, This is the second weighting coefficient. This is the second loss value.

[0083] Among them, the total loss value generation model is the model for generating the total loss value.

[0084] The total loss value between the predicted expected image and the true expected image describes the overall difference between them. The larger the total loss value, the greater the overall difference between them; the smaller the total loss value, the smaller the overall difference.

[0085] S25, when the total loss value is less than the preset value, stop training the diffusion model and obtain the trained diffusion model;

[0086] When the total loss value is less than the preset value, it indicates that the performance of the diffusion model has reached an acceptable level during the training process. At this point, stopping the training of the diffusion model can avoid overtraining, which helps to reduce training time and improve training efficiency.

[0087] S26. Acquire the image of the current tool captured by the camera, select the image captured by the current tool as the current tool image, fuse the feature vector of the current tool image and the feature vector of the current teaching image to obtain a second fused vector, process the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, transmit the current desired image to the robot's visual servo system, and control the visual servo system to adjust the position and orientation of the current tool according to the current desired image.

[0088] The current tool image is the current tool image.

[0089] The current teaching image is the current teaching image.

[0090] The second fusion vector not only contains key information about the feature vectors of the current tool image and the current teaching image, but also enhances the interaction between the feature vectors of the current tool image and the current teaching image through a mutual attention mechanism, enabling the diffusion model to better understand the current tool image and the current teaching image.

[0091] For example, controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image includes:

[0092] The vision servoing system is controlled to process the current desired image to obtain the pose information of the current tool, and the position coordinates and rotation angle in the pose information are read.

[0093] The robot's end effector adjusts the position of the current tool based on the position coordinates, and adjusts the orientation of the current tool based on the rotation angle.

[0094] In this system, the robot's end effector adjusts the position of the current tool based on its location coordinates, and adjusts its orientation based on its rotation angle. This ensures that the tool accurately reaches its position and performs its task with the correct orientation, thereby improving operational precision and reliability. This precise control not only enhances work efficiency and product quality but also reduces errors and failure rates during operation.

[0095] For ease of explanation, let's take an electric screwdriver as an example. Here's an example:

[0096] The robot's end effector adjusts the screwdriver's position based on the position coordinates and the screwdriver's orientation based on the rotation angle. This precise control ensures that the screwdriver can accurately reach the predetermined position and tighten the screw in the correct orientation. This greatly improves the accuracy and reliability of the operation and avoids problems such as loosening or damage to the screw due to position deviation or incorrect orientation.

[0097] For ease of explanation, let's take an electric wrench as an example, as follows:

[0098] The robot's end effector adjusts the position of the electric wrench based on the position coordinates, and adjusts the orientation of the electric wrench based on the rotation angle. This precise control ensures that the electric wrench can accurately reach the predetermined position and tighten the bolt with the correct orientation. This greatly improves the accuracy and reliability of the operation and avoids problems such as bolt loosening or damage caused by position deviation or incorrect orientation.

[0099] The visual servoing method includes the following steps: acquiring an image of the current tool captured by a camera; selecting the image captured by the current tool as the current tool image; fusing the feature vector of the current tool image and the feature vector of the current teaching image to obtain a second fused vector; processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image; transmitting the current desired image to the robot's visual servoing system; and controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image.

[0100] Obtain the adjustment results returned by the robot, create a display window, and display the adjustment results through the display window.

[0101] In this embodiment of the invention, the beneficial effects are twofold. Firstly, by acquiring an image of the current tool captured by a camera, selecting the image captured by the current tool as the current tool image, and fusing the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, and then processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, the technical problem of how to acquire the current desired image is solved. Since there is no need for manual acquisition of the current desired image, the acquisition time of the current desired image is reduced, which is beneficial to improving the acquisition efficiency of the current desired image. Secondly, by transmitting the current desired image to the robot's visual servo system and controlling the visual servo system to adjust the position and orientation of the current tool according to the current desired image, the technical problem of how to provide the current desired image to the robot's visual servo system is solved. Since the current desired image is automatically transmitted to the robot's visual servo system, it is not affected by manual intervention, which is beneficial to improving the reliability of the current desired image.

[0102] Please see Figure 3 , Figure 3 yes Figure 2 A detailed flowchart of a specific implementation method for step S23 is described below:

[0103] S31, Obtain the diffusion model, and input the first fusion vector into the decoder of the diffusion model;

[0104] S32, the first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image.

[0105] The predicted expected image is the desired image predicted by the diffusion model during training. The diffusion model is in the training phase, extracting features and learning patterns from training samples. The predicted expected image can be used to evaluate the performance of the diffusion model during training.

[0106] In this embodiment of the invention, the predicted expected image can be used to evaluate the performance of the diffusion model during the training process. By analyzing the predicted expected image, the extent to which the diffusion model understands the input data can be understood, and the training strategy can be adjusted to improve the performance and generalization ability of the diffusion model.

[0107] Please see Figure 4 , Figure 4 yes Figure 2 A detailed flowchart of a specific implementation method for step S25 is described below:

[0108] S41, when the total loss value is less than the preset value, stop training the diffusion model and save the model parameters;

[0109] When the total loss value is less than the preset value, it means that the model parameters have been optimized to the best state, so the model parameters are saved.

[0110] S42, load the model parameters into the model structure, and select the model structure with the loaded model parameters as the trained diffusion model.

[0111] In this embodiment of the invention, model parameters are loaded into the model structure, and the model structure with loaded model parameters is selected as the trained diffusion model, ensuring that the trained diffusion model has high accuracy and generalization performance.

[0112] Please see Figure 5 , Figure 5 yes Figure 2 A detailed flowchart of a specific implementation method for step S26 is described below:

[0113] S51, acquire the image of the current tool captured by the camera, select the image captured by the current tool as the current tool image, input the current tool image and the current teaching image corresponding to the current tool image into the variational autoencoder, extract features from the current tool image through the variational autoencoder to obtain the feature vector of the current tool image, extract features from the current teaching image through the variational autoencoder to obtain the feature vector of the current teaching image;

[0114] Among them, the variational autoencoder can map the high-dimensional data of the current tool image and the current teaching image to a low-dimensional latent space, which is beneficial for processing the high-dimensional data of the current tool image and the current teaching image.

[0115] S52, through the mutual attention mechanism, the feature vectors of the current tool image and the feature vectors of the current teaching image are fused to obtain a second fused vector. The second fused vector is processed by the decoder in the trained diffusion model to obtain the current desired image. The transmission command is obtained and executed to transmit the current desired image to the robot's visual servo system. The visual servo system is controlled to adjust the position and orientation of the current tool according to the current desired image.

[0116] For example, controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image includes:

[0117] The vision servoing system is controlled to process the current desired image to obtain the pose information of the current tool, and the position coordinates and rotation angle in the pose information are read.

[0118] The robot's end effector adjusts the position of the current tool based on the position coordinates, and adjusts the orientation of the current tool based on the rotation angle.

[0119] In this embodiment of the invention, the feature vectors of the current tool image and the current teaching image are fused through a mutual attention mechanism to obtain a second fused vector. The second fused vector is then processed by the decoder in the trained diffusion model to obtain the current desired image. Since there is no need to manually acquire the current desired image, the acquisition time of the current desired image is reduced, which is beneficial to improving the acquisition efficiency of the current desired image.

[0120] Please see Figure 6 , Figure 6 This is a schematic diagram of a visual servo device according to an embodiment of the present invention, as shown below. Figure 6 As shown, the visual servoing device includes a first acquisition module 101, a fusion module 102, a processing module 103, a second acquisition module 104, a stop module 105, and a control module 106. Detailed descriptions of each functional module are as follows:

[0121] The first acquisition module 101 is used to acquire an image obtained by the camera shooting a preset tool, select the image obtained by shooting the preset tool as the preset tool image, and combine the preset tool image, the preset teaching image corresponding to the preset tool image and the real expected image corresponding to the preset tool image into a training sample;

[0122] The fusion module 102 is used to train a diffusion model based on training samples, and fuse the feature vectors of the preset tool image and the feature vectors of the preset teaching image to obtain a first fusion vector;

[0123] Processing module 103 is used to process the first fusion vector through the decoder in the diffusion model to obtain the predicted desired image;

[0124] The second acquisition module 104 is used to acquire the total loss value between the predicted expected image and the true expected image;

[0125] Stop module 105 is used to stop training the diffusion model when the total loss value is less than a preset value, and obtain the trained diffusion model.

[0126] The control module 106 is used to acquire the image of the current tool captured by the camera, select the image captured by the current tool as the current tool image, fuse the feature vector of the current tool image and the feature vector of the current teaching image to obtain a second fused vector, process the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, transmit the current desired image to the robot's visual servo system, and control the visual servo system to adjust the position and orientation of the current tool according to the current desired image.

[0127] In this embodiment of the invention, the beneficial effects are twofold. Firstly, by acquiring an image of the current tool captured by a camera, selecting the image captured by the current tool as the current tool image, and fusing the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, and then processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, the technical problem of how to acquire the current desired image is solved. Since there is no need for manual acquisition of the current desired image, the acquisition time of the current desired image is reduced, which is beneficial to improving the acquisition efficiency of the current desired image. Secondly, by transmitting the current desired image to the robot's visual servo system and controlling the visual servo system to adjust the position and orientation of the current tool according to the current desired image, the technical problem of how to provide the current desired image to the robot's visual servo system is solved. Since the current desired image is automatically transmitted to the robot's visual servo system, it is not affected by manual intervention, which is beneficial to improving the reliability of the current desired image.

[0128] For specific limitations on visual servoing devices, please refer to the limitations on visual servoing methods mentioned above, which will not be repeated here.

[0129] Each module in the aforementioned visual servoing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0130] Please see Figure 7 , Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. In one embodiment, a computer device is provided, which is a server device or a client device, and its internal structure diagram can be as follows. Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external devices. When executed by the processor, the computer program can implement the functions or steps of a diffusion-based visual servoing method.

[0131] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0132] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions of the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0133] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), and network processors (NPs); they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0134] The foregoing description and accompanying drawings fully illustrate embodiments of the present disclosure to enable those skilled in the art to practice them. Throughout this document, each embodiment is emphasized to highlight its differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods or products disclosed in the embodiments, if they correspond to the method section of the disclosed embodiments, the relevant details can be found in the description of the method section.

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A visual servoing method based on a diffusion model, characterized in that, include: Acquire images obtained by the camera from the preset tool, select the images obtained by the camera from the preset tool as the preset tool images, and combine the preset tool images, the preset teaching images corresponding to the preset tool images, and the real expected images corresponding to the preset tool images into a training sample; The diffusion model is trained based on the training samples, and the feature vectors of the preset tool image and the preset teaching image are fused to obtain the first fused vector; The first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image; Obtain the total loss value between the predicted expected image and the true expected image; When the total loss value is less than the preset value, stop training the diffusion model and obtain the trained diffusion model. The system acquires an image of the current tool captured by the camera, selects the image captured by the current tool as the current tool image, fuses the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, processes the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmits the current desired image to the robot's visual servoing system to control the visual servoing system to adjust the position and orientation of the current tool according to the current desired image.

2. The visual servoing method according to claim 1, characterized in that, The diffusion model trained based on training samples fuses the feature vectors of the preset tool image and the feature vectors of the preset teaching image to obtain a first fused vector, including: The diffusion model is trained based on the training samples, and the preset tool images and the preset teaching images corresponding to the preset tool images in the training samples are input into the variational autoencoder. The feature vector of the preset tool image is obtained by extracting features from the preset tool image through a variational autoencoder. The feature vector of the preset teaching image is obtained by extracting features from the preset teaching image through a variational autoencoder. The feature vectors of the preset tool image and the preset teaching image are fused through a mutual attention mechanism to obtain the first fused vector.

3. The visual servoing method according to claim 1, characterized in that, The step of processing the first fusion vector through the decoder in the diffusion model to obtain the predicted desired image includes: Obtain the diffusion model and input the first fusion vector into the decoder of the diffusion model; The first fusion vector is processed by the decoder in the diffusion model to obtain the predicted desired image.

4. The visual servoing method according to claim 1, characterized in that, The process of obtaining the total loss value between the predicted expected image and the true expected image includes: The first loss value between the predicted expected image and the true expected image is calculated using the KL divergence loss function, and the second loss value between the predicted expected image and the true expected image is calculated using the mean square error loss function. The first loss value and the second loss value are added together to obtain the total loss value between the predicted expected image and the true expected image.

5. The visual servoing method according to claim 1, characterized in that, The step of stopping training the diffusion model when the total loss value is less than a preset value, and obtaining the trained diffusion model, includes: When the total loss value is less than the preset value, stop training the diffusion model and save the model parameters. The model parameters are loaded into the model structure, and the model structure with the loaded model parameters is selected as the trained diffusion model.

6. The visual servoing method according to claim 1, characterized in that, The process of acquiring an image of the current tool captured by a camera, selecting the image captured by the current tool as the current tool image, fusing the feature vector of the current tool image with the feature vector of the current teaching image to obtain a second fused vector, processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmitting the current desired image to the robot's visual servoing system to control the visual servoing system to adjust the position and orientation of the current tool according to the current desired image includes: The system acquires an image of the current tool captured by the camera, selects the image captured by the current tool as the current tool image, inputs the current tool image and the corresponding current teaching image into the variational autoencoder, extracts features from the current tool image through the variational autoencoder to obtain the feature vector of the current tool image, and extracts features from the current teaching image through the variational autoencoder to obtain the feature vector of the current teaching image. The feature vectors of the current tool image and the current teaching image are fused through a mutual attention mechanism to obtain a second fused vector. The second fused vector is then processed by the decoder in the trained diffusion model to obtain the current desired image. A transmission command is then obtained and executed to transmit the current desired image to the robot's visual servo system. The visual servo system is then controlled to adjust the position and orientation of the current tool according to the current desired image.

7. The visual servoing method according to claim 1, characterized in that, The visual servoing method, after acquiring an image of the current tool captured by a camera, selecting the image as the current tool image, fusing the feature vectors of the current tool image and the current teaching image to obtain a second fused vector, processing the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmitting the current desired image to the robot's visual servoing system, and controlling the visual servoing system to adjust the position and orientation of the current tool according to the current desired image, includes: Obtain the adjustment results returned by the robot, create a display window, and display the adjustment results through the display window.

8. A visual servoing device based on a diffusion model, characterized in that, include: The first acquisition module is used to acquire images obtained by the camera shooting the preset tool, select the images obtained by shooting the preset tool as the preset tool images, and combine the preset tool images, the preset teaching images corresponding to the preset tool images and the real expected images corresponding to the preset tool images into a training sample; The fusion module is used to train a diffusion model based on training samples, and to fuse the feature vectors of the preset tool image and the feature vectors of the preset teaching image to obtain the first fusion vector; The processing module is used to process the first fusion vector through the decoder in the diffusion model to obtain the predicted desired image; The second acquisition module is used to acquire the total loss value between the predicted expected image and the true expected image; The stop module is used to stop training the diffusion model when the total loss value is less than a preset value, thus obtaining the trained diffusion model. The control module is used to acquire images of the current tool captured by the camera, select the image captured by the current tool as the current tool image, fuse the feature vector of the current tool image and the feature vector of the current teaching image to obtain a second fused vector, process the second fused vector through the decoder in the trained diffusion model to obtain the current desired image, and transmit the current desired image to the robot's visual servo system to control the visual servo system to adjust the position and orientation of the current tool according to the current desired image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the visual servoing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the visual servoing method as described in any one of claims 1 to 7.

Images