Image processing method and device

By combining color and depth images for information complementary image processing methods, the problem of low data acquisition and model training efficiency in robot learning is solved, and an image processing model with efficient training in complex environments is realized to adapt to the prediction of unfamiliar scenes .

CN113808061BActive Publication Date: 2024-11-22SHENZHEN SENSETIME TECH CO LTD
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
CN202111111394.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-04-28
Publication Date
2024-11-22
Estimated Expiration
2039-04-28

AI Technical Summary

Technical Problem

In the data acquisition and model training process, existing robot learning methods have problems such as large amount of data, long time, trial and error security risks, and the inability to make appropriate predictions for unfamiliar scenarios, especially in real environments, it is difficult to collect data efficiently and quickly. Training network models with good results.

Method used

Color images and depth images are used as training samples, information complementarity is performed through image processing models, and color and depth image processing models are trained step by step, and color image processing models are trained using depth image supervision, combining data augmentation processing to improve training efficiency and accuracy.

Benefits of technology

A color image processing model with good results is achieved with fewer training samples, which reduces training difficulty and time, improves the training efficiency and control accuracy of the model, and adapts to scene prediction in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113808061B_ABST
    Figure CN113808061B_ABST
Patent Text Reader

Abstract

An embodiment of the present application discloses an image processing method and device, the method comprising: obtaining a training sample set, the training sample set comprising at least one color sample image and at least one depth sample image corresponding to the color sample image; inputting the training sample set into an image processing model for processing; inputting the color sample image into a first encoding module of the color image processing model, and outputting an encoded first color sample image; inputting the depth sample image into a second encoding module of the depth image processing model, and outputting an encoded first depth sample image; using the encoded first color sample image and the encoded first depth sample image, calculating a second loss parameter; using the second loss parameter as a first loss parameter; adjusting the color image processing model based on the first loss parameter to obtain a trained color image processing model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of Chinese patent application No. 201910351289.4, filed on April 28, 2019, and entitled “An Image Processing Method and Device”. Technical Field

[0002] The present application relates to robotics technology, and in particular to an image processing method and device. Background Art

[0003] The research on robot learning is mainly divided into two directions: supervised learning and reinforcement learning. Among them, supervised learning has sub-directions such as imitation learning and self-supervised learning. The imitation learning method trains the network model by collecting expert teaching information. The disadvantage is that it requires a lot of expert teaching information, and it usually assumes that the environment is known and limited, which is not effective for open and complex scenes in reality; self-supervised learning collects labeled data through robot trial and error experiments. The disadvantages are that the success rate of trial and error experiments is low, data collection is very inefficient, and trial and error in real environments will pose safety risks. The reinforcement learning method learns to complete the specified task through a large amount of robot exploration. The disadvantage is that it requires a lot of exploration times, which is almost impossible to complete in reality.

[0004] In the existing robot learning process, color images are directly used as network model inputs, and a generative adversarial network (GAN) model is used to learn an encoder that can well encode the input RGB image. The output of the encoder is then input into the long short-term memory network (LSTM) model to let the network learn to complete a sequence of actions. The disadvantage of this method is that the amount of data required is still very large, with a total of about 2,500 expert teaching examples collected. This method is very time-consuming to collect data, and physical wear and tear of the robot may occur. In addition, the scenes and actions of expert teaching are limited, and the robot cannot make appropriate predictions for unfamiliar scenes. Therefore, how to collect data efficiently and quickly, and how to train a good network model with a small amount of data, are technical problems that need to be solved in robotics technology. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application hope to provide an image processing method and device.

[0006] The technical solution of this application is implemented as follows:

[0007] In a first aspect, an image processing method is provided, the method comprising: acquiring a training sample set, the training sample set comprising at least one color sample image and at least one depth sample image corresponding to the color sample image;

[0008] Inputting the training sample set into an image processing model for processing, the image processing model comprising a color image processing model and a depth image processing model, the color image processing model being used to process the color sample image to obtain a color reconstructed image, and the depth image processing model being used to process the depth image to obtain a depth reconstructed image;

[0009] A first loss parameter of the image processing model is determined based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, and the color image processing model is adjusted based on the first loss parameter to obtain a trained color image processing model.

[0010] In this way, using color images and depth images as training samples at the same time has the effect of information complementarity, and a good color image processing model can be trained with fewer training samples. The step-by-step training method can reduce the difficulty of training and improve the training efficiency.

[0011] In the above scheme, the method also includes: adjusting the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.

[0012] In this way, during the image processing model training process, the color image processing model and the depth image processing model can be trained at the same time. The trained depth image processing model is used to supervise the training of the color image processing model, which can improve the training effects of both at the same time.

[0013] In the above scheme, the color image processing model includes a first encoding module, the depth image processing model includes a second encoding module, and the first loss parameter of the image processing model is determined based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, including: inputting the color sample image into the first encoding module of the color image processing model, and outputting the encoded first color sample image; inputting the depth sample image into the second encoding module of the color image processing model, and outputting the encoded first depth sample image; and obtaining the first loss parameter based on the encoded first color sample image and the encoded first depth sample image.

[0014] In the above scheme, the color image processing model also includes a first decoding module, and the depth image processing model also includes a second decoding module. The first loss parameter is obtained based on the encoded first color sample image and the encoded first depth sample image, including: inputting the encoded first color sample image into the first decoding module, and outputting the color reconstructed image; inputting the encoded first depth sample image into the second decoding module, and outputting the depth reconstructed image; using the encoded first color sample image and the encoded first depth sample image, calculating the second loss parameter; determining the third loss parameter based on the encoded first color sample image and the color reconstructed image; determining the fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determining the first loss parameter based on the second loss parameter, the third loss parameter and the fourth loss parameter.

[0015] In this way, using the second loss function obtained by the encoded first color sample image and the encoded first depth sample image as a loss parameter of the image processing model can ensure that the encoded color image and depth image obtained in the image processing model are as similar as possible, thereby improving the training effect.

[0016] In the above scheme, the training sample set also includes: at least one robot state label corresponding to the color sample image;

[0017] The method further includes: inputting the color sample image into a first encoding module of the trained color image processing model, and outputting an encoded second color sample image; determining a fifth loss parameter of a control model based on the encoded second color sample image and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain a trained control model;

[0018] Alternatively, the method also includes: inputting the color sample image into a first encoding module of the trained color image processing model, and outputting an encoded second color sample image; inputting the depth sample image into a second encoding module of the trained depth image processing model, and outputting an encoded second depth sample image; determining a fifth loss parameter of the control model based on the encoded second color sample image and the encoded second depth sample image, as well as a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain a trained control model.

[0019] In this way, after the control model is trained to obtain the predicted state information, the state label is used to verify whether the predicted state information is accurate. By making the predicted state information the same or similar to the state label, the training effect of the control model can be improved.

[0020] In the above scheme, the state label includes at least one of the following: joint angle, joint linear velocity and joint angular velocity.

[0021] In the above scheme, the method also includes: when controlling the robot to perform a target task on a target object, obtaining a target color image containing the target object; inputting the target color image into the first encoding module of the trained color image processing model, and outputting the encoded target color image; inputting the encoded target color image into the trained control model to obtain the result status information of the robot, and controlling the robot based on the result status information.

[0022] In this way, the image recognition ability of the trained color image processing model can be used to identify the scene information in the color image, and then the control ability of the trained control model can be used to determine the control operations that can be performed on the current scene information. The step-by-step operation reduces the control difficulty.

[0023] In the above scheme, the method also includes: when controlling the robot to perform a target task on a target object, obtaining a target color image and a target depth image containing the target object; inputting the target color image into the first encoding module of the trained color image processing model, and outputting the encoded target color image; inputting the target depth image into the second encoding module of the trained depth image processing model, and outputting the encoded target depth image; inputting the encoded target color image and the encoded target depth image into the trained control model to obtain the result status information of the robot, and controlling the robot based on the result status information.

[0024] In this way, the image recognition capabilities of the trained color image processing model and the trained depth image processing model can be utilized to obtain more comprehensive scene information. Using this information, the trained control model can obtain more accurate control information and improve control accuracy.

[0025] In the above solution, the obtaining of the training sample set includes: performing data enhancement processing on the collected color sample images and the collected depth sample images to increase the number of samples, thereby obtaining the training sample set.

[0026] In this way, data enhancement processing can not only increase the amount of data, but also alleviate network overfitting and improve model training efficiency.

[0027] In a second aspect, an embodiment of the present application further provides an image processing device, the device comprising:

[0028] An acquisition unit, configured to acquire a training sample set, wherein the training sample set includes at least one color sample image and at least one depth sample image corresponding to the color sample image;

[0029] a processing unit, configured to input the training sample set into an image processing model for processing, wherein the image processing model includes a color image processing model and a depth image processing model, wherein the color image processing model is configured to process the color sample image to obtain a color reconstructed image, and the depth image processing model is configured to process the depth image to obtain a depth reconstructed image;

[0030] The processing unit is also used to determine a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, and adjust the color image processing model based on the first loss parameter to obtain a trained color image processing model.

[0031] In this way, using color images and depth images as training samples at the same time has the effect of information complementarity, and a good color image processing model can be trained with fewer training samples. The step-by-step training method can reduce the difficulty of training and improve the training efficiency.

[0032] In the above solution, the processing unit is also used to adjust the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.

[0033] In this way, during the image processing model training process, the color image processing model and the depth image processing model can be trained at the same time. The trained depth image processing model is used to supervise the training of the color image processing model, which can improve the training effects of both at the same time.

[0034] In the above scheme, the color image processing model includes a first encoding module, the depth image processing model includes a second encoding module, and the processing unit is specifically used to input the color sample image into the first encoding module of the color image processing model, and output the encoded first color sample image; input the depth sample image into the second encoding module of the color image processing model, and output the encoded first depth sample image; and obtain a first loss parameter based on the encoded first color sample image and the encoded first depth sample image.

[0035] In the above scheme, the color image processing model also includes a first decoding module, and the depth image processing model also includes a second decoding module. The processing unit is specifically used to input the encoded first color sample image into the first decoding module and output the color reconstructed image; input the encoded first depth sample image into the second decoding module and output the depth reconstructed image; use the encoded first color sample image and the encoded first depth sample image to calculate the second loss parameter; determine the third loss parameter based on the encoded first color sample image and the color reconstructed image; determine the fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determine the first loss parameter based on the second loss parameter, the third loss parameter and the fourth loss parameter.

[0036] In this way, using the second loss function obtained by the encoded first color sample image and the encoded first depth sample image as a loss parameter of the image processing model can ensure that the encoded color image and depth image obtained in the image processing model are as similar as possible, thereby improving the training effect.

[0037] In the above scheme, the training sample set also includes: at least one robot state label corresponding to the color sample image;

[0038] The processing unit is further configured to input the color sample image into the first encoding module of the trained color image processing model, and output an encoded second color sample image; determine a fifth loss parameter of the control model based on the encoded second color sample image and the state label of the robot; and adjust the control model based on the fifth loss parameter to obtain a trained control model;

[0039] Alternatively, the processing unit is further used to input the color sample image into the first encoding module of the trained color image processing model, and output the encoded second color sample image; input the depth sample image into the second encoding module of the trained depth image processing model, and output the encoded second depth sample image; determine the fifth loss parameter of the control model based on the encoded second color sample image and the encoded second depth sample image, and the state label of the robot; and adjust the control model based on the fifth loss parameter to obtain the trained control model.

[0040] In this way, after the control model is trained to obtain the predicted state information, the state label is used to verify whether the predicted state information is accurate. By making the predicted state information the same or similar to the state label, the training effect of the control model can be improved.

[0041] In the above scheme, the state label includes at least one of the following: joint angle, joint linear velocity and joint angular velocity.

[0042] In the above scheme, the acquisition unit is also used to control the robot to perform a target task on the target object, and to acquire a target color image containing the target object; the processing unit is also used to input the target color image into the first encoding module of the trained color image processing model, and output the encoded target color image; the encoded target color image is input into the trained control model to obtain the result status information of the robot, and the robot is controlled based on the result status information.

[0043] In this way, the image recognition ability of the trained color image processing model can be used to identify the scene information in the color image, and then the control ability of the trained control model can be used to determine the control operations that can be performed on the current scene information. The step-by-step operation reduces the control difficulty.

[0044] In the above solution, the acquisition unit is further used to control the robot to perform a target task on the target object, thereby acquiring a target color image and a target depth image containing the target object;

[0045] The processing unit is further used to input the target color image into the first encoding module of the trained color image processing model, and output the encoded target color image; input the target depth image into the second encoding module of the trained depth image processing model, and output the encoded target depth image; input the encoded target color image and the encoded target depth image into the trained control model to obtain the result status information of the robot, and control the robot based on the result status information.

[0046] In this way, the image recognition capabilities of the trained color image processing model and the trained depth image processing model can be utilized to obtain more comprehensive scene information. Using this information, the trained control model can obtain more accurate control information and improve control accuracy.

[0047] In the above solution, the acquisition unit is specifically used to perform data enhancement processing on the collected color sample images and the collected depth sample images to increase the number of samples and obtain the training sample set.

[0048] In this way, data enhancement processing can not only increase the amount of data, but also alleviate network overfitting and improve model training efficiency.

[0049] In a third aspect, an embodiment of the present application also provides another image processing device, which includes a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of any of the aforementioned methods when running the computer program.

[0050] In a fourth aspect, an embodiment of the present application further provides a computer storage medium on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0051] By adopting the above technical solution, the color sample image and the depth sample image are input into the image processing model, and the color image processing model is adjusted by using the first loss parameter of the image processing model to obtain the trained color image processing model. In this way, using the color sample image and the depth sample image as training samples at the same time has the effect of information complementarity, and a color image processing model with good effect can be trained with fewer training samples, thereby reducing the difficulty of training and improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 1 ;

[0053] Figure 2 This is a schematic diagram of the structure of the data acquisition system in the embodiment of the present application;

[0054] Figure 3 Schematic diagram of the composition structure of the image processing model in the embodiment of the present application;

[0055] Figure 4 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 2 ;

[0056] Figure 5 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 3 ;

[0057] Figure 6 The process diagram of the image processing method in the embodiment of the present application is as follows: Figure 4 ;

[0058] Figure 7 This is a schematic diagram of the structure of the robot control system in the embodiment of the present application;

[0059] Figure 8 The structure of the image processing device in the embodiment of the present application is shown in FIG. Figure 1 ;

[0060] Fig. 9 The structure of the image processing device in the embodiment of the present application is shown in FIG. Figure 2 . DETAILED DESCRIPTION

[0061] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0062] Embodiment 1

[0063] like Figure 1 As shown, the image processing method may specifically include the following:

[0064] Step 101: Acquire a training sample set, wherein the training sample set includes at least one color sample image and at least one depth sample image corresponding to the color sample image;

[0065] Step 102: inputting the training sample set into an image processing model for processing, wherein the image processing model includes a color image processing model and a depth image processing model, wherein the color image processing model is used to process the color sample image to obtain a color reconstructed image, and the depth image processing model is used to process the depth image to obtain a depth reconstructed image;

[0066] Step 103: Determine a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, and adjust the color image processing model based on the first loss parameter to obtain a trained color image processing model.

[0067] Here, the execution subject of step 101 to step 103 may be a processor of the image processing device.

[0068] In practical applications, when controlling a robot for learning, it is necessary to use sample data to train at least one network model of the robot. At this time, the trainer controls the robot to perform at least one historical target task on the target object and collects sample data. Here, the sample data includes a color sample image and its corresponding depth sample image collected by the image acquisition device, and a training sample set is established by these color sample images and depth sample images.

[0069] For example, the image acquisition device may include a camera for acquiring color images and a camera for acquiring depth images. For example, a Kinect is used to acquire a color image and a depth image of a target object. Here, the color image may include information such as the color and texture of the target object, and the pixel value of each pixel in the depth image represents the distance from the sensor to the physical object.

[0070] like Figure 2As shown, the trainer controls the robot to complete a task through the controller. During this process, the data acquisition system records the color image and depth image of the camera and the status information of the robot at each moment through the Robot Operating System (ROS). The information that the data acquisition system can collect includes: color images and depth images of different models of cameras (such as Kinect V2), with adjustable image resolution; status information of different models of robots (such as UR robots). Here, the status information may include joint angles, joint linear velocities, joint angular velocities, robot load, end effector position and posture, etc.

[0071] In practical applications, the step of obtaining the training sample set includes: performing data enhancement processing on the collected color sample images and the collected depth sample images to increase the number of samples, thereby obtaining the training sample set.

[0072] Here, in order to increase the amount of data and alleviate network overfitting, the collected color sample images and depth sample images are enhanced to obtain more training samples. Multiple images can be obtained based on one image, and the number of samples can be expanded to improve the efficiency of model training. The enhancement processing includes: random cropping, horizontal flipping, resizing, hue adjustment, brightness adjustment, saturation adjustment and other preprocessing of the given sample image.

[0073] In practical applications, the color image processing model includes a first encoding module, and the depth image processing model includes a second encoding module. The first loss parameter of the image processing model is determined based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, including: inputting the color sample image into the first encoding module of the color image processing model, and outputting the encoded first color sample image; inputting the depth sample image into the second encoding module of the color image processing model, and outputting the encoded first depth sample image; and obtaining the first loss parameter based on the encoded first color sample image and the encoded first depth sample image.

[0074] Here, the first loss parameter is used to measure the training effect of the image processing model. After the training sample set is input into the image processing model, the encoded color image and the encoded depth image will be obtained. The encoded color image and the encoded depth image should be as similar as possible to obtain a color image processing model with good effect.

[0075] Here, the color sample image input into the color image processing model and the depth sample image input into the depth image processing model are image information collected for the same scene at the same time. The color sample image and the depth sample image at the same time carry more comprehensive scene information. The color sample image and the depth sample image have an information complement effect. The feature map obtained after encoding the color sample image is as similar as possible to the feature map obtained after encoding the depth sample image, so that the scene information can be better learned, and a good image processing model can be trained with less data. In the training process, the embodiment of the present application can use the depth image processing model to supervise the training of the color image processing model, thereby improving the training efficiency and training effect of the color image processing model.

[0076] In some embodiments, the color image processing model also includes a first decoding module, and the depth image processing model also includes a second decoding module. The first loss parameter is obtained based on the encoded first color sample image and the encoded first depth sample image, including: inputting the encoded first color sample image into the first decoding module, and outputting the color reconstructed image; inputting the encoded first depth sample image into the second decoding module, and outputting the depth reconstructed image; using the encoded first color sample image and the encoded first depth sample image, calculating the second loss parameter; determining the third loss parameter based on the encoded first color sample image and the color reconstructed image; determining the fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determining the first loss parameter based on the second loss parameter, the third loss parameter and the fourth loss parameter.

[0077] Exemplarily, the sum of the second loss parameter, the third loss parameter and the fourth loss parameter is calculated, and the sum result is used as the first loss parameter.

[0078] like Figure 3As shown, the image processing model includes a color image processing model 31 and a depth image processing model 32, wherein the color image processing model 31 includes a first encoding module 311 and a first decoding module 312, and the depth image processing model 32 includes a second encoding module 321 and a second decoding module 322; the color sample image is sequentially passed through the first encoding module and the first decoding module to obtain a color reconstructed image, and the depth sample image is sequentially passed through the second encoding module and the second decoding module to obtain a depth reconstructed image, and the encoded color image output by the first encoding module 311 and the encoded depth image output by the second encoding module 321 are compared to obtain a second loss parameter (loss2), the color sample image and the color reconstructed image are compared to obtain a third loss parameter (loss3), and the depth sample image and the depth reconstructed image are compared to obtain a fourth loss parameter (loss4), and loss2, loss3 and loss4 are summed to obtain the first loss parameter (loss1).

[0079] Here, the parameters of the color image processing model and the depth image processing model in the image processing model are randomly initialized, and the initialized image processing model is trained.

[0080] In some other embodiments, obtaining the first loss parameter based on the encoded first color sample image and the encoded first depth sample image includes: calculating a second loss parameter using the encoded first color sample image and the encoded first depth sample image; and using the second loss parameter as the first loss parameter;

[0081] Alternatively, the first loss parameter is determined based on the second loss parameter and the third loss parameter;

[0082] Alternatively, the first loss parameter is determined based on the second loss parameter and the fourth loss parameter.

[0083] In practical applications, the color image processing model is adjusted based on the first loss parameter to obtain a trained color image processing model, including: when the first loss parameter is greater than the first target loss parameter, the color image processing model is adjusted to obtain a trained color image processing model.

[0084] That is to say, when training the color image processing model, the first target loss parameter is used as the basis for the completion of the color image processing model training. When the first loss parameter is greater than the first target loss parameter, it indicates that the current color image processing model has not achieved the expected effect, and the training has failed. It is necessary to continue to adjust the model parameters. After adjusting the model parameters, a new first loss parameter is obtained; when the first loss parameter is less than or equal to the first target loss parameter, the model training is determined to be completed and the model parameters are fixed.

[0085] In some embodiments, the method further includes: adjusting the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.

[0086] That is to say, the color image processing model and the depth image processing model will be adjusted simultaneously based on the first loss parameter. Similarly, the first target loss parameter is used as the basis for the completion of the training of the depth image processing model. When the first loss parameter is greater than the first target loss parameter, it indicates that the current depth image processing model has not achieved the expected effect, and the training is determined to have failed. The model parameters need to be adjusted further, and a new first loss parameter is obtained after adjusting the model parameters; when the first loss parameter is less than or equal to the first target loss parameter, the model training is determined to be completed and the model parameters are fixed.

[0087] Here, methods for adjusting the color image processing model and the depth image processing model may include gradient descent, stochastic gradient descent, and mini-batch.

[0088] In practical applications, after obtaining the trained color image processing model, the method also includes training the control model. The process of training the control model can be regarded as a robot testing process to detect whether the trained image processing model obtained in the learning process can enable the robot to perform correct control operations. Only when both learning and testing achieve the expected results can the robot be put into production life.

[0089] In some embodiments, the training sample set also includes: at least one robot status label corresponding to the color sample image; the method for training the control model includes: inputting the color sample image into the first encoding module of the trained color image processing model, and outputting the encoded second color sample image; based on the encoded second color sample image and the status label of the robot, determining the fifth loss parameter of the control model; adjusting the control model based on the fifth loss parameter to obtain the trained control model.

[0090] Here, the fifth loss parameter is used to characterize the training effect of the control model. After obtaining the trained color image processing model, the robot needs to be tested. The control model obtains the predicted state information based on the encoded second color sample image, and judges whether the preset state information is accurate based on the state label corresponding to the color sample image. Only when the predicted state information is the same or similar to the state label can a good control model be obtained.

[0091] In other words, the fifth loss parameter is used to characterize the error between the predicted state information and the state label, and the error between the predicted state information and the state label can be: joint angle error, robot speed error, end effector position error or other state information error. The second target loss parameter can be an error threshold, and the error threshold is specifically set for different state information, for example: the joint angle error corresponds to an error threshold of plus or minus 2°, the robot speed error corresponds to an error threshold of 5 meters / minute, and the end effector position error corresponds to an error threshold of 2 mm.

[0092] The error may specifically be the mean square error between the predicted state information and the state label, and the calculation formula of the mean square error is used as the loss function of the control model.

[0093] Furthermore, the fifth loss parameter of the control model is determined based on the encoded second color sample image and the state label of the robot, including: inputting the encoded second color sample image into the control model to obtain the predicted state information of the robot; and determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.

[0094] In other embodiments, the method for training a control model includes: inputting the color sample image into a first encoding module of the trained color image processing model, and outputting an encoded second color sample image; inputting the depth sample image into a second encoding module of the trained depth image processing model, and outputting an encoded second depth sample image; determining a fifth loss parameter of the control model based on the encoded second color sample image and the encoded second depth sample image, as well as a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain a trained control model.

[0095] Furthermore, the fifth loss parameter of the control model is determined based on the encoded second color sample image and the encoded second depth sample image, as well as the state label of the robot, including: inputting the encoded second color sample image and the encoded second depth sample image into the control model to obtain the predicted state information of the robot; and determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.

[0096] In other embodiments, the method for training a control model includes: inputting the depth sample image into a second encoding module of the trained depth image processing model, and outputting an encoded second depth sample image; determining a fifth loss parameter of the control model based on the encoded second depth sample image and a state label of the robot; and adjusting the control model based on the fifth loss parameter to obtain a trained control model.

[0097] Furthermore, the fifth loss parameter of the control model is determined based on the encoded second depth sample image and the state label of the robot, including: inputting the encoded second depth sample image into the control model to obtain the predicted state information of the robot; and determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.

[0098] That is to say, when training the control model, the control model can be trained using only the encoded second color sample image, or the control model can be trained using the encoded second color sample image and the encoded depth sample image, or the control model can be trained using only the encoded second depth sample image.

[0099] The control model is adjusted based on the fifth loss parameter to obtain a trained control model, including: when the fifth loss parameter is greater than the second target loss parameter, adjusting the control model so that the fifth loss parameter corresponding to the trained control model is not greater than the second target loss parameter.

[0100] That is to say, when training the control model, the second target loss parameter is used as the basis for the completion of the control model training. When the fifth loss parameter is greater than the second target loss parameter, it indicates that the current control model has not achieved the expected effect, and the training has failed. It is necessary to continue to adjust the model parameters. After adjusting the model parameters, a new fifth loss parameter is obtained; when the fifth loss parameter is less than or equal to the second target loss parameter, the model training is determined to be completed and the model parameters are fixed.

[0101] In practical applications, the state label includes at least one of the following: joint angle, joint linear velocity and joint angular velocity.

[0102] Accordingly, the predicted state information includes at least one of the following: joint angle, joint linear velocity and joint angular velocity. The result state information includes at least one of the following: joint angle, joint linear velocity and joint angular velocity.

[0103] Here, methods for adjusting the control model can include gradient descent, stochastic gradient descent, and mini-batch.

[0104] By adopting the above technical solution, the color sample image and the depth sample image are input into the image processing model, and the color image processing model is adjusted by using the first loss parameter of the image processing model to obtain the trained color image processing model. In this way, using the color sample image and the depth sample image as training samples at the same time has the effect of information complementarity, and a color image processing model with good effect can be trained with fewer training samples, thereby reducing the difficulty of training and improving the training efficiency.

[0105] Embodiment 2

[0106] In order to better reflect the purpose of this application, further examples are given based on the first embodiment of this application. Figure 4 As shown, after obtaining the trained color image processing model, the image processing method further includes:

[0107] Step 401: input the color sample image into the first encoding module of the trained color image processing model, and output an encoded second color sample image;

[0108] Here, the trained color image processing model also includes: a first encoding module and a first decoding module.

[0109] Step 402: Determine a fifth loss parameter of a control model based on the encoded second color sample image and the state label of the robot;

[0110] In practical applications, the training sample set also includes: at least one robot state label corresponding to the color sample image.

[0111] Specifically, the image acquisition device acquires the color sample image and the depth sample image containing the target object and simultaneously obtains the historical state information of the robot, and uses the historical state information as the state label.

[0112] The method for determining the fifth loss parameter may specifically include: inputting the encoded second color sample image into the control model to obtain the predicted state information of the robot; and determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.

[0113] In some embodiments, when the control model only includes the first control model, the encoded second color sample image is used as input information of the control model to output the predicted state information of the robot. The predicted state information may include joint angles, joint linear velocities, joint angular velocities, robot load, end effector position and posture, etc.

[0114] In some embodiments, when the control model includes at least two control models, such as a first control model and a second control model, the encoded second color sample image is used as input information of the second control model, and the output information of the second control model is used as input information of the first control model to output the predicted state information. For example, the second control model is an LSTM model with a memory function, and the first control model is a Mixture Density Networks (MDN) model.

[0115] Step 403: adjusting the control model based on the fifth loss parameter to obtain a trained control model;

[0116] Specifically, when the fifth loss parameter is greater than the second target loss parameter, the control model is adjusted so that the fifth loss parameter corresponding to the trained control model is not greater than the second target loss parameter.

[0117] Step 404: when controlling the robot to perform a target task on a target object, obtaining a target color image containing the target object;

[0118] Specifically, when conducting robot testing, the robot is controlled to perform a target task on a target object (such as controlling a robot arm to perform a grasping operation), a color camera captures a target color image, and then the result status information for the target object is determined through the target object information carried in the target color image.

[0119] Step 405: input the target color image into the first encoding module of the trained color image processing model, and output the encoded target color image;

[0120] Step 406: Input the encoded target color image into the trained control model to obtain result state information of the robot, and control the robot based on the result state information.

[0121] That is, when controlling the robot to perform an action, the trained color image processing model and control model are used to determine the result state information of the robot. For example, the result state information may include the angle information of the robot's six joints, and the operation of the robot is controlled based on the state control information.

[0122] Embodiment 3

[0123] In order to better reflect the purpose of this application, further examples are given based on the first embodiment of this application. Figure 5 As shown, after obtaining the trained color image processing model and the trained control model, the image processing method further includes:

[0124] Step 501: input the color sample image into the first encoding module of the trained color image processing model, and output an encoded second color sample image;

[0125] Here, the trained color image processing model also includes: a first encoding module and a first decoding module.

[0126] Step 502: input the depth sample image into the second encoding module of the trained depth image processing model, and output an encoded second depth sample image;

[0127] Here, the trained deep image processing model also includes: a second encoding module and a second decoding module.

[0128] In practical applications, the first encoding module and the second encoding module may be encoders, and the first decoding module and the second decoding module may be decoders.

[0129] Step 503: determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot;

[0130] In practical applications, the training sample set also includes: at least one robot state label corresponding to the color sample image.

[0131] Specifically, the image acquisition device acquires the color sample image and the depth sample image containing the target object and simultaneously obtains the historical state information of the robot, and uses the historical state information as the state label.

[0132] The method for determining the fifth loss parameter may specifically include: inputting the encoded second color sample image and the encoded second depth sample image into the control model to obtain the predicted state information of the robot; and determining the fifth loss parameter based on the predicted state information of the robot and the robot state label.

[0133] In some embodiments, when the control model only includes the first control model, the encoded second color sample image is used as input information of the control model to output the predicted state information of the robot. The predicted state information may include joint angles, joint linear velocities, joint angular velocities, robot load, end effector position and posture, etc.

[0134] In some embodiments, when the control model includes at least two control models, such as a first control model and a second control model, the encoded second color sample image is used as input information of the second control model, and the output information of the second control model is used as input information of the first control model to output the predicted state information. For example, the second control model is an LSTM model with a memory function, and the first control model is an MDN model.

[0135] Step 504: adjusting the control model based on the fifth loss parameter to obtain a trained control model;

[0136] Specifically, when the fifth loss parameter is greater than the second target loss parameter, the control model is adjusted so that the fifth loss parameter corresponding to the trained control model is not greater than the second target loss parameter.

[0137] Step 505: when controlling the robot to perform a target task on a target object, obtaining a target color image and a target depth image containing the target object;

[0138] Specifically, when conducting robot testing, the robot is controlled to perform a target task on the target object (such as controlling the robot arm to perform a grasping operation), the color camera collects the target color image, and the depth camera collects the target depth image, and then the result status information for the target object is determined based on the target object information carried in the color image and the depth image.

[0139] Step 506: input the target color image into the first encoding module of the trained color image processing model, and output the encoded target color image; input the target depth image into the second encoding module of the trained depth image processing model, and output the encoded target depth image;

[0140] Step 507: Input the encoded target color image and the encoded target depth image into the trained control model to obtain result state information of the robot, and control the robot based on the result state information.

[0141] That is to say, when controlling the robot to perform an action, the trained color image processing model, depth image processing model and control model can be used to determine the result state information of the robot. For example, the result state information may include the angle information of the robot's six joints, and the operation of the robot is controlled based on the state control information.

[0142] In other embodiments, the trained deep image processing model and control model may also be used to determine the result state information of the robot.

[0143] Based on the above embodiment, the following application scenario is given, in which the image processing model is a variational auto encoder (VAE) model, the first control model is an MDN model, and the second control model is an LSTM model; Figure 6 As shown, the image processing method includes:

[0144] Step 601: data collection;

[0145] like Figure 2As shown in the figure, the human instructor controls the robot to complete a task through the controller. During this process, the data acquisition system records the image information of the camera and the status information of the robot at each moment through the Robot Operating System (ROS). The information that the data acquisition system can collect includes: color images and depth images collected by cameras of different models (such as Kinect V2), with adjustable image resolution; status information of robots of different models (such as UR robots), which may include joint angles, joint linear velocities, joint angular velocities, robot loads, end effector positions and postures, etc.

[0146] Step 602: data preprocessing;

[0147] In order to increase the amount of data and alleviate network overfitting, the collected image data is enhanced to increase the number and diversity of samples. Enhancement processing includes: random cropping, horizontal flipping, resizing, hue adjustment, brightness adjustment, saturation adjustment and other preprocessing of given sample images. For example, the original color image size is 540*540, which can be randomly cropped to 480*480, randomly rotated 1~3°, and randomly flipped left and right. This data enhancement method can increase the amount of data by dozens of times.

[0148] Step 603: training an image processing model;

[0149] In the embodiment of the present application, a two-stream network structure for cross-modal information fusion is adopted, which integrates information from multiple modalities such as color images, depth images, and robot state information. By making the feature map obtained by color image encoding and the feature map obtained by depth image encoding similar, the two-stream network can better learn scene information and train a deep neural network with good results with less data.

[0150] like Figure 7 As shown in the figure, the robot control system consists of a two-stream network, which includes a first branch with a color image as input and a second branch with a depth image as input. The first branch includes encoder 1 and decoder 1, and the second branch includes encoder 2 and decoder 2. The encoder can encode the image and compress the image into a shorter vector, which can implicitly represent important information in the image, such as the position and posture of the robot arm, the color, shape and position of the object, etc. The vector is sampled and used as the input of the decoder, which is decoded by the decoder into a reconstructed image. By comparing it with the sample image, the effect of VAE network training can be intuitively observed.

[0151] Exemplarily, encoder 1 and encoder 2 are respectively composed of 4 convolutional layers, and each convolutional layer is followed by Batch Normalization. The convolutional layer uses a 4*4 convolution kernel with a step size of 2. Decoder 1 and decoder 2 are composed of 4 deconvolutional layers, and each convolutional layer is followed by Batch Normalization. The deconvolutional layer uses a 4*4 convolution kernel with a step size of 2. The last deconvolutional layer uses tanh as the activation function, and the remaining layers use the leaky_relu function as the activation function. After the decoder performs deconvolution on the image, it reconstructs the image to obtain a reconstructed image. The activation function can also be ReLU, Sigmoids, softmax, etc. Optionally, the image data is normalized before being input into the VAE network to reduce the image pixels proportionally to reduce the difficulty of computer data processing and increase the data processing speed.

[0152] The embodiment of the present application uses two encoders and two decoders, wherein encoder 1 and decoder 1 are used to encode and decode color images, and encoder 2 and decoder 2 are used to decode and encode depth images. Encoder 1 encodes the color image and outputs the encoded color image (i.e., the first feature map) as F 1 ; Encoder 2 encodes the depth image and outputs the encoded depth image (i.e., the second feature map) as F 2 Since the input color image and depth image correspond to the same scene, by making the feature maps output by the two encoders as similar as possible, information from different modalities can supervise each other to achieve better learning of the scene, thereby training a good network with less data.

[0153] The mean square error between the two feature maps will be used as the first loss function of the VAE network:

[0154]

[0155] Among them, the first loss function is used to calculate the second loss parameter, L1 is the mean square error between the first feature map and the second feature map (i.e., the second loss parameter), and F i 1 is the i-th vector in the first feature map, F i 2 is the i-th vector in the second feature map, and N is the total number of vectors in the feature map. Figure 7 In the formula, mse loss 2 is used to calculate the second loss parameter. The input of mse loss 2 is the first feature map and the second feature map, and the output is the second loss parameter.

[0156] The first reconstructed image output by decoder 1 is compared with the color sample image, and the mean square error between the two images is used as the second loss function of the VAE network:

[0157]

[0158] Among them, the second loss function is used to calculate the third loss parameter, L2 is the color sample image and the first reconstructed image The mean square error (i.e. the third loss parameter), is the data of the pixel at the i-th row and j-th column of the color sample image, is the data of the pixel point in the i-th row and j-th column of the first reconstructed image, H is the number of rows of image pixels, and W is the number of columns of image pixels. Figure 7 In the above example, mse loss 3 is used to calculate the third loss parameter. The input of mse loss 3 is the color sample image and the first reconstructed image, and the output is the third loss parameter.

[0159] The second reconstructed image output by decoder 2 is compared with the depth sample image of the depth image, and the mean square error between the two images is used as the third loss function of the VAE network:

[0160]

[0161] Among them, the third loss function is used to calculate the fourth loss parameter, L3 is the depth sample image and the second reconstructed image The mean square error (i.e. the fourth loss parameter), is the data of the pixel point in the i-th row and j-th column of the original image, is the data of the pixel point in the i-th row and j-th column of the second reconstructed image, H is the number of rows of image pixels, and W is the number of columns of image pixels. Figure 7 In the mse loss4, mse loss4 is used to calculate the fourth loss parameter. The input of mse loss4 is the depth sample image and the second reconstructed image, and the output is the fourth loss parameter.

[0162] The calculation formula of the first loss parameter L can be:

[0163] L=L1+L2+L3

[0164] Figure 7 In the figure, VAE loss1 is used to calculate the first loss parameter. The input of VAE loss1 is the second loss parameter, the third loss parameter and the fourth loss parameter, and the output is the first loss parameter.

[0165] Step 604: training a control model;

[0166] Next, the vector z output by encoder 1 is sampled and input into the LSTM model of the control network, so that the robot can learn how to complete a sequence of actions. This application uses an LSTM layer, which consists of 100 LSTM neurons, each of which includes a forget gate, an input gate, and an output gate.

[0167] Finally, the output information of the LSTM network is input into the MDN model, and the predicted joint angle of each joint of the robot is output after sampling. The MDN model is suitable for solving multiple solution problems, which are widely present in the field of robotics. This application uses 360 neurons to form the MDN layer to represent the angle information of the robot's 6 joints, and every 60 neurons represent the relevant information of a joint angle: 20 neurons represent the mean μ of the joint, 20 neurons represent the variance σ of the joint, and 20 neurons represent the mixing coefficient α of the joint.

[0168] The loss function of the MDN network is defined as:

[0169]

[0170] in,

[0171] Among them, x is the predicted joint angle output by the MDN network (i.e., joint output), y is the historical joint angle (i.e., joint label), m is the number of neurons, and c is the number of degrees of freedom of the robot. For example, every 60 neurons represent the relevant information of a joint angle: 20 neurons represent the mean of the joint, 20 neurons represent the variance of the joint, and 20 neurons represent the mixing coefficient of the joint, then m = 20, and in the UR5 robot, c = 6.

[0172] When setting up training, we first use the historical image information set to train the VAE model. After the VAE model is trained, its parameters are fixed, and the LSTM model and MDN model are trained using historical color images and robot state information. All three models use the Adam optimizer for gradient descent, the learning rate is set to 10e-4, the mini-batch size is 64, and all parameters are randomly initialized.

[0173] Step 605: Execute control operation.

[0174] Here, the trained image processing model and control model are used to perform control operations, that is, image information containing the target object is collected, and the image information can be a color image or a depth image. The image information encoded by the image processing model is used as the input information of the control model, and the control model outputs state control information for at least one moment of the target object, and the operation is performed on the target object based on the state control information at at least one moment. For example, when the target object is an object to be grasped by the robot, since the grasping operation is a series of actions, it is necessary to determine the state information of the robot at multiple moments to achieve continuous control operations.

[0175] Here, the state information at at least one moment may be the state information of the robot at the last moment, or the state information corresponding to equally spaced moments on a predetermined moving path.

[0176] Embodiment 4

[0177] Based on the same inventive concept, the present application also provides an image processing device, such as Figure 8 As shown, the device comprises:

[0178] An acquiring unit 801 is configured to acquire a training sample set, wherein the training sample set includes at least one color sample image and at least one depth sample image corresponding to the color sample image;

[0179] A processing unit 802 is used to input the training sample set into an image processing model for processing, wherein the image processing model includes a color image processing model and a depth image processing model, wherein the color image processing model is used to process the color sample image to obtain a color reconstructed image, and the depth image processing model is used to process the depth image to obtain a depth reconstructed image;

[0180] The processing unit 802 is also used to determine a first loss parameter of the image processing model based on the color sample image, the color reconstructed image, the depth sample image and the depth reconstructed image, and adjust the color image processing model based on the first loss parameter to obtain a trained color image processing model.

[0181] In some embodiments, the processing unit is further used to adjust the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.

[0182] In some embodiments, the color image processing model includes a first encoding module, the depth image processing model includes a second encoding module, and the processing unit is specifically used to input the color sample image into the first encoding module of the color image processing model, and output the encoded first color sample image; input the depth sample image into the second encoding module of the color image processing model, and output the encoded first depth sample image; and obtain a first loss parameter based on the encoded first color sample image and the encoded first depth sample image.

[0183] In some embodiments, the color image processing model further includes a first decoding module, and the depth image processing model further includes a second decoding module.

[0184] The processing unit is specifically configured to input the encoded first color sample image into the first decoding module, and output the color reconstructed image;

[0185] Inputting the encoded first depth sample image into the second decoding module, and outputting the depth reconstructed image;

[0186] Calculate a second loss parameter using the encoded first color sample image and the encoded first depth sample image;

[0187] Determining a third loss parameter based on the encoded first color sample image and the color reconstructed image;

[0188] Determining a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image;

[0189] The first loss parameter is determined based on the second loss parameter, the third loss parameter and the fourth loss parameter.

[0190] In some embodiments, the training sample set further includes: at least one robot state label corresponding to the color sample image;

[0191] The processing unit is further configured to input the color sample image into the first encoding module of the trained color image processing model, and output an encoded second color sample image; determine a fifth loss parameter of the control model based on the encoded second color sample image and the state label of the robot; and adjust the control model based on the fifth loss parameter to obtain a trained control model;

[0192] Alternatively, the processing unit is further used to input the color sample image into the first encoding module of the trained color image processing model, and output the encoded second color sample image; input the depth sample image into the second encoding module of the trained depth image processing model, and output the encoded second depth sample image; determine the fifth loss parameter of the control model based on the encoded second color sample image and the encoded second depth sample image, and the state label of the robot; and adjust the control model based on the fifth loss parameter to obtain the trained control model.

[0193] In some embodiments, the state label includes at least one of the following: joint angle, joint linear velocity and joint angular velocity.

[0194] In some embodiments, the acquisition unit is further used to acquire a target color image containing the target object when controlling the robot to perform a target task on the target object;

[0195] The processing unit is also used to input the target color image into the first encoding module of the trained color image processing model, and output the encoded target color image; input the encoded target color image into the trained control model to obtain the result status information of the robot, and control the robot based on the result status information.

[0196] In some embodiments, the acquisition unit is further used to acquire a target color image and a target depth image containing the target object when controlling the robot to perform a target task on the target object;

[0197] The processing unit is further used to input the target color image into the first encoding module of the trained color image processing model, and output the encoded target color image; input the target depth image into the second encoding module of the trained depth image processing model, and output the encoded target depth image; input the encoded target color image and the encoded target depth image into the trained control model to obtain the result status information of the robot, and control the robot based on the result status information.

[0198] In some embodiments, the acquisition unit is specifically used to perform data enhancement processing on the collected color sample images and the collected depth sample images to increase the number of samples and obtain the training sample set.

[0199] By adopting the above technical solution, the color sample image and the depth sample image are input into the image processing model, and the color image processing model is adjusted by using the first loss parameter of the image processing model to obtain the trained color image processing model. In this way, using the color sample image and the depth sample image as training samples at the same time has the effect of information complementarity, and a color image processing model with good effect can be trained with fewer training samples, thereby reducing the difficulty of training and improving the training efficiency.

[0200] The present application also provides another image processing device, such as Fig. 9 As shown, the apparatus includes: a processor 901 and a memory 902 configured to store a computer program that can be run on the processor;

[0201] The processor 901 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0202] Of course, in practical applications, Fig. 9 As shown, the components in the image processing device are coupled together via a bus system 903. It is understood that the bus system 903 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 903 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 903.

[0203] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0204] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0205] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory 902 including a computer program, and the above-mentioned computer program can be executed by the processor 901 of the image processing device to complete the above-mentioned method steps.

[0206] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0207] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0208] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the instructions in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0209] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0210] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a training sample set, wherein the training sample set includes at least one color sample image and at least one depth sample image corresponding to the color sample image; Inputting the training sample set into an image processing model for processing, the image processing model comprising a color image processing model and a depth image processing model, the color image processing model being used to process the color sample image to obtain a color reconstructed image, and the depth image processing model being used to process the depth image to obtain a depth reconstructed image; Inputting the color sample image into the first encoding module of the color image processing model, and outputting the encoded first color sample image; Inputting the depth sample image into a second encoding module of the depth image processing model, and outputting an encoded first depth sample image; Calculate a second loss parameter using the encoded first color sample image and the encoded first depth sample image; determining a third loss parameter based on the encoded first color sample image and the color reconstructed image; determining a first loss parameter based on the second loss parameter and the third loss parameter; or, Determine a fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determine the first loss parameter based on the second loss parameter and the fourth loss parameter; The color image processing model is adjusted based on the first loss parameter to obtain a trained color image processing model.

2. The method according to claim 1, characterized in that The method also includes: adjusting the depth image processing model based on the first loss parameter to obtain a trained depth image processing model.

3. The method according to claim 1, characterized in that The training sample set also includes: at least one robot state label corresponding to the color sample image, and the method further includes: Inputting the color sample image into the first encoding module of the trained color image processing model, and outputting an encoded second color sample image; Determining a fifth loss parameter of a control model based on the encoded second color sample image and the state label of the robot; Adjust the control model based on the fifth loss parameter to obtain a trained control model; Alternatively, the method further comprises: Inputting the color sample image into the first encoding module of the trained color image processing model, and outputting an encoded second color sample image; Inputting the depth sample image into a second encoding module of the trained depth image processing model, and outputting an encoded second depth sample image; Determining a fifth loss parameter of a control model based on the encoded second color sample image and the encoded second depth sample image, and a state label of the robot; The control model is adjusted based on the fifth loss parameter to obtain a trained control model.

4. The method according to claim 3, characterized in that The step of determining a fifth loss parameter of a control model based on the encoded second color sample image and the state label of the robot comprises: Inputting the encoded second color sample image into the control model to obtain predicted state information of the robot; The fifth loss parameter is determined based on the predicted state information of the robot and the robot state label.

5. The method according to claim 4, characterized in that The control model includes a first control model and a second control model, and the step of inputting the encoded second color sample image into the control model to obtain the predicted state information of the robot includes: The encoded second color sample image is used as input information of the second control model, the output information of the second control model is used as input information of the first control model, and the predicted state information is output.

6. The method according to claim 3, characterized in that The step of determining a fifth loss parameter of a control model based on the encoded second color sample image, the encoded second depth sample image, and the state label of the robot comprises: Inputting the encoded second color sample image and the encoded second depth sample image into the control model to obtain predicted state information of the robot; The fifth loss parameter is determined based on the predicted state information of the robot and the robot state label.

7. The method according to claim 3, characterized in that The method further comprises: When controlling the robot to perform a target task on a target object, a target color image containing the target object is obtained; Inputting the target color image into the first encoding module of the trained color image processing model, and outputting the encoded target color image; The encoded target color image is input into the trained control model to obtain result state information of the robot, and the robot is controlled based on the result state information.

8. The method according to claim 3, characterized in that The method further comprises: When controlling the robot to perform a target task on a target object, a target color image and a target depth image containing the target object are acquired; Inputting the target color image into the first encoding module of the trained color image processing model, and outputting the encoded target color image; Inputting the target depth image into a second encoding module of the trained depth image processing model, and outputting an encoded target depth image; The encoded target color image and the encoded target depth image are input into the trained control model to obtain result state information of the robot, and the robot is controlled based on the result state information.

9. The method according to claim 3, characterized in that: The obtaining of the training sample set comprises: Data enhancement processing is performed on the collected color sample images and the collected depth sample images to increase the number of samples, thereby obtaining the training sample set.

10. An image processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a training sample set, wherein the training sample set includes at least one color sample image and at least one depth sample image corresponding to the color sample image; a processing unit, configured to input the training sample set into an image processing model for processing, wherein the image processing model includes a color image processing model and a depth image processing model, wherein the color image processing model is configured to process the color sample image to obtain a color reconstructed image, and the depth image processing model is configured to process the depth image to obtain a depth reconstructed image; The processing unit is further used to input the color sample image into the first encoding module of the color image processing model, and output the encoded first color sample image; input the depth sample image into the second encoding module of the depth image processing model, and output the encoded first depth sample image; calculate the second loss parameter using the encoded first color sample image and the encoded first depth sample image; determine the third loss parameter based on the encoded first color sample image and the color reconstructed image; determine the first loss parameter based on the second loss parameter and the third loss parameter; or determine the fourth loss parameter based on the encoded first depth sample image and the depth reconstructed image; determine the first loss parameter based on the second loss parameter and the fourth loss parameter; adjust the color image processing model based on the first loss parameter to obtain a trained color image processing model.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. An image processing device, characterized in that: The apparatus comprises a processor and a memory configured to store a computer program executable on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 9 when running the computer program.

Citation Information

Patent Citations

  • Image visual saliency region detection method based on deep automatic encoder reconfiguration

    CN105139385A

  • Anti-robustness image feature extraction method based on variational spherical projection

    CN108520202A

  • Article classification method based on depth recovery information

    CN108520535A

  • Image processing method and apparatus

    CN108876745A

  • Monocular depth estimation method, device and equipment and storage medium

    CN108961327A