Method and apparatus for updating an object detection model

Abstract

The present disclosure provides a method and apparatus for updating an object detection model. A specific embodiment of the method includes: constructing a three-dimensional model of a target object based on image data of the target object at multiple angles; generating a synthetic image including a target object object to represent the target object based on the three-dimensional model; taking the synthetic image as a sample image, taking the target object object as a label, obtaining training samples, generating a training sample set; and training an object detection model using the training sample set to obtain an updated object detection model.

Description

[Technical field]

[0001] <CROSS REFERENCE TO RELATED APPLICATIONS> This application claims priority to a Chinese patent application filed on August 17, 2021, bearing application number 202110957524.X and entitled "Method and apparatus for updating target detection model," the entire text of which is incorporated herein by reference.

[0002] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and apparatus for updating an object detection model. [Background technology]

[0003] In the field of intelligent automated storage, the method of a picking robot performing a picking task has good application prospects. A picking robot performing a picking task refers to the picking robot being visually guided to pick a target article to a specified position. High-precision detection of target detection is the core technology foundation of the application scene. Object detection technology based on deep learning is currently a relatively practical technology. Deep learning technology for object detection belongs to the category of supervised learning, and the basic process is as follows: (1) collect scene data in batches in advance; (2) manually label the data; (3) train an object detection model; (4) deploy the trained object detection model in an online application. It can be understood that object detection models usually have timeliness problems. For example, in e-commerce scenes, products are frequently replaced, and the accuracy of an object detection model trained on early data will deteriorate over time. Re-running the above process of obtaining an object detection model can restore the accuracy of the model, but the process takes time and effort. Summary of the Invention

[0004] SUMMARY OF THE DISCLOSURE The embodiments of the present disclosure provide a method and apparatus for updating an object detection model.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for updating an object detection model, the method including: constructing a three-dimensional model of a target article based on image data of the target article at multiple angles; generating a synthetic image including a target article object to represent the target article based on the three-dimensional model; using the synthetic image as a sample image and the target article object as a label, acquiring training samples, and generating a training sample set; and training the object detection model using the training sample set to obtain an updated object detection model, wherein the object detection model represents a correspondence between an input image and a detection result corresponding to the target article object in the input image.

[0006] In some embodiments, the step of constructing a three-dimensional model of the target item based on image data of the target item at multiple angles includes, in a process of controlling a picking robot to perform a picking task of the target item based on a detection result of the target item object in the input image by the target detection model, collecting two-dimensional image data and three-dimensional image data at multiple angles of the target item at a pre-set information collection position by a first image collection device, and constructing a three-dimensional model of the target item based on the two-dimensional image data and the three-dimensional image data.

[0007] In some embodiments, generating a composite image including a target item object to represent the target item based on the three-dimensional model includes determining a coordinate system correspondence between a second image collection device corresponding to the picking robot, a preset information collection position, and the first image collection device, determining an adjusted three-dimensional model of the three-dimensional model from the viewpoint of the second image collection device based on the coordinate system correspondence, and generating a composite image including a target item object corresponding to the target item based on the adjusted three-dimensional model and a preset background image representing a picking scene of the picking robot.

[0008] In some embodiments, the above method further includes a step of identifying a weight of the target item, and the step of generating a synthetic image including a target item object corresponding to the target item based on the above-mentioned adjusted 3D model and a preset background image representing a picking scene of the picking robot includes a step of generating a synthetic image including a target item object corresponding to the target item based on the adjusted 3D model, the preset background image, the weight, a preset resolution of the synthetic image, and parameters of the second image collection device.

[0009] In some embodiments, the step of training the object detection model using the above-mentioned training sample set to obtain an updated object detection model includes a step of training the object detection model using the training sample set by a machine learning algorithm to obtain an updated object detection model in response to determining that the detection accuracy of the object detection model is lower than a predetermined threshold.

[0010] In some embodiments, the method further includes the steps of performing object detection on the subsequent input image using the updated object detection model to obtain a detection result, and controlling the picking robot to perform the picking task based on the detection result.

[0011] In a second aspect, an embodiment of the present disclosure provides an apparatus for updating an object detection model, comprising: a construction unit configured to construct a three-dimensional model of a target article based on image data of the target article at multiple angles; an image generation unit configured to generate a synthetic image including a target article object to represent the target article based on the three-dimensional model; a sample generation unit configured to obtain training samples and generate a training sample set, with the synthetic image as a sample image and the target article object as a label; and an update unit configured to train the object detection model using the training sample set to obtain an updated object detection model, wherein the object detection model represents a correspondence between an input image and a detection result corresponding to the target article object in the input image.

[0012] In some embodiments, the construction unit is further configured to collect two-dimensional image data and three-dimensional image data at multiple angles of the target item at a preset information collection position by the first image collection device, and construct a three-dimensional model of the target item based on the two-dimensional image data and the three-dimensional image data, in the process of controlling the picking robot to perform a picking task of the target item based on the detection result of the target item object in the input image by the target detection model.

[0013] In some embodiments, the image generation unit is further configured to determine a coordinate system correspondence between a second image collection device corresponding to the picking robot, a predetermined information collection position, and the first image collection device, determine an adjusted three-dimensional model of the three-dimensional model from the viewpoint of the second image collection device based on the coordinate system correspondence, and generate a composite image including a target item object corresponding to the target item based on the adjusted three-dimensional model and a predetermined background image representing a picking scene of the picking robot.

[0014] In some embodiments, the apparatus further comprises an identification unit configured to identify a weight of the target item, and the image generation unit is further configured to generate a synthetic image including a target item object corresponding to the target item based on the adjusted three-dimensional model, the pre-set background image, the weight, the pre-set resolution of the synthetic image, and parameters of the second image capture device.

[0015] In some embodiments, the update unit is further configured, in response to determining that the detection accuracy of the object detection model is lower than a predetermined threshold, to train the object detection model using a machine learning algorithm with the training sample set to obtain an updated object detection model.

[0016] In some embodiments, the apparatus further comprises an acquisition unit configured to perform object detection on subsequent input images with the updated object detection model to obtain a detection result, and an execution unit configured to control the picking robot to perform the picking task based on the detection result.

[0017] In a third aspect, an embodiment of the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method according to any of the embodiments of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides an electronic device comprising one or more processors and a storage device having one or more programs stored therein, the one or more programs, when executed by the one or more processors, causing the one or more processors to implement a method according to any of the embodiments of the first aspect.

[0019] In a fifth aspect, an embodiment of the present disclosure provides a computer program product that, when executed by a processor, implements a method according to any of the embodiments of the first aspect. but Provided. [Brief description of the drawings]

[0020] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the following detailed description of non-limiting embodiments, taken in conjunction with the drawings in which: [Figure 1] FIG. 1 illustrates an exemplary system architecture to which an embodiment of the present disclosure can be applied. [Diagram 2] FIG. 2 illustrates another exemplary system architecture to which an embodiment of the present disclosure can be applied. [Diagram 3] 1 is a flowchart of an embodiment of a method for updating an object detection model according to the present disclosure. [Figure 4] 1 is a schematic diagram illustrating an application scene of a method for updating an object detection model according to an embodiment of the present invention. [Diagram 5] 1 is a flowchart of another embodiment of a method for updating an object detection model according to the present disclosure. [Figure 6] FIG. 2 is a structural diagram of an embodiment of an apparatus for updating an object detection model according to the present disclosure; [Figure 7] FIG. 1 is a schematic structural diagram of a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The present disclosure will be described in more detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described in this specification are merely for the purpose of explaining the related invention, and do not limit the invention. For the sake of convenience, only parts related to the invention are shown in the drawings.

[0022] In addition, the embodiments and features of the embodiments of the present disclosure can be combined with each other as long as no contradiction occurs. Hereinafter, the present disclosure will be described in detail with reference to the drawings and embodiments.

[0023] FIG. 1 illustrates an example system architecture 100 in which the object detection model updating method and apparatus of the present disclosure may be applied.

[0024] 1, the system architecture 100 may include image collection devices 101, 102, networks 103, 104, a controller 105, and a picking robot 106. The communication connections between the image collection devices 101, 102, the controller 105, and the picking robot 106 constitute a topological network, and the networks 103, 104 are a medium for providing communication links between the image collection devices 101, 102, the controller 105, and the picking robot 106. The networks 103, 104 may include various types of connections, such as wired, wireless communication links, or fiber optic cables.

[0025] The image collecting devices 101 and 102 may be hardware devices or software having 2D image collecting functions, 3D image collecting functions, and information transmission functions. If the image collecting devices 101 and 102 are hardware, they may be various electronic devices that support functions such as network connection, image acquisition, interaction, display, processing, etc., including, but not limited to, 2D cameras, 3D cameras, smartphones, tablet computers, and desktop computers. If the image collecting devices 101 and 102 are software, they may be installed in the above-exemplified electronic devices. It may be implemented, for example, as multiple software or software modules for providing distributed services, or may be implemented as a single software or software module. There is no particular limitation here.

[0026] The control device 105 may be a server that provides various services, for example, a server that automatically generates training data and automatically updates an object detection model in a process of controlling a picking robot to perform a picking task based on a detection result of a target item in an image to be detected by an object detection model. As an example, the control device 105 may be a cloud server. Specifically, the control device 105 obtains a detection result of a target item in an image to be detected using an object detection model, controls a picking robot to perform a picking task based on the detection result, and obtains image data of the picked target item at multiple angles to construct a three-dimensional model of the target item to obtain training data in the process of the picking robot performing the picking task, and updates the target detection model using the training data.

[0027] The control device may be hardware or software. If the control device is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. If the control device is software, it may be implemented as multiple software programs or software modules (e.g., software programs or software modules for providing distributed services), or as a single software program or software module. There are no particular limitations here.

[0028] The picking robot may be any robot having a picking function, for example, a multi-degree-of-freedom robot arm.

[0029] The object detection model updating method and apparatus provided by the embodiments of the present disclosure include the steps of: constructing a 3D model of the target article based on image data of the target article at multiple angles; generating a synthetic image including a target article object to represent the target article based on the 3D model; using the synthetic image as a sample image and the target article object as a label, obtaining training samples, and generating a training sample set; and training the object detection model using the training sample set to obtain an updated object detection model, thereby providing a method for automatically generating training data and automatically updating the object detection model, thereby improving the convenience of updating the object detection model and the accuracy of the detection results.

[0030] As shown in FIG. 2, another exemplary architecture 200 of a system for updating an object detection model is shown, including a first image collection device 201, a second image collection device 202, a third image collection device 203, a control device 204, and a picking robot 205.

[0031] The third image collection device 203 is provided at a picking station, a returnable box containing a target item to be picked is placed at the picking station, and the third image collection device 203 is configured to collect an image including a target object corresponding to the target item in the returnable box. The control device 204 acquires a detection result using the image collected by the third image collection device 203 as an input image of a target detection model, and controls the picking robot to execute a picking task (moving the target item from the picking station to the stacking station) based on the detection result.

[0032] The second image collecting device 202 is provided corresponding to the picking robot. The first image collecting device 201 is provided at a preset information collecting position. The preset information collecting position may be any position through which the picking robot passes during the movement process of picking the target item. For example, the preset position is set near a picking station where the target item to be picked is placed. The first image collecting device 201 is configured to acquire image data of the target item at a plurality of angles in response to determining that the picking robot has picked the target item and moved to the preset information collecting position during the process of the picking robot picking the target item from the picking station to the stacking station. The control device 204 is further configured to generate training data by constructing a three-dimensional model of the target item based on the image data of the target item at a plurality of angles, and update the target detection model.

[0033] The object detection model updating method according to an embodiment of the present disclosure may be executed by a control device, and accordingly, each part (e.g., each unit) included in the device for updating the object detection model may all be provided in the control device.

[0034] It should be understood that the number of image collecting devices, networks, control devices, and picking robots in FIG. 1 is merely exemplary. The number of image collecting devices, networks, control devices, and picking robots may be arbitrarily increased or decreased according to the needs of implementation. If the electronic device in which the object detection model update method operates does not need to transmit data with other electronic devices, the system architecture may only include an electronic device (e.g., a control device) in which the object detection model update method operates.

[0035] Reference is now made to FIG. 3, which illustrates a flow 300 of one embodiment of a method for updating an object detection model, which includes the following steps:

[0036] In step 301, a three-dimensional model of the target article is constructed based on image data of the target article at multiple angles.

[0037] In this embodiment, an entity executing the target detection model updating method (e.g., the terminal device or server in Figure 1) can remotely or locally acquire image data of the target object at multiple angles via a wired connection network or a wireless connection network, and further construct a three-dimensional model of the target object based on the image data of the target object at the multiple angles.

[0038] The object detection model is used to represent the correspondence between the input image and the detection result corresponding to the target object in the input image, and can be trained by a neural network model having a target object detection function. The object detection model includes, but is not limited to, a convolutional neural network, a residual neural network, and a recurrent neural network. Note that the object detection model is a model that is pre-trained and applied to the picking scene.

[0039] By inputting the image to be detected into the object detection model, a detection frame showing the target item in the image to be detected can be determined, and further, based on the orientation information of the image collection device (e.g., the third image collection device 203 in FIG. 2) that has acquired the image to be detected, the position information of the target item on the picking station in the real environment can be identified. Furthermore, a picking robot can be controlled to pick the target item at the position represented by the identified position information.

[0040] Here, the target object may be various entities. By way of example, the target object may be various products.

[0041] The image data includes 2D (2-dimension) image data and 3D image data. For example, the 2D image data is a RGB (red, green, blue) image, and the 3D image data is a point cloud image data.

[0042] In this embodiment, image data of the target object can be collected at multiple angles by multiple sets of cameras at different positions, each set of cameras including a 2D camera and a 3D camera.

[0043] In this embodiment, the executing entity can input image data of the target item from multiple angles into a network model for 3D modeling (e.g., Reality Capture published by Capturing Reality, Inc.) to obtain a three-dimensional model of the target item.

[0044] In some optional embodiments of this embodiment, the executing entity may first perform step 301 to collect two-dimensional image data and three-dimensional image data of the target item at multiple angles at a pre-set information collection position by a first image collection device in the process of controlling a picking robot to perform a picking task of the target item based on the detection result of the target item object in the input image by the target detection model, and then construct a three-dimensional model of the target item based on the two-dimensional image data and the three-dimensional image data.

[0045] Specifically, a preset information collection position is provided between the picking station and the stacking station, and a first image collection device including a plurality of sets of cameras is provided around the preset information collection position. In the process of picking the target item at the picking station and moving to the stacking station based on the detection result of the target detection model, the picking robot temporarily places the target item at the preset information collection position upon reaching the preset information collection position, so that the first image collection device can obtain two-dimensional image data and three-dimensional image data of the target item at a plurality of angles.

[0046] In step 302, a synthetic image including a target article object to represent the target article is generated based on the three-dimensional model.

[0047] In this embodiment, the actor may generate a synthetic image including a target article object to represent the target article based on a three-dimensional model.

[0048] For example, the execution subject inputs a 3D model of a target object into an image synthesis tool (e.g., Keyshot), and sets a background image on the 3D model to obtain a synthetic image. Specifically, multiple synthetic images can be generated for the 3D model of the same target object based on different angles of the 3D model and / or different background images.

[0049] In order to obtain a more accurate synthetic image and make the updated object detection model based on the training data obtained from the synthetic image more suitable for the actual scene of the picking robot workstation as shown in FIG. 2, in some optional embodiments of this embodiment, the execution entity may perform the above step 302 as follows:

[0050] First, a correspondence relationship between the second image collecting device corresponding to the picking robot, a preset information collecting position, and the first image collecting device is determined.

[0051] The second image collecting device, the preset information collecting position and the first image collecting device have corresponding coordinate systems. The execution subject can determine the correspondence relationship between the second image collecting device corresponding to the picking robot, the preset information collecting position and the first image collecting device through a location algorithm.

[0052] Second, an adjusted 3D model of the 3D model at the viewpoint of the second image capture device is determined based on the coordinate system correspondence.

[0053] By adjusting the 3D model to the viewpoint of the corresponding second image acquisition device of the picking robot, it can be well matched to the actual scene of the workstation of the picking robot.

[0054] Third, a synthetic image including a target item object corresponding to the target item is generated based on the adjusted 3D model and a pre-set background image representing a picking scene of the picking robot.

[0055] For example, the target items at the picking station are usually placed in tote boxes, and the preset background image may be a background image including a tote box object corresponding to an empty tote box, which can further improve the authenticity of the composite image.

[0056] In some optional embodiments of this embodiment, the execution subject may also specify a weight of the target object. For example, a weight sensor is provided at the preset information collection position to obtain the weight of the target object.

[0057] In this embodiment, the execution subject may execute the third step as follows.

[0058] A composite image including a target article object corresponding to the target article is generated based on the adjusted 3D model, the preset background image, the weights, the preset resolution of the composite image, and parameters of the second image capture device.

[0059] Here, the preset resolution is used to represent the resolution of the desired composite image, and the parameters of the second image capture device represent data of its internal parameters.

[0060] Specifically, the executing entity inputs data such as the adjusted 3D model, the preset background image, weights, the preset resolution of the composite image, parameters of the second image collection device, etc. as input data into a virtual physics engine tool (e.g., pybullet) to obtain a composite image.

[0061] The generated composite image may include multiple target object objects of the same type, or multiple target object objects of different types. For example, the different types of target objects may be multiple types of target objects that have been picked up so far. In the process of picking multiple types of target objects, a composite image of multiple target objects of different types can be obtained by a virtual physics engine tool based on the adjusted 3D model and weight information of each type of target object obtained by the execution subject, and a target detection model with higher detection accuracy can be trained and obtained by using the training data consisting of the composite image.

[0062] In step 303, the synthetic image is taken as a sample image, the target object is taken as a label, training samples are obtained, and a training sample set is generated.

[0063] In this embodiment, the execution subject can use the synthetic image as a sample image, use the target object as a label, obtain training samples, and generate a training sample set.

[0064] Synthetic images are synthesized based on a 3D model, and the target object can be captured naturally, eliminating the need for a manual labeling step and expanding the training sample set for updating the target detection model.

[0065] In step 304, the object detection model is trained using the training sample set to obtain an updated object detection model.

[0066] In this embodiment, the actor may train the object detection model using the training sample set to obtain an updated object detection model.

[0067] Specifically, first, an untrained training sample is selected from the training sample set, and a sample image in the selected training sample is input into the object detection model, and the label corresponding to the input training sample is the desired output to obtain the actual output of the object detection model. Then, the loss between the actual output and the label is calculated. Finally, the gradient is calculated based on the loss, and the parameters of the object detection model are updated based on the gradient descent method and the stochastic gradient descent method.

[0068] The execution entity may repeat the above update process until a preset termination condition is satisfied and an updated object detection model is obtained. Note that the preset termination condition may be, for example, that the number of training times exceeds a preset number threshold, that the training time exceeds a preset time threshold, or that the loss has converged.

[0069] In some optional embodiments of this embodiment, the execution entity may perform step 304 as follows.

[0070] In response to determining that the detection accuracy of the object detection model is lower than a preset threshold, a machine learning algorithm is used to train the object detection model using a training sample set to obtain an updated object detection model.

[0071] Here, the preset threshold value may be specifically set according to the actual situation. For example, when the detection accuracy required by the picking task is high, the preset threshold value may be set higher, and when the detection accuracy required by the picking task is not high, the preset threshold value may be set lower.

[0072] As an example, the executing entity may input a synthetic image into an unupdated target detection model to obtain a detection frame of the target object actually output by the target detection model, and further determine the IoU (Intersection-over-Union) between the actually output detection frame and the detection frame corresponding to the label, and average the IoUs corresponding to multiple synthetic images to determine the detection accuracy of the target detection model.

[0073] In some optional embodiments of this embodiment, after obtaining the updated object detection model, the execution entity performs object detection on subsequent input images via the updated object detection model, obtains a detection result, and controls the picking robot to perform a picking task based on the detection result.

[0074] Next, refer to FIG. 4, which is a schematic diagram 400 of one application scene of the method for updating the object detection model according to the present embodiment. The application scene in FIG. 4 includes an image collecting device 401, an image collecting device 402, a server 403, and a picking robot 404, and the server 403 is provided with an object detection model. The image collecting device 401 is installed at a picking station, photographs the object at the picking station to obtain an image to be detected, detects the object using the object detection model, obtains a detection result, and controls the picking robot to pick up the object and move it to the stacking station. When the picking robot performs a picking task for the object, image data at multiple angles of the object at a preset information collection position is acquired by the image collecting device 402. After acquiring the image data at multiple angles, the server first constructs a three-dimensional model of the object based on the image data of the object at multiple angles. Then, based on the three-dimensional model, generates a composite image including an object object to represent the object. The composite image is taken as a sample image, the object object is taken as a label, training samples are acquired, and a training sample set is generated. Then, the training sample set is used to train the object detection model to obtain an updated object detection model.

[0075] The method provided by the above embodiment of the present disclosure includes the steps of constructing a 3D model of the target article based on image data of the target article at multiple angles, generating a synthetic image including a target article object to represent the target article based on the 3D model, using the synthetic image as a sample image and the target article object as a label, obtaining training samples, and generating a training sample set, and training the object detection model using the training sample set to obtain an updated object detection model, thereby providing a method for automatically generating training data and automatically updating the object detection model, thereby improving the convenience of updating the object detection model and the accuracy of the detection results.

[0076] Reference is now made to Figure 5, which illustrates a schematic flow 500 of one embodiment of a method for updating an object detection model according to the present disclosure. The flow 500 includes the following steps:

[0077] In step 501, in the process of controlling a picking robot to perform a picking task of a target item based on the detection result of a target item object in an input image by a target detection model, a first image collection device collects 2D image data, 3D image data and weights at multiple angles of the target item located at a pre-set information collection position.

[0078] In step 502, a three-dimensional model of the target article is constructed based on the two-dimensional image data and the three-dimensional image data.

[0079] In step 503, a correspondence relationship between the second image collecting device corresponding to the picking robot, the preset information collecting position, and the coordinate system of the first image collecting device is determined.

[0080] In step 504, an adjusted 3D model of the 3D model from the viewpoint of the second image capture device is determined based on the coordinate system correspondence.

[0081] In step 505, a composite image including a target article object corresponding to the target article is generated based on the adjusted 3D model, the preset background image, the weights, the preset resolution of the composite image, and parameters of the second image capture device.

[0082] In step 506, the synthetic image is used as a sample image, the target object is used as a label, training samples are obtained, and a training sample set is generated.

[0083] In step 507, the training sample set is used to train the object detection model to obtain an updated object detection model.

[0084] Among them, the object detection model is used to represent the correspondence between the input image and the detection result corresponding to the target article object in the input image.

[0085] In step 508, object detection is performed on the subsequent input image using the updated object detection model to obtain a detection result.

[0086] In step 509, the picking robot is controlled to perform the picking task based on the detection result.

[0087] As can be seen from this embodiment, compared with the embodiment corresponding to Figure 3, the flow 500 of the object detection model updating method in this embodiment specifically describes the process of constructing a synthetic image and the application process of the updated object detection model, and provides a closed-loop continuous learning method for the object detection model, and creates a training sample set by constructing a 3D model of the target object, which can be applied to target objects of any shape, thereby improving the universality and detection accuracy of the object detection model.

[0088] Continuing to refer to FIG. 6, as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of an apparatus for updating an object detection model, which embodiment of the apparatus corresponds to the embodiment of the method shown in FIG. 3, and which can be specifically applied to various electronic devices.

[0089] As shown in FIG. 6, the apparatus for updating an object detection model includes: a construction unit 601 configured to construct a three-dimensional model of a target article based on image data of the target article at multiple angles; an image generation unit 602 configured to generate a synthetic image including a target article object to represent the target article based on the three-dimensional model; a sample generation unit 603 configured to take the synthetic image as a sample image and the target article object as a label, obtain training samples, and generate a training sample set; and an update unit 604 configured to train the object detection model using the training sample set to obtain an updated object detection model, where the object detection model represents a correspondence relationship between an input image and a detection result corresponding to the target article object in the input image.

[0090] In some optional embodiments of this embodiment, the construction unit 601 is further configured to collect two-dimensional image data and three-dimensional image data at multiple angles of the target item at a pre-set information collection position by the first image collection device in the process of controlling the picking robot to perform a picking task of the target item based on the detection result of the target item object in the input image by the target detection model, and construct a three-dimensional model of the target item based on the two-dimensional image data and the three-dimensional image data.

[0091] In some optional embodiments of this embodiment, the image generation unit 602 is further configured to determine a coordinate system correspondence between a second image collection device corresponding to the picking robot, a predetermined information collection position, and the first image collection device, determine an adjusted three-dimensional model of the three-dimensional model from the viewpoint of the second image collection device based on the coordinate system correspondence, and generate a composite image including a target item object corresponding to the target item based on the adjusted three-dimensional model and a predetermined background image representing a picking scene of the picking robot.

[0092] In some optional embodiments of this embodiment, the apparatus further comprises an identification unit (not shown) configured to identify a weight of the target item, and the image generation unit 602 is further configured to generate a composite image including a target item object corresponding to the target item based on the adjusted three-dimensional model, a pre-set background image, the weight, a pre-set resolution of the composite image, and parameters of the second image capture device.

[0093] In some optional embodiments of this embodiment, the update unit 604 is further configured to, in response to determining that the detection accuracy of the object detection model is lower than a predetermined threshold, train the object detection model using a training sample set by a machine learning algorithm to obtain an updated object detection model.

[0094] In some optional embodiments of this embodiment, the apparatus further includes an acquisition unit (not shown) configured to perform object detection on subsequent input images using the updated object detection model to obtain a detection result, and an execution unit (not shown) configured to control the picking robot to perform a picking task based on the detection result.

[0095] In this embodiment, a construction unit in the apparatus for updating the object detection model includes the steps of constructing a three-dimensional model of the target article based on image data of the target article at multiple angles, generating a synthetic image including a target article object to represent the target article based on the three-dimensional model, using the synthetic image as a sample image and the target article object as a label, obtaining training samples, and generating a training sample set, and using the training sample set to train an object detection model (wherein the object detection model represents the correspondence between an input image and a detection result corresponding to the target article object in the input image) to obtain an updated object detection model, thereby providing an apparatus that automatically generates training data and automatically updates the object detection model, thereby improving the convenience of updating the object detection model and the accuracy of the detection result.

[0096] 7, which shows a structural schematic diagram of a computer system 700 suitable for implementing the embodiments of the present disclosure (e.g., the devices 101, 102, 105, and 106 shown in FIG. 1). The device shown in FIG. 7 is merely an example and does not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0097] 7, computer system 700 includes a processor (e.g., CPU, central processing unit) 701 capable of performing various appropriate operations and processes according to programs stored in read only memory (ROM) 702 or programs loaded from storage unit 708 into random access memory (RAM) 703. RAM 603 further stores various programs and data necessary for the operation of system 700. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0098] An input unit 706 including a keyboard, a mouse, etc., an output unit 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc., a storage unit 708 including a hard disk, etc., and a communication unit 709 including a network interface card such as a LAN card and a modem are connected to an I / O interface 705. The communication unit 709 executes communication processing via a network such as the Internet. A driver 710 is connected to the I / O interface 705 as necessary. Removable media 711 such as a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory are installed in a drive 710 as necessary so that a computer program read from the removable media 711 is installed in the storage unit 708 as necessary.

[0099] In particular, according to embodiments of the present disclosure, the processes described with reference to the flowcharts above may be implemented as a computer software program. For example, embodiments of the present disclosure may be implemented as a computer program embodied in a computer readable medium. of The computer program includes a program code for executing the method shown in the flowchart. In this embodiment, the computer program can be downloaded and installed from a network via the communication unit 709 and / or can be installed from a removable medium 711. When the computer program is executed by the processor 701, it performs the above-mentioned functions defined in the method of the present disclosure.

[0100] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of both. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by, or that can be embedded in, an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium. The computer-readable medium may transmit, propagate, or transmit a program used by or embedded in an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, wireless, wired, optical cable, RF, or the like, or any suitable combination thereof.

[0101] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages, or combinations thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, and traditional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via an Internet connection service by an Internet service provider).

[0102] The flowcharts and block diagrams in the drawings illustrate apparatus, methods and computer program products according to various embodiments of the present disclosure. ToThe diagrams illustrate the system architecture, functions, and operations that can be realized by the above-mentioned. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code. The module, program segment, or portion of code includes one or more executable instructions for implementing a certain logic function. It should be noted that in some optional embodiments, the functions shown in the blocks may be executed in a different order from that shown in the drawings. For example, two blocks shown in succession may actually be executed substantially in parallel or sometimes in the reverse order depending on the function involved. It should be further noted that all blocks in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that executes a certain function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0103] The units described in the embodiments of the present disclosure may be implemented in software or hardware. The described units may be provided in a processor, and may be described as "a processor including a construction unit, an image generation unit, a sample generation unit, and an update unit." Here, the names of these units are not intended to limit the units themselves in some cases, and for example, the update unit may be described as "a unit configured to train an object detection model using a training sample set to obtain an updated object detection model."

[0104] Meanwhile, the present disclosure further provides a computer-readable medium, which may be included in the device described in the above embodiment, or may exist separately from the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, causes the computer to execute the following steps: constructing a three-dimensional model of a target object based on image data of the target object at multiple angles; generating a synthetic image including a target object object to represent the target object based on the three-dimensional model; taking the synthetic image as a sample image and the target object object as a label, acquiring training samples, generating a training sample set; and training an object detection model using the training sample set to obtain an updated object detection model. Here, the object detection model represents a correspondence relationship between an input image and a detection result corresponding to a target object object in the input image.

[0105] The above description is merely a description of the preferred embodiments of the present disclosure and the technical principles applied thereto. Those skilled in the art should understand that the scope of the present disclosure is not limited to the technical solution consisting of a specific combination of the above technical features, but should also include other technical solutions consisting of any combination of the above technical features or their equivalent features within the scope of the above disclosure. For example, the above features and technical features having similar functions disclosed in the present disclosure (but not limited thereto) may be substituted for each other.

Claims

1. 1. A method for updating an object detection model, comprising: constructing a three-dimensional model of the target object based on image data of the target object at a plurality of angles; generating a composite image including a target item object for representing the target item based on the three-dimensional model, the composite image including the target item object for representing the target item includes: determining a correspondence relationship of coordinate systems between a second image collection device corresponding to a picking robot, a preset information collection position, and a first image collection device for collecting image data of the target item; determining an adjusted three-dimensional model of the three-dimensional model from the viewpoint of the second image collection device based on the correspondence relationship of coordinate systems; and generating a composite image including a target item object for representing the target item based on the adjusted three-dimensional model and a preset background image representing a picking scene of the picking robot; The synthetic image is used as a sample image, the target object is used as a label, and a training sample is obtained to generate a training sample set; training the object detection model using the training sample set to obtain an updated object detection model; the object detection model represents a correspondence between an input image and a detection result corresponding to a target object in the input image; How to update the object detection model.

2. constructing a three-dimensional model of the target object based on image data of the target object at a plurality of angles, In a process of controlling a picking robot to perform a picking task of the target item based on a detection result of a target item object in an input image by the target detection model, a first image collection device collects two-dimensional image data and three-dimensional image data of the target item at a plurality of angles at a preset information collection position; and constructing a three-dimensional model of the target article based on the two-dimensional image data and the three-dimensional image data.

3. determining a weight of the target item; A step of generating a synthetic image including a target item object for representing the target item based on the adjusted three-dimensional model and a preset background image representing a picking scene of the picking robot, and generating a synthetic image including a target article object to represent the target article based on the adjusted three-dimensional model, the preset background image, the weights, a preset resolution of the synthetic image, and parameters of the second image capture device.

4. Training the object detection model using the training sample set to obtain an updated object detection model includes: The method of claim 1 , further comprising: in response to determining that the detection accuracy of the object detection model is lower than a predetermined threshold, training the object detection model using the training sample set by a machine learning algorithm to obtain an updated object detection model.

5. performing object detection on a subsequent input image using the updated object detection model to obtain a detection result; controlling a picking robot to perform a picking task based on the detection result; The method of any one of claims 1 to 4, further comprising:

6. 1. An apparatus for updating an object detection model, comprising: a construction unit configured to construct a three-dimensional model of the target object based on image data of the target object at a plurality of angles; an image generating unit configured to generate a composite image including a target item object for representing the target item based on the three-dimensional model, the image generating unit determining a correspondence relationship of a coordinate system between a second image collecting device corresponding to a picking robot, a preset information collecting position, and a first image collecting device for collecting image data of the target item, determining an adjusted three-dimensional model of the three-dimensional model from a viewpoint of the second image collecting device based on the correspondence relationship of the coordinate system, and generating a composite image including a target item object for representing the target item based on the adjusted three-dimensional model and a preset background image representing a picking scene of the picking robot; a sample generation unit configured to obtain training samples using the synthetic image as a sample image and the target object as a label, and generate a training sample set; an updating unit configured to train the object detection model using the training sample set to obtain an updated object detection model; the object detection model represents a correspondence between an input image and a detection result corresponding to a target object in the input image; An apparatus for updating an object detection model.

7. The building blocks further include In a process of controlling a picking robot to perform a picking task of the target item based on a detection result of the target item object in the input image by the target detection model, a first image collection device collects two-dimensional image data and three-dimensional image data of the target item at a plurality of angles at a preset information collection position; configured to construct a three-dimensional model of the target object based on the two-dimensional image data and the three-dimensional image data.

7. The apparatus of claim 6.

8. and a determination unit configured to determine a weight of the target item; The image generating unit further comprises: generating a composite image including a target object object to represent the target object based on the adjusted three-dimensional model, the preset background image, the weights, a preset resolution of the composite image, and parameters of the second image capture device.

8. The apparatus of claim 7.

9. The update unit further comprises: In response to determining that the detection accuracy of the object detection model is lower than a preset threshold, the object detection model is trained using the training sample set by a machine learning algorithm to obtain an updated object detection model.

7. The apparatus of claim 6.

10. an acquisition unit configured to perform object detection on a subsequent input image by the updated object detection model to obtain a detection result; an execution unit configured to control the picking robot to execute a picking task based on the detection result; The apparatus of any one of claims 6 to 9, further comprising:

11. A computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 4.

12. one or more processors; An electronic device comprising: a storage device in which one or more programs are stored; An electronic device, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method according to any one of claims 1 to 4.

13. A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images