Model training method and device, bearing detection method and device and electronic equipment

By using a synthetic image training sample method, the problem of insufficient generalization and robustness of detection models in robot delivery scenarios is solved, and the number and diversity of training samples are expanded, thereby improving the detection accuracy and adaptability of the model.

CN119048862APending Publication Date: 2024-11-29KEENON ROBOTICS CO LTD +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411159504.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In robot delivery scenarios, the types of delivery objects are limited, resulting in a limited number of images available for training detection models, making it difficult to guarantee the generalization and robustness of the detection models.

Method used

The training sample size is increased by synthesizing images. A first image of the robot carrying device is obtained, and a second image containing the target object is obtained from other means. The target object is segmented to obtain a local image, which is then fused with the first image to generate a synthetic image. This synthetic image is used as a training sample to train the carrying detection model.

Benefits of technology

This increased the quantity and diversity of training samples, avoided model overfitting, and enhanced the generalization and robustness of the load detection model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048862A_ABST
    Figure CN119048862A_ABST
Patent Text Reader

Abstract

The invention discloses a model training method, a bearing detection method, a model training device, a bearing detection device, electronic equipment and a computer readable storage medium. The training method comprises the following steps: acquiring a first image of a bearing device comprising a robot; the view angle of the first image comprises the view angle of the robot; segmenting the target object from a pre-acquired second image containing the target object to obtain a local image; fusing the local image with a specified image in the first image to obtain a composite image; the specified image is a first image which does not contain the target object; taking the composite image and the first image as training samples; and training a pre-constructed bearing detection model through the training sample to obtain a trained bearing detection model. According to the training method, the number of the training samples is increased by synthesizing the images, the types of the target objects in the training samples are enriched, and the generalization and robustness of the detection model can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a model training method, a load detection method, a model training device, a load detection device, an electronic device and a computer-readable storage medium. Background Art

[0002] During use, the robot can recognize the collected images through the trained detection model to improve the degree of automation and user experience. Generally, the detection model requires a large number of relevant training samples to ensure the robustness of the detection model.

[0003] Taking the robot delivery scenario as an example, there is currently a problem of a single delivery object category in the delivery scenario, resulting in limited images that can be used to train the detection model. For example, the food of a single restaurant is generally within the scope of its cuisine, and the food categories are relatively small, so the number of images that the restaurant can collect is small and the categories are single. It is difficult to ensure the generalization and robustness of the detection model by training the detection model with such training samples. Summary of the invention

[0004] The present application provides a model training method, a load detection method, a model training device, a load detection device, an electronic device and a computer-readable storage medium. By synthesizing images to expand the number of training samples and enrich the categories of target objects in the training samples, the generalization and robustness of the detection model can be effectively improved.

[0005] In a first aspect, the present application provides a model training method, the training method comprising: acquiring a first image including a carrying device of a robot; the perspective of the first image includes the perspective of the robot; segmenting a target object from a pre-acquired second image including the target object to obtain a partial image; fusing the partial image with a designated image in the first image to obtain a composite image; the designated image is the first image not including the target object; and using the composite image and the first image as training samples;

[0006] The pre-built load detection model is trained through training samples to obtain a trained load detection model.

[0007] In a second aspect, the present application provides a load-bearing detection method, which is applied to a robot, wherein the robot is equipped with a load-bearing device for carrying a target object; the load-bearing detection method comprises: acquiring a third image including the load-bearing device; detecting the target object in the third image based on a trained load-bearing detection model to obtain a detection result; controlling the operation of the robot based on the detection result and the current task status of the robot; wherein the load-bearing detection model is obtained using a training method of any model in the first aspect.

[0008] In a third aspect, the present application provides a model training device, comprising: an acquisition module for acquiring a first image of a carrying device including a robot; the perspective of the first image includes the perspective of the robot; a segmentation module for segmenting a target object from a pre-acquired second image containing the target object to obtain a local image; a fusion module for fusing the local image with a designated image in the first image to obtain a composite image; the designated image is the first image that does not contain the target object; a determination module for using the composite image and the first image as training samples; and a training module for training a pre-constructed carrying detection model through training samples to obtain a trained carrying detection model.

[0009] In a fourth aspect, the present application provides a load-bearing detection device, which is applied to a robot, wherein the robot is equipped with a load-bearing device that carries a target object; the load-bearing detection device includes: an acquisition module, which is used to acquire a third image including the load-bearing device; a detection module, which is used to detect the target object in the third image based on a trained load-bearing detection model to obtain a detection result; and a control module, which is used to control the operation of the robot based on the detection result and the current task status of the robot; wherein the load-bearing detection model is obtained using the steps of the method of the first aspect.

[0010] In a fifth aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0011] In a sixth aspect, the present application provides a robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the second aspect when executing the computer program.

[0012] In a seventh aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method of the first aspect or the second aspect are implemented.

[0013] In an eighth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by one or more processors, the steps of the method of the first aspect or the second aspect are implemented.

[0014] Compared with the prior art, the present application has the following advantages: the number of training samples is expanded by synthesizing images, and the pre-built load detection model is trained with sufficient training samples, thereby improving the robustness of the trained load detection model.

[0015] Specifically, in order to make the synthetic image close to the image collected by the robot, a first image including the robot's carrying device in the real scene can be obtained; wherein the perspective of the first image can include the perspective of the robot. At the same time, in order to avoid the single category of the target object to be detected in the training sample, the second image containing the target object can be obtained through other means to improve the diversity of the target object. On this basis, the target object is segmented from the second image to obtain a local image containing the target object. By fusing the local image with the specified image in the first image, a large number of synthetic images rich in target objects can be obtained; using the synthetic image and the first image as training samples can not only increase the number of training samples, but also increase the diversity of training samples. Training the carrying detection model with such training samples can avoid overfitting of the carrying detection model and improve the generalization and robustness of the carrying detection model.

[0016] It can be understood that the beneficial effects of the second to fifth aspects can be found in the relevant description of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of the training method of the model provided in the embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the structure of a salient object detection model provided in an embodiment of the present application;

[0020] Figure 3 is a flow chart of a method for generating a composite image provided in an embodiment of the present application;

[0021] Figure 4 is a comparison chart of the effects of different data set generation methods provided in the embodiments of the present application;

[0022] Figure 5 It is a target detection result diagram of a word prompt generation data set provided in an embodiment of the present application;

[0023] Figure 6 is a target detection result diagram of a phrase prompt generation data set provided in an embodiment of the present application;

[0024] Figure 7 It is a diagram of the ablation experiment results of Yolov5 provided in the embodiment of the present application;

[0025] Figure 8 It is a structural schematic diagram of a load-bearing detection model provided in an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of the structure of the robot provided in an embodiment of the present application;

[0027] Fig.10 It is a flowchart of the load detection method provided in the embodiment of the present application;

[0028] Fig.11 It is a structural schematic diagram of a training device for a model provided in an embodiment of the present application;

[0029] Fig.12 is a schematic diagram of the structure of a load-bearing detection device provided in an embodiment of the present application;

[0030] Fig.13 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] In the related technology, during the use of the robot, the collected images can be recognized through the trained detection model to improve the degree of automation and user experience. Generally, the detection model requires a large number of relevant training samples to ensure the robustness of the detection model.

[0033] Taking the robot delivery scenario as an example, there is currently a problem of a single delivery object category in the delivery scenario, resulting in limited images that can be used to train the detection model. For example, the food of a single restaurant is generally within the scope of its cuisine, and the food categories are relatively small, so the number of images that the restaurant can collect is small and the categories are single. It is difficult to ensure the generalization and robustness of the detection model by training the detection model with such training samples.

[0034] Through research, this application found that when the generalization and robustness of the detection model are low, it is difficult for the robot to improve the degree of automation and user experience through image recognition.

[0035] In order to solve this problem, the present application proposes a model training method, which expands the number of training samples by synthesizing images and enriches the categories of training samples to improve the generalization and robustness of the detection model. The training method proposed in the present application will be described below through specific embodiments.

[0036] The model training method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0037] In order to illustrate the technical solution proposed in this application, each embodiment will be described below with an electronic device as the execution subject.

[0038] Figure 1 A schematic flow chart of a training method for a model provided in the present application is shown, and the training method for the model includes:

[0039] Step 110: The electronic device obtains a first image of a carrier device including the robot.

[0040] During use, the robot generally carries the target object through a carrying device to complete the corresponding task. For example, in the scenario of robot delivery of food, the robot can carry food or empty plates through a tray to complete the food delivery task or empty plate recovery task; in the scenario of robot delivery of infusion bags, the robot can carry the infusion bags through the delivery cabin to complete the infusion bag delivery task.

[0041] That is to say, the robot can be configured with different load-bearing devices in different application scenarios. For example, if the robot is used in a restaurant, the load-bearing device configured therein may be a tray or an item basket; if the robot is used in a hospital, the load-bearing device configured therein may include a tray (surgical instruments), an item bin (medicines, infusion bags, etc.), an item basket (bedding), etc. The specific load-bearing device can be configured according to the scenario in which the robot is used, and is not limited in this application.

[0042] The target object is similar to the carrier, and is closely related to the application scenario of the robot. The target object can be items that need to be delivered or recycled in the corresponding application scenario. For example, when the robot is used in a restaurant, the target objects may include empty plates, food, bottles, drinking cups, empty dishes, kettles, napkins, seasonings, drinks, straws, bags, tableware, tickets, towels, etc. When the robot is used in a hospital, the target objects may include surgical instruments, drugs, test samples, clothing and blankets, infusion bags, etc.

[0043] The image captured by the robot has a relatively fixed capture angle and range, and is mainly aimed at the area where its carrying device is located. Based on this, in order to enable the carrying detection model to accurately identify the image captured by the robot, the electronic device can obtain a first image including the robot's carrying device as a training sample for the carrying detection model. It can be understood that the first image at least includes the carrying device. Exemplarily, the first image may include only an empty carrying device, that is, a carrying device that does not carry any target object; it may also include a carrying device that carries each target object.

[0044] The first image collected in the real scene is close to the image collected by the robot. By training the load-bearing detection model with such training samples, the accuracy of the load-bearing detection model in image recognition can be improved.

[0045] The perspective of the first image may include the perspective of the robot, so that the first image can be closer to the image captured by the robot. The first image may be a single image captured by a camera, or a partial image in multiple frames of images, which may be determined according to actual needs.

[0046] The image acquisition device configured for the robot can acquire images including the load-bearing device. During operation, the load-bearing detection model configured for the robot can be used to detect the target object on the load-bearing device in the image. Based on this, preferably, the acquisition device for the first image includes an acquisition device for acquiring images of the load-bearing device during the operation of the robot, which can make the first image closer to the image detected by the load-bearing detection model. By training the load-bearing detection model with the first image, the accuracy of the model can be improved. In addition, the acquisition device for the first image can also include an external camera, which can acquire images including the load-bearing device from different positions, so as to improve the generalization of the model and facilitate the expansion of the use of the load-bearing detection model on different robots.

[0047] Step 120: The electronic device segments the target object from a pre-acquired second image containing the target object to obtain a partial image.

[0048] Step 130: The electronic device fuses the local image with the designated image in the first image to obtain a composite image.

[0049] In practical applications, training the load detection model only by relying on the first image may result in poor generalization and robustness of the trained load detection model due to the small number of first images and / or the single category of the target object to be identified included in the first image.

[0050] In order to solve this problem, the generalization and robustness of the load detection model obtained by training are improved. The electronic device can efficiently generate a large number of synthetic images based on the first image and the available second image including the target object to expand the number of training samples. The synthetic image uses the first image actually acquired as the background image, and integrates the target object segmented from the second image into the background image, so that the synthetic image has real details and is close to the first image actually acquired, and can ensure the quality of the training samples while expanding the training samples.

[0051] The second image including the target object can be obtained through a variety of ways. For example, an open source image dataset may also include an image of the carrier and the target object taken by a camera configured by the robot, or an image obtained by taking a single shot of the target object. In a delivery scenario, the second image may include pictures of common items such as food, tableware, receipts, and paper towels. In a restaurant scenario, the second image may include food with a plate (the food may occupy part or all of the tray), or may include simply packaged food, such as French fries, hamburgers, etc.

[0052] After obtaining the second image, the electronic device can segment the target object from the second image to obtain a partial image. The partial image is fused with the first image as the background image to obtain a composite image. The first image, also known as the designated image, is a first image that does not contain the target object; in other words, the designated image is a first image that includes an empty carrier. Such a first image can provide real scene details for the recognition of the target object, which helps to improve the authenticity of the composite image, thereby improving the quality of the training sample.

[0053] Exemplarily, the electronic device can be based on a random sampling method of probability distribution, such as a random sampling method of Gaussian distribution, with a specified image as the background, and randomly change the local image to fuse a new image, namely a synthetic image. Through this method, the position of the target object on the carrier can be controlled by a random sampling method to simulate the diversity of the placement of the target object in the real scene. This not only increases the naturalness and diversity of the synthetic image, but also provides richer and more realistic training samples for the carrier detection model, which helps to improve the recognition and generalization capabilities of the carrier detection model for different target object placement situations. Through the method of synthetic images, a large number of sample images can be efficiently generated according to the needs of the target scene, reducing the pressure and difficulty of image acquisition.

[0054] Step 140: The electronic device uses the synthesized image and the first image as training samples.

[0055] Step 150: The electronic device trains the pre-built load detection model using training samples to obtain a trained load detection model.

[0056] After synthesizing a sufficient number of synthetic images, the synthetic images can be mixed with the first image as training samples. Compared with the training samples that only include the first image, the training samples have two advantages: on the one hand, the number of the training samples is sufficient, which can avoid the under-fitting of the trained load detection model caused by insufficient number of training samples; on the other hand, the training samples contain target objects of various categories, which can avoid over-fitting of the load detection model. In other words, the training samples obtained by this embodiment can improve the robustness of the load detection model by training the load detection model.

[0057] In this embodiment, the electronic device uses the designated image in the first image actually acquired as the background, and segments the second image obtained by other means to obtain a local image corresponding to the target object with rich categories, and by fusing the background image and the local image, a synthetic image that retains the real details required for training can be obtained. Using the synthetic image and the first image as training samples can expand the number of training samples and enrich the categories of target objects included in the training samples. Training the load detection model with the training samples can improve the generalization and robustness of the load detection model.

[0058] In some embodiments, it can be seen from the above embodiments that the categories of target objects obtained by other means are rich and varied. If the current mainstream semantic or instance segmentation method is used, that is, the local image corresponding to the target object is segmented from the second image based on a fixed category, the segmentation effect is poor. Considering the characteristic that salient target detection is independent of the target category, the electronic device can use a saliency detection method to eliminate background interference and only leave information related to the target object, thereby segmenting the local image corresponding to the target object with rich and varied categories from each second image. Specifically, the above step 120 includes:

[0059] Step A1: The electronic device performs salient object detection on the second image to obtain a detection result.

[0060] In order to be able to identify target objects of rich categories and segment and obtain a local image corresponding to the target object, the electronic device may perform salient target detection on the second image to obtain a corresponding detection result.

[0061] Among them, salient object detection algorithms can be divided into three categories: sparse detection methods, dense detection methods, and weakly supervised methods. Sparse detection methods detect through key points or feature points, which has the advantage of high computational efficiency, but may not perform well in complex backgrounds. Dense detection methods detect at each pixel or region, with high accuracy but high computational overhead. Weakly supervised methods use partial labeled data for training, saving labeling costs, but may not be as accurate as fully supervised methods. The three types of algorithms have their own advantages and disadvantages, and can be selected according to actual needs.

[0062] The saliency detection algorithm can be implemented by a corresponding model. For example, see Figure 2 , optional Figure 2 The two-stream asymmetric detection model shown is used to extract the salient target features and edge features of the target object in the second image. The two-stream asymmetric detection model includes a backbone network, a salient target network, an edge network, and a fusion network. Among them, except that the backbone network is an encoder network, the remaining networks are decoder networks.

[0063] Specifically, the backbone network can be ResNet-50. For the output of each remaining scale, two stacked convolution layers are used to convert the output of each scale into a feature map of 64 channels; for the salient target decoding stream, the global guidance feature can automatically adapt to the features of each scale and guide the feature fusion; finally, a 3×3 single-layer convolution is used to merge the features into a single-channel salient target feature map. Since the edge of the salient target includes fine edge features and position information, if the global guidance feature is directly used to guide the shallow features to generate edge features, the number of parameters will be greatly reduced. Therefore, the edge stream only contains the deepest and shallowest features. Finally, a 3×3 single-layer convolution is used to merge the features into a single-channel d target edge feature map. After obtaining the complementary feature maps (salient target feature map and target edge feature map), the salient target feature map and the target edge feature map are aggregated under the guidance of the global guidance feature in the fusion stage to obtain the detection result.

[0064] Step A2: The electronic device segments the target object in the second image based on the detection result to obtain a partial image.

[0065] After obtaining the detection result, the electronic device can segment the second image to obtain a partial image based on the detection result.

[0066] In the embodiment of the present application, the electronic device can accurately identify the target objects with rich categories in the second image through the salient target detection algorithm and obtain the corresponding detection result; based on the detection result, the electronic device can segment the second image to obtain the local image corresponding to the target object. This makes it easier for the model to be better applied to different usage scenarios, such as restaurant delivery, restaurant recycling, hotel delivery, and medical delivery.

[0067] In some embodiments, the edge accuracy of the target object in the local image obtained by segmentation through the salient target detection algorithm may be low, which is not conducive to the accurate identification of the target object by the detection model through the edge information of the target object. Based on this, the local image obtained in the above embodiment can be recorded as the initial local image, and the initial local image is optimized by the dense conditional random field (Dense CRF) model to obtain a local image with higher segmentation accuracy. Therefore, after the above step A2, it also includes:

[0068] Step A3: The electronic device optimizes the initial local image through a dense conditional random field model to obtain a local image.

[0069] The optimization of the initial local image by the dense conditional random field model is a post-processing of the segmented image, which can make the edge of the optimized local image finer, so that the detection model can obtain richer and more detailed edge information. Specifically, the dense conditional random field model can implement multiple rounds of iterations through the energy field function, thereby continuously correcting the local detail information in the initial local image.

[0070] In this embodiment, the electronic device first detects and segments target objects with rich categories in the second image through a significant target detection algorithm to obtain an initial local image; in order to improve the segmentation accuracy of the local image, the electronic device optimizes the initial local image through a dense conditional random field model to obtain a local image with higher segmentation accuracy.

[0071] In some embodiments, in order to improve the optimization effect of the dense conditional random field model on the dense conditional random field model, the aforementioned step A3 specifically includes:

[0072] Step A31: The electronic device performs a feature extraction operation on the initial local image to obtain a feature extraction result.

[0073] The electronic device may perform feature extraction on the initial partial image, such as extracting features such as color or position, to obtain a feature extraction result. The feature extraction result may be used to calculate the similarity between each pixel in each initial partial image, so as to generate a fully connected graph.

[0074] Step A32: The electronic device generates a fully connected graph based on all pixels of the initial local image.

[0075] Each pixel in the initial local image can be used as a node, and an edge can be introduced between each pixel. The weight of the edge is determined by the similarity between the pixels, that is, the similarity can be further calculated through the feature extraction result to determine the edge between every two pixels. In this way, the electronic device can generate a fully connected graph based on all the pixels in the initial local image.

[0076] Step A33: The electronic device determines an energy function of the fully connected graph based on the feature extraction result and the local image.

[0077] The energy function mainly consists of two parts, namely, unilateral potential (Unary Potentials) and potential (Pairwise Potentials). Among them, the unilateral potential is used to describe the cost of each pixel belonging to a certain label, which can be determined by the feature extraction results of the aforementioned embodiment. The bilateral potential is used to describe the cost of interaction between pixels and encourages similar pixels to have the same label. The bilateral potential can be determined by calculating the similarity between pixels using a Gaussian kernel.

[0078] Exemplarily, the energy function E(z) can be written as follows:

[0079]

[0080] Among them, X is the initial local image, is the unilateral potential of pixel i in the initial local image; is the bilateral potential between pixel i and pixel j in the initial local image.

[0081] Specifically, It can be specifically expressed as:

[0082]

[0083] Among them, the kernel function k is used to calculate the weighted similarity and difference between different pixels, which can be specifically expressed as:

[0084]

[0085] Among them, p i and p j are the two-dimensional coordinate information of pixel i and pixel j respectively; i and I j Represented as the pixel value of pixel i and pixel j; θ α and θ β represents the control pixel distance, θ γ Indicates control contrast.

[0086] Step A34: The electronic device iteratively optimizes the energy function until the energy function converges.

[0087] Step A35: The electronic device optimizes the initial local image based on the converged energy function to obtain a local image.

[0088] After obtaining the energy function, the electronic device can optimize the energy function so that the label distribution probability corresponding to each pixel point approaches the global optimal solution and the energy function converges.

[0089] Exemplarily, the electronic device can iteratively optimize the energy function efficiently through a message passing algorithm. The message passing algorithm may include mean field approximation (Mean Field Approximation), loopy belief propagation (LBP), tree re-parameterization algorithm (Tree-Reweighted Message Passing, TRW), expectation propagation (Expectation Propagation, EP), etc.

[0090] For example, the mean field approximation has the advantages of high computational efficiency, strong scalability, good convergence, simple implementation and wide applicability, making it excellent in processing large-scale images and high-dimensional data, and widely used in a variety of practical applications. By simplifying the original complex probability distribution, the mean field approximation can be calculated quickly and provide reliable results, which is particularly suitable for dense conditional random field optimization problems in the fields of image processing and computer vision.

[0091] After several iterations of optimization, a converged energy function can be obtained. At this time, the pixel label of each pixel in the initial image is the maximum probability label, which can accurately segment the edge of the target object.

[0092] In this embodiment, the dense conditional random field model introduces global interactions between pixels in a fully connected graph, and the energy function of the fully connected graph can be determined based on the feature extraction results and global interactions of the aforementioned embodiment; the electronic device iteratively optimizes the energy function so that the energy function converges, and can perform more accurate image segmentation on the initial local image. That is, based on the consideration of the similarity between pixels, the smoothness and consistency of the segmentation result can be improved, and a local image with higher segmentation accuracy can be obtained.

[0093] In some embodiments, see Figure 3 , Figure 3 A flow chart of a method for generating a composite image is shown, wherein the target object is food. After the electronic device performs saliency detection on a second image containing food, a corresponding detection result can be obtained. Based on the detection result, the electronic device can segment the second image to obtain a partial image corresponding to the food, and fuse the partial image with a specified image to obtain a composite image containing food.

[0094] It can be understood that the food in the first image can be rich and varied, for example, it can be food corresponding to various cuisines such as Shandong cuisine, Hunan cuisine, and Cantonese cuisine, and it can also be food corresponding to different foreign dining styles such as Western cuisine, Korean cuisine, and Vietnamese cuisine. The synthetic image obtained based on such a first image can ensure the richness of the sample; by testing the load detection model through the training sample, the generalization and robustness of the load detection model can be improved.

[0095] In some embodiments, the food is used as the target object and the tray is used as the carrier, and the corresponding synthetic image can be called tray data. There are many methods for generating tray data sets. The current mainstream data generation methods mainly include Food2k, SAM and diffusion model. Among them, Food2k is a detection model proposed for large-scale food data; the diffusion model can synthesize close to real detailed images based on text prompts.

[0096] The pallet data generation method described in the above embodiment of the present application is compared with the current mainstream data set generation method. The effects of each method can be found in Figure 4 . See Figure 4 It can be seen that in the task of generating tray data, the user experience of the method is mainly evaluated from three dimensions: degree of automation, time efficiency, and versatility. First, in terms of the degree of automation, the classification method of Food2k can automatically select the required dishes, while the methods of SAM and diffusion models require manual input prompts. The generation method of this application can automatically separate dishes from the background area, realizing fully automated processing. Secondly, in terms of time efficiency, the discriminant method is less time-consuming, while the interactive method usually requires a longer time for interaction and algorithm iteration. Finally, in terms of versatility, except for Food2k, which relies on image classification, other methods are not limited to the training of data in specific fields and have better versatility. On the whole, the tray data generation method of this application has great advantages in terms of ease of use and generation efficiency.

[0097] In some embodiments, the labeling of training samples is an important means of introducing prior knowledge, providing the necessary supervisory signals for the model, so that the model can improve performance by learning and optimizing parameters, and perform better in actual tasks. However, the labeling of training samples is currently mostly done manually, which is time-consuming and labor-intensive, and the labeling efficiency is low. In order to solve this problem, the electronic device can automatically label the unlabeled synthetic images, specifically including:

[0098] Step B1: The electronic device extracts features of the composite image and the guide text corresponding to the composite image to obtain text features and image features.

[0099] In order to efficiently and accurately select and annotate the target object in the composite image, a corresponding guide text may be configured for each composite image, so that the target object can be selected from the composite image based on the instructions of the guide text and its category can be annotated.

[0100] In order to facilitate processing by the electronic device, the electronic device may first extract features of the composite image and the guide text, and then obtain corresponding text features and image features.

[0101] Exemplarily, the electronic device may use a convolutional neural network (CNN) to extract image features such as edges, textures, and color distribution from a synthetic image; and use natural language processing (NLP) techniques such as word embedding, sentence embedding, or a pre-trained language model (such as BERT) to extract text features reflecting the content and meaning of a guide text.

[0102] Step B2: The electronic device fuses the text feature with the image feature to obtain a fused feature.

[0103] In order to locate the target object related to the text feature in the image feature, the electronic device can effectively combine the image feature with the text feature to obtain a fusion feature.

[0104] Exemplarily, the electronic device may input the image features and text features into a feature enhancement module including a plurality of feature enhancers. The feature enhancement module includes a deformable self-attention mechanism and a common self-attention mechanism, wherein the deformable self-attention mechanism may be used to enhance the image features, and the common attention mechanism may be used to enhance the text features. Then, the cross-modal attention calculations from image to text and from text to image are sequentially implemented through the cross-attention mechanism to obtain enhanced image features and text features, and the two features are fused to obtain a more expressive fusion feature.

[0105] Step B3: The electronic device generates a query vector based on the text features.

[0106] Step B4: The electronic device performs bounding box regression and classification on the fused features based on the query vector to mark the box and category of the target object in the synthesized image.

[0107] In order to select and annotate the corresponding target object in the image under the guidance of the text, the electronic device can use the text feature as a query condition, that is, generate a corresponding query vector. Thus, the electronic device can select and annotate the fused features through the query vector, that is, perform bounding box regression and classification, and thus annotate the annotation box and category of the target object in the synthesized image.

[0108] Among them, bounding box regression is to find the relevant area in the fused features according to the query vector and accurately locate the boundary of the target object; classification is to identify the category of the target according to the query vector.

[0109] Step B4: The electronic device determines the annotated composite image and the first image as training samples.

[0110] The annotated composite image can be used together with the annotated first image as training samples with prior knowledge for training the load detection model.

[0111] In this embodiment, the electronic device performs text enhancement and fusion on the composite image and the guide text through different attention mechanisms to obtain a fused image, and then uses the query vector corresponding to the text feature as a query guide to select and annotate the fused features respectively, so as to annotate the annotation box and category of the target object in the composite image, thereby automatically annotating the composite image efficiently. Finally, the annotated composite image and the first image are used as training samples to further improve the training effect of the model.

[0112] For example, the automatic annotation model uses a pre-trained grounding DINO model. This automated process significantly improves the efficiency of processing massive data and greatly reduces the cost of the data preparation stage. Since automatic annotation can ensure the consistency and quality of the data set and reduce subjective bias and inconsistency in manual annotation, this method not only enhances the reliability of data processing, but also improves the reproducibility of the annotation process.

[0113] In some embodiments, see Figure 5 , Figure 5 The figure shows the performance of the model obtained from different training sets on a new dataset generated by the grounding DINO model using words as prompts. Figure 5 It can be seen that the grounding DINO model trained on the original dataset has a low test multi-category average accuracy (mAP) of 49.5%, while the grounding DINO model trained on the augmented dataset with words and phrases as prompts has an increase in the test multi-category average accuracy, the former is 61.2%, and the latter is 85.8%. Obviously, using the data augmented dataset obtained by the prompt words to train the grounding DINO model can effectively improve the performance of the model.

[0114] See also Figure 6 , Figure 6 The figure shows the performance of the models obtained by different training sets on a new dataset generated by the grounding DINO model using phrases as prompt words. The distribution of the average accuracy of the three models in multiple categories is similar to Figure 5The above experimental results show that the model trained only on the original dataset has unsatisfactory generalization performance on the augmented dataset, while the model trained on the augmented dataset performs better, and the model trained on the +grounding DINO_phrase dataset has higher multi-category accuracy than the model trained on the +grounding DINO_word dataset.

[0115] In order to avoid the above test results being affected by underfitting, overfitting and other situations of the model, this embodiment respectively calculates the changes in category loss, confidence loss, and prediction box loss of the above three models during their respective training processes as the training rounds progress. The results all show that the models have converged during the training process, and through the comparison of the three groups of curves, it is found that the third group of curves becomes stable after 60 rounds of training, while the first two groups of curves still have an obvious downward trend after 60 rounds. Therefore, it can be concluded that the model trained under the +grounding DINO_phrase dataset not only shows the highest multi-category accuracy in the test phase, but also has a faster convergence speed in the training phase.

[0116] In some embodiments, for a converged load model, in order to comprehensively and accurately measure the performance of the load detection model, the load detection model can be verified on a validation set. Specifically, traditional target detection indicators can be used to evaluate the performance of the load detection model. The indicators include precision (Precision, p), recall (Recall, R) and meanAveragePrecision (mAP).

[0117] The accuracy rate is also called the precision rate, which is used to measure the accuracy of the load detection model in predicting positive samples, that is, the proportion of positive samples predicted by the load detection model that are actually positive. The formula is as follows:

[0118]

[0119] Among them, TP (True Positive) is the true class, that is, the number of samples correctly predicted as the positive class; FP (False Positive) is the false positive class, that is, the number of samples incorrectly predicted as the positive class.

[0120] Recall rate is also called sensitivity or true positive rate, which is used to measure the ability of the load detection model to correctly identify positive samples, that is, the proportion of samples that are actually positive that are correctly predicted as positive. The formula is as follows:

[0121]

[0122] Among them, FN (False Negative) is the false negative class, that is, the number of samples that are mistakenly predicted as negative classes.

[0123] The average precision combines the accuracy and recall rate. By calculating the average precision of different categories to measure the overall performance of the model, it can more comprehensively reflect the performance of the model. The formula is as follows:

[0124]

[0125] Among them, AP i Represents the average accuracy of the i-th category detection, M i It represents the total number of samples of the i-th category, and mAP represents the average accuracy of all categories.

[0126] In some embodiments, the load detection model may include a backbone network (Back bone), a neck network (Neck), and a detection head (Prediction). The backbone network is used to extract features from the input image to obtain a feature image; the neck network is used to fuse the feature image to obtain a fused feature; and the detection head is used to detect the fused feature to obtain a detection result.

[0127] As for the model structure itself, this example conducts an ablation experiment on the original dataset to determine whether Yolov5 needs an attention module. The experimental results are shown in Figure 7 . The experimental results show that when only the Channel Block Attention Module (CBAM) module is introduced into the Yolov5 model structure, the mAP on the test set reaches 83.7%, which is better than Yolov5 with an mAP of 81.8% on the test set. When the SE module is introduced into the model structure, the model performance has declined to varying degrees. Therefore, the final model structure selects Yolov5+CBAM as the load detection model. The introduction of the CBAM module allows the load detection model to focus on the information of channels that are more relevant to the current task.

[0128] For example, the load detection model can be obtained by improving Yolov5. For the specific structure, see Figure 8. The load detection model includes a convolution (Conv) module, a CBAM module, a C3 module and an improved version of the Spatial Pyramid Pooling Fast (SPPF) module. The Conv module is the most basic module in the network structure, which specifically includes a convolution layer, a normalization layer and an activation function. The three work together on the input image to extract more abstract semantic features. The C3 module is used for the segmentation of intermediate features and semantic extraction at different scales. The input is divided into two parts, one of which is processed by the above-mentioned Conv module and the neck network for semantic feature extraction, and the other branch is only processed by the Conv module. The features of the two branches are then merged and output. The SPPF module is used to fuse feature maps under different receptive fields. After the input passes through the above-mentioned Conv module, it undergoes three maximum pooling operations in succession, and then the output after each pooling is merged.

[0129] Among them, the CBAM module will replace the C3 module in the backbone network. For example, for two-channel input features, each channel of the input feature can be processed separately by maximum pooling and average pooling, and then the two processed feature maps are respectively passed through the same fully connected network to obtain the relevance weights of each channel of different feature maps for the current task, and finally the two are summed and applied to each channel of the input feature, so that the model pays attention to the information of the channel that is more relevant to the current task.

[0130] In some embodiments, the total loss function of the load detection model can be obtained by weighted summation of the classification loss function, the positioning loss function, and the confidence loss function. In order to balance the distance between the classification label and the true value, the binary cross entropy can be used to measure the classification loss y i , which can be written as follows:

[0131]

[0132] In some embodiments, in order to improve the robustness of the load detection model, the first image can be collected manually or automatically. Fig. 9 A schematic diagram of the structure of a robot is shown. The robot is at least equipped with a carrying device and a camera. Based on this example, the first image can be collected by the camera of the robot.

[0133] In order to further improve the generalization of the load detection model, the first image in at least two scenes can be collected by the camera on the robot. The scene may include a basic scene and a feature scene. Taking restaurant delivery as an example, the feature scene includes shooting scenes with different lighting, shooting scenes with different shooting resolutions, shooting scenes with water vapor, shooting scenes with different angles and / or different degrees of shadows, shooting scenes with multiple target objects, and shooting scenes with large-size target objects. At least one of the following shooting scenes. The samples obtained can fully consider the different lighting conditions of restaurant scenes, the possibility that the robot is equipped with cameras of different resolutions, the impact of food steam in the restaurant on the camera's image acquisition, the possibility that the surrounding crowd may cause shadows to block the food, the size and shape of Chinese food and containers are varied, and the user may place one or more food to be delivered on each layer of the load-bearing device. These special scenes further improve the generalization of the model and the accuracy in the delivery scene.

[0134] In this embodiment, the first image collected in the feature scene belongs to the difficult sample in the training sample, which can not only enrich the scene conditions of various target recognition, but also improve the robustness and generalization ability of the model. These difficult samples usually include complex backgrounds, occlusions, target objects or large target objects under different lighting conditions, etc. By adding these samples to the training, the load detection model can learn to process the target object features in more complex and changeable scenes, and improve its recognition ability of target objects in actual application environments.

[0135] In some embodiments, when the robot is used in a smart restaurant environment, the detection task of the load detection model is tray detection, that is, to detect the target object in the tray. According to the specific requirements of the detection task, a tray dataset can be constructed from a mobile phone dataset in an actual restaurant scene. In order to effectively detect the plate, a load detection model based on Yolov5++CBAM is designed. Due to the limitation of the types of dishes in a single restaurant, the target objects (such as dishes, tableware, etc.) carried by the tray in the data set are single, especially the types of dishes are single. Based on this situation, this embodiment introduces a tray data synthesis method based on saliency detection, and uses its category-independent characteristics to extract diversified target object data from different scenes, effectively solving the problems of small training data volume and single target object type. In terms of annotation of the synthetic data set, this embodiment explores an effective prompt word method and uses grounding DINO technology to achieve high-quality automatic annotation. The experimental results show that the plate detection model proposed in this embodiment can effectively identify dishes on the plate in a smart restaurant environment, and the data synthesis method based on saliency detection significantly improves the quality of the data set and the performance of the plate detection model.

[0136] In some embodiments, when the robot is in use, if the accuracy of image detection is low, human assistance is required for the robot to successfully perform tasks, which reduces the intelligence of the robot.

[0137] In order to solve this problem, this embodiment proposes a load-bearing detection method based on the training method of the model of the above embodiment, which is applied to a robot. The robot is equipped with a load-bearing detection model, which is mainly trained by the training method of the model of the above embodiment.

[0138] See also Fig.10 , Fig.10 A schematic flow chart of a load detection method provided in an embodiment of the present application is shown, and the load detection method includes:

[0139] Step 101: The robot collects a third image including a carrying device.

[0140] Step 102: The robot detects the target object in the third image based on the trained load-bearing detection model to obtain a detection result.

[0141] In the process of performing the task, the robot can collect an image to be identified including the load-bearing device, that is, the third image. The robot is configured with a trained load-bearing detection model, so that the robot can input the third image into the load-bearing detection model to obtain a detection result. The detection result is a detection result of the target object in the third image, for example, whether a certain target object is present or not.

[0142] Step 103: The robot controls the operation of the robot based on the detection result and the current task status of the robot.

[0143] Different detection results can reflect the current progress of task execution. Thanks to the load detection model trained by the aforementioned model training method, the detection result has a high accuracy. Therefore, the robot can control its own operation according to the detection results and current task status without human assistance, so as to improve the intelligence of the robot.

[0144] For example, take the robot delivering food in a restaurant scenario. In the relevant scenario, because the detection accuracy of the detection model configured by the robot is relatively low, after the robot obtains the result through the detection of the configured detection model, it needs to send out corresponding prompt information in combination with the detection result and the current task status to prompt the user to confirm whether to perform subsequent control on the robot. For example, when the robot receives a food delivery task, the robot can first move to the food pickup area to wait for food pickup. While waiting for food pickup, the robot can collect a third image through the camera for detection. When the detection result is food, the robot can determine the next operation based on the current task status of waiting for food pickup and the detection result of food pickup, that is, move from the food pickup area to the delivery point. However, due to inaccurate detection results, the robot will issue a prompt to let the user confirm whether to deliver the currently picked-up food to the delivery point. That is, the robot delivery at this time is semi-automatic delivery, and the intelligence level is relatively low.

[0145] However, if the robot is equipped with the load detection model trained by the above embodiment, a detection result with higher accuracy can be obtained. Therefore, the robot can directly perform subsequent movement control based on the current task state of waiting for food pickup and the detection result of the food, that is, move from the food pickup area to the delivery point to achieve delivery.

[0146] After moving to the delivery point, the robot can continue to collect the third image to determine whether the food has been taken away through detection. In some cases, the table corresponding to the delivery point is in the early stage of serving, and no empty plates have been generated. In this case, if the detection result obtained by the robot through the load detection model is that there is no target object, then the robot can determine that the current delivery task has been completed in combination with the task status of delivery, and can return directly or perform the next task; in other cases, the table corresponding to the delivery point is in the middle and late stages of serving, and empty plates may be generated. In this case, if the detection result obtained by the robot through the load detection model is that there is no target object, the robot can perform the aforementioned control operation on itself; but if the detection result is the target object of the empty plate, then the robot can determine that the current delivery task has been completed in combination with the task status of delivery, and has received a recycling task, and can directly perform the plate recycling task.

[0147] The target object may be different in different robot application scenarios and different task states in the same scenario. For example, during the model training process, the detection results may be classified, and when applied, the corresponding category may be determined as the target object in combination with the application scenario and task state.

[0148] For example, in a restaurant scenario, the model detection results can be classified and labeled as follows: 0-food; 1-dirty beverage cup; 2-dirty plate; 3-empty tray; 4-kettle; 5-receipt; 6-clean tableware. When the robot is performing a meal delivery task, the output detection result number 0 means that the target object exists, and the others mean that the target object does not exist; this can avoid the impact of the omission of the receipt and other situations on the detection results, and facilitate the robot to automatically confirm the completion of the meal, quickly execute the next task, and improve delivery efficiency. When the robot performs the tableware delivery task, the output detection result number 6 means that the target object exists, and the others mean that the target object does not exist; when the robot performs the recycling task, the output detection result numbers 3 and 2 mean that the target object exists. That is, the load detection model of the present application, through the classification of detection results, is convenient for the later combination of each category to be applied to different scene tasks, improves the generalization and accuracy of the model, and is also convenient for the expansion of the model application scenarios.

[0149] In some embodiments, the carried items may be snacks, and the carried items may be placed in a containing device, which is placed on the carrying device of the robot, and the carried items or the user's arm are detected as target objects. The robot can move along a fixed route in a cruising mode, and the user can pick up the carried items by himself and freely take the items. When the detection result of the camera detection shows that the user's arm or the carried items have changed, it can be determined that someone is currently taking the items. At this time, the robot stops moving, and when the camera detection result determines that the user's arm has disappeared, or the carried items no longer change, it is determined that the current picking is completed, and the robot continues to move to perform the cruising task. Through load detection, the robot can stop moving at the right time in the cruising mode and wait in place to pick up the items, which improves the user experience.

[0150] It is worth noting that the above examples are only for illustration and not limitation. For robots in hospital scenarios and other scenarios, the above examples can be used as analogies to improve the intelligence of different types of robots, so that various tasks in different scenarios can be completed efficiently.

[0151] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0152] Corresponding to the training method of the model in the above embodiment, Fig.11 A structural block diagram of a training device 11 for a model provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0153] Reference Fig.11, the training device 11 of the model includes: an acquisition module 111, used to acquire a first image of a carrying device including a robot; the perspective of the first image includes the perspective of the robot; a segmentation module 112, used to segment the target object from a pre-acquired second image containing the target object to obtain a local image;

[0154] The fusion module 113 is used to fuse the local image with the specified image in the first image to obtain a composite image; the specified image is the first image that does not contain the target object; the determination module 114 is used to use the composite image and the first image as training samples; the training module 115 is used to train the pre-constructed load detection model through the training samples to obtain a trained load detection model.

[0155] Optionally, the fusion module 113 may include: a detection unit, used to perform significant target detection on the second image to obtain a detection result; a segmentation unit, used to segment the target object in the second image based on the detection result to obtain an initial local image; and an optimization unit, used to optimize the initial local image through a dense conditional random field model to obtain a local image.

[0156] Optionally, the optimization unit is specifically used to: perform a feature extraction operation on the initial local image to obtain a feature extraction result; generate a fully connected graph based on all pixels of the initial local image; determine an energy function of the fully connected graph based on the feature extraction result and the local image; iteratively optimize the energy function until the energy function converges;

[0157] The initial local image is optimized based on the converged energy function to obtain a local image.

[0158] Optionally, the first image is a labeled image; the determination module 114 is specifically used to: respectively extract features of the composite image and the guide text corresponding to the composite image to obtain text features and image features; fuse the text features with the image features to obtain fused features; generate a query vector based on the text features; perform bounding box regression and classification on the fused features based on the query vector to annotate the labeling box and category of the target object in the composite image; and determine the annotated composite image and the first image as training samples.

[0159] Optionally, the detection model includes a backbone network, a neck network and a detection head; the backbone network is used to extract features from the input image to obtain a feature image; the neck network is used to fuse the feature image to obtain a fused feature; the detection head is used to detect the fused feature to obtain a detection result; wherein the backbone network is provided with a channel attention module.

[0160] Optionally, the acquisition module 111 is specifically used to: collect the first images in at least two scenes through a camera, the scenes including basic scenes and feature scenes, the feature scenes including shooting scenes with different lighting, shooting scenes with different shooting resolutions, shooting scenes with water vapor, shooting scenes with different angles and / or different degrees of shadows, shooting scenes with multiple target objects and at least one of the shooting scenes of a large-size target object.

[0161] Corresponding to the load detection method in the above embodiment, Fig.12 A structural block diagram of a load-bearing detection device 12 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0162] Reference Fig.12 The load-bearing detection device 12 includes: an acquisition module 121, used for acquiring a third image including the load-bearing device; a detection module 122, used for detecting the target object in the third image based on the trained load-bearing detection model to obtain a detection result; a control module 123, used for controlling the operation of the robot based on the detection result and the current task status of the robot; wherein the load-bearing detection model is obtained using the training method of the model in the aforementioned embodiment.

[0163] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0164] Fig.13 This is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. Fig.13 As shown, the electronic device 13 of this embodiment includes: at least one processor 131 ( Fig.13 Only one processor is shown in the figure), a memory 132, and a computer program 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the computer program 133, the steps in the training method embodiment of any of the above models are implemented, such as Figure 1 Steps 110-150 are shown.

[0165] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0166] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 13, such as a hard disk or memory of the electronic device 13. In other embodiments, the memory 132 may also be an external storage device of the electronic device 13, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 13.

[0167] Furthermore, the memory 132 may include both an internal storage unit of the electronic device 13 and an external storage device. The memory 132 is used to store operating devices, applications, boot loaders, data, and other programs, such as program codes of computer programs, etc. The memory 132 may also be used to temporarily store data that has been output or is to be output.

[0168] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0169] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0170] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0174] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A model training method, It is characterized in that The training method comprises: Acquire a first image of a carrying device including a robot; the perspective of the first image includes the perspective of the robot; Segmenting the target object from a pre-acquired second image containing the target object to obtain a partial image; fusing the local image with a specified image in the first image to obtain a composite image; the specified image is the first image that does not contain the target object; Using the composite image and the first image as training samples; The pre-built load detection model is trained using the training samples to obtain a trained load detection model.

2. The training method according to claim 1, It is characterized in that The step of extracting the target object from a pre-acquired second image containing the target object to obtain a partial image includes: Performing salient object detection on the second image to obtain a detection result; Segmenting the target object in the second image based on the detection result to obtain an initial local image; The initial local image is optimized through a dense conditional random field model to obtain the local image.

3. The training method according to claim 1, It is characterized in that The step of optimizing the initial local image by using a dense conditional random field model to obtain the local image includes: Performing a feature extraction operation on the initial local image to obtain a feature extraction result; Generate a fully connected graph based on all pixels of the initial local image; Determine an energy function of the fully connected graph based on the feature extraction result and the local image; Iteratively optimizing the energy function until the energy function converges; The initial local image is optimized based on the converged energy function to obtain the local image.

4. The training method according to claim 1, It is characterized in that The first image is a labeled image; and using the synthesized image and the first image as training samples includes: Extracting features of the composite image and the guide text corresponding to the composite image respectively to obtain text features and image features; Fusing the text feature with the image feature to obtain a fused feature; generating a query vector based on the text features; Based on the query vector, the fused feature is regressed and classified into a bounding box to mark a box and a category of a target object in the synthetic image; The annotated composite image and the first image are determined as training samples.

5. The training method according to any one of claims 1 to 4, It is characterized in that The detection model includes a backbone network, a neck network and a detection head; The backbone network is used to extract features from the input image to obtain a feature image; The neck network is used to fuse the feature images to obtain fused features; The detection head is used to detect the fusion feature and obtain a detection result; Among them, the backbone network is provided with a channel attention module.

6. The training method according to any one of claims 1 to 4, It is characterized in that The step of acquiring a first image of a carrying device including a robot comprises: The first images in at least two scenes are collected by a camera, and the scenes include basic scenes and feature scenes. The feature scenes include at least one of shooting scenes with different lighting, shooting scenes with different shooting resolutions, shooting scenes with water vapor, shooting scenes with different angles and / or different degrees of shadows, shooting scenes with multiple target objects, and shooting scenes with large-size target objects.

7. A load detection method, It is characterized in that Applied to a robot, the robot is provided with a carrying device for carrying a target object; The load detection method comprises: capturing a third image including the carrier; Detect the target object in the third image based on the trained load detection model to obtain a detection result; Based on the detection result and the current task status of the robot, control the operation of the robot; Wherein, the load detection model is obtained using the training method of the model described in any one of claims 1 to 7.

8. A training device for a model, It is characterized in that include: An acquisition module, configured to acquire a first image of a carrying device including a robot; the viewing angle of the first image includes the viewing angle of the robot; A segmentation module, used to segment the target object from a pre-acquired second image containing the target object to obtain a local image; a fusion module, configured to fuse the local image with a specified image in the first image to obtain a composite image; the specified image is the first image that does not contain the target object; A determination module, configured to use the composite image and the first image as training samples; The training module is used to train the pre-built load detection model through the training samples to obtain a trained load detection model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the load detection method according to claims 1-6 is implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the load detection method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • RGB-D salient target detection method and system based on cross-modal correlation learning

    CN119360177A

  • Segmenter-free training data set generation method for abnormal image segmentation

    CN119919758A

  • A method for generating a training dataset without a segmenter for abnormal image segmentation

    CN119919758B