Training method, device, equipment and storage medium for target detection model

By clustering product features and adjusting parameters in the object detection model, the problem of category bias in existing models in dense product scenarios is solved, achieving more accurate product recognition and handling detection.

CN118365852BActive Publication Date: 2024-11-26SHANGHAI SHANGTANG INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410368894.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-28
Publication Date
2024-11-26
Estimated Expiration
2044-03-28

AI Technical Summary

Technical Problem

Existing object detection algorithms suffer from category bias in scenarios with densely packed goods, due to a lack of detailed classification data annotations. This makes them unable to effectively identify different goods, especially in scenarios such as vending machines and smart refrigerators, which affects detection performance.

Method used

By clustering the features of goods in multiple second sample images and pre-determining the initial cluster centers, the parameters of the object detection model are adjusted to enable the model to better distinguish different goods. The model's recognition ability is then optimized by training with labeled third sample images.

Benefits of technology

This improves the accuracy of the object detection model in identifying different products and how they are handled, reduces reliance on human annotation, and enhances the model's detection performance in scenarios such as vending machines and smart refrigerators.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118365852B_ABST
    Figure CN118365852B_ABST
Patent Text Reader

Abstract

The present application discloses a training method, device, equipment and storage medium for a target detection model. The training method for the target detection model includes: obtaining a number of first sample images, at least some of which are images containing commodities; using a target detection model to extract features from each first sample image to obtain the features of the commodities in each first sample image; clustering the features of each commodity to determine the distance between each commodity and the features of a number of initial clustering centers, wherein, for each commodity feature, a clustering center whose distance with the commodity feature meets a preset condition is used as the target clustering center corresponding to the commodity feature, and a number of initial clustering centers are obtained in advance by clustering a second sample image containing at least one commodity; adjusting the parameters of the target detection model according to the distance between the features of each commodity and each target clustering center. The above scheme enables the model to better distinguish different commodities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, apparatus, device and storage medium for a target detection model. Background Art

[0002] Object detection algorithms are widely used in the field of computer vision. The purpose of an object detection algorithm is to detect objects of all target categories in an image for use by subsequent algorithms. In scenarios where goods are densely placed, such as vending machines and smart refrigerators, due to the large differences between goods or the differences in the way goods are held, the use of data annotations that lack subdivisions will cause the model to have category bias, which is often unpredictable and requires a lot of manpower for annotation. Most existing object detection algorithms cannot achieve satisfactory results, limiting the use scenarios of the product. Summary of the invention

[0003] The present application at least provides a training method, apparatus, device and storage medium for a target detection model.

[0004] The present application provides a method for training a target detection model, comprising: obtaining a number of first sample images, at least some of which are images containing commodities; using a target detection model to extract features from each of the first sample images to obtain features of the commodities in each of the first sample images; clustering the features of each of the commodities to determine the distance between the features of each of the commodities and a number of initial clustering centers, wherein, for each feature of the commodity, a clustering center whose distance to the feature of the commodity satisfies a preset condition is used as a target clustering center corresponding to the feature of the commodity, and the number of initial clustering centers are obtained in advance by clustering a second sample image containing at least one commodity; and adjusting the parameters of the target detection model according to the distance between the features of each of the commodities and each of the target clustering centers.

[0005] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0006] In some embodiments, the method further includes: in response to the existence of product features whose distances from each initial cluster center do not satisfy a preset condition, using the product features as candidate cluster centers; in response to the existence of other product features whose distances from the candidate cluster centers satisfy a preset condition in at least one iteration, updating the candidate cluster centers using other product features, and the time of at least one iteration is after the time of acquiring several first sample images; and adding the updated candidate cluster centers to several initial cluster centers.

[0007] In the above scheme, when obtaining the initial cluster center, it may not cover all the goods or the holding conditions of all the goods. If there are other product features that are close to the candidate cluster center during the subsequent iterative training process, it is very likely that these goods are of the same category or the holding conditions of the same goods and are different from the categories of other goods or the holding conditions of the same goods in the initial cluster center. Therefore, an initial cluster center can be added to several initial cluster centers so that the product or the holding condition of the product can still be identified later.

[0008] In some embodiments, at least some of the second sample images contain commodities of different categories or are held in different situations, and the method further includes: extracting features from each second sample image to obtain features of commodities of different categories or features in different holding situations; clustering the features of commodities of different categories or clustering the features of each commodity in different holding situations to obtain a number of initial clustering centers, wherein different clustering centers correspond to features of commodities of different categories or features of each commodity in different holding situations.

[0009] In the above scheme, by clustering the features of goods of different categories, the initial cluster center of each category of goods can be obtained, which is convenient for the subsequent target detection model to extract that the features of goods of different categories are more different, and the differences of goods of the same category are smaller. By clustering the features of each product in different holding conditions, the target detection model can better identify the holding conditions of the goods in the future, and thus better identify whether each product is foreground or background.

[0010] In some embodiments, the target detection model includes a first sub-model and a second sub-model, and the target detection model is used to extract features from each first sample image to obtain features of the goods in each first sample image, including: using the first sub-model to determine the target image area in each first sample image, the target image area includes hands and goods; extracting each target image area from each first sample image to obtain a number of local images; using the second sub-model to extract features from each local image to obtain features of each product.

[0011] In the above solution, by first using the first sub-model to determine the hand+product area of ​​the first sample image, irrelevant noise in the first sample image can be removed, reducing interference in the feature extraction process of the second sub-model.

[0012] In some embodiments, the target image area in each first sample image is determined using the first sub-model, including: using the first sub-model to determine the first image area where the hand is located and the second image area where each commodity is located in each first sample image; for each first sample image, determining the overlapping areas between the first image area in the first sample image and each second image area, and determining the target overlapping area, the target overlapping area being the overlapping area that meets the preset selection conditions; using the first image area and the area where the second image area corresponding to the target overlapping area in each first sample image are located as the target image area in each first sample image.

[0013] In the above scheme, if there is no overlapping area between the hand and the product or the overlapping area is too small, it means that the hand may not have taken the product, and the product can be excluded from target detection. Selecting the product that needs to be detected by the overlapping area between the hand and the product can make the subsequent target detection results more accurate.

[0014] In some embodiments, the preset selection condition includes that the ratio of the overlapping area to the first image area is greater than or equal to a first preset ratio, or the area of ​​the overlapping area is the largest and the ratio of the overlapping area to the first sample image is greater than or equal to a second preset ratio.

[0015] In the above scheme, a larger overlapping area indicates that the product is likely to be held by hand, so the product is likely to be the foreground. By extracting and analyzing the features, it can be determined whether the product is the foreground.

[0016] In some embodiments, the method also includes: obtaining a number of third sample images, in which the category label and position label of each product in the image are marked, and the category label indicates whether the product belongs to the foreground category or the background category; using the target detection model to extract features of each third sample image to obtain features of the product in each third sample image; based on the features of each product in each third sample image, determining the target detection results of each product in the third sample image, the target detection results include the predicted category of each product and the predicted image position of each product in each third sample image; based on the difference between the predicted category and the category label of each product in each third sample image, and the difference between the preset image position and the position label of each product in each third sample image, adjusting the parameters in the target detection model.

[0017] In the above solution, by training the target detection model in combination with the third sample image with a label, the target detection result obtained in the subsequent application of the target detection model can be more accurate.

[0018] In some embodiments, the method further includes: acquiring an image to be detected captured by an image acquisition device; performing target detection on the image to be detected using a trained target detection model to obtain a target detection result of the image to be detected.

[0019] The present application provides a training device for a target detection model, including: an image acquisition module, a feature extraction module, a clustering module, and a parameter adjustment module. The image acquisition module is used to acquire a number of first sample images, at least some of which are images containing commodities; the feature extraction module is used to extract features from each first sample image using a target detection model to obtain the features of the commodities in each first sample image; the clustering module is used to cluster the features of each commodity and determine the distance between each commodity and the features of a number of initial clustering centers, wherein, for each commodity feature, the clustering center whose distance with the commodity feature meets a preset condition is used as the target clustering center corresponding to the commodity feature, and the number of initial clustering centers are obtained in advance based on clustering of a second sample image containing at least one commodity; the parameter adjustment module is used to adjust the parameters of the target detection model according to the distance between the features of each commodity and each target clustering center.

[0020] The present application provides a training device for a target detection model, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned training method for the target detection model.

[0021] The present application provides a computer-readable storage medium having program instructions stored thereon, which implement the training method of the above-mentioned target detection model when the program instructions are executed by a processor.

[0022] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0025] Figure 1 It is a flowchart of an embodiment of a training method for a target detection model of the present application;

[0026] Figure 2 It is another flowchart of an embodiment of a training method for a target detection model of the present application;

[0027] Figure 3 This is a sub-process diagram showing step S12 of an embodiment of a training method for a target detection model of the present application;

[0028] Figure 4 It is another flowchart of an embodiment of the training method of the target detection model of the present application;

[0029] Figure 5 It is a structural schematic diagram of an embodiment of a training device for a target detection model of the present application;

[0030] Figure 6 It is a structural schematic diagram of an embodiment of a training device for a target detection model of the present application;

[0031] Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0032] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0033] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0034] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0035] The target detection model training method may be implemented by a training device for the target detection model. In some possible implementations, the target detection model training method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0036] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a training method for an image detection model of the present application. Specifically, the following steps may be included:

[0037] Step S11: Acquire a number of first sample images.

[0038] Several may be at least one. The first sample image may be a labeled image or an unlabeled image. This solution takes the first sample image as an unlabeled image as an example. At least part of the first sample images are images containing goods. Among them, the first sample image may be selected from images taken by various smart refrigerators, unmanned vending machines, unmanned shelves, etc. For the convenience of explanation, the following explanation will be given using a smart refrigerator as an example. The smart refrigerator is provided with several layers of shelves, and goods are placed on the shelves. The specific selection method may be to select images of hands placed on shelves, images of hands suspended without taking goods, images of hands holding one product, and images of hands holding at least two products, etc., which are images taken in any scene that a smart refrigerator may encounter during actual application.

[0039] The smart refrigerator will calculate the price based on the captured image and after confirming that the user has taken the product. The role of target detection is to determine the location of the product held by the user, so that the subsequent target recognition model can calculate the price based on the category of the product held by the user.

[0040] Step S12: extracting features from each first sample image using the target detection model to obtain features of the commodity in each first sample image.

[0041] The features of the commodities in each first sample image may include the features of each commodity or the features of some commodities in each first sample image. Exemplarily, the features of the commodities in each first sample image may be the features of the commodities held in the hand.

[0042] Step S13: Cluster the features of each product and determine the distance between the features of each product and several initial cluster centers.

[0043] Among them, for each feature of a commodity, the cluster center whose distance from the feature of the commodity meets the preset conditions is used as the target cluster center corresponding to the feature of the commodity, and several initial cluster centers are obtained in advance by clustering a second sample image containing at least one commodity. The second sample image can be considered as a labeled image or an unlabeled image. For example, the initial cluster center of each category of commodities can be obtained in advance based on the clustering of the features of each category of commodities in an image marked with commodity category labels. The preset condition can be the shortest distance, or the shortest distance and the distance is less than or equal to the preset distance, or the shortest distance and the distance is greater than or equal to the average of other distances. Optionally, other distances are the distances between the features of other commodities and their nearest initial cluster centers. Optionally, each feature of a commodity can correspond to at most one target cluster center, and the features of some commodities may not have a target cluster center.

[0044] In some other embodiments, the initial cluster centers are obtained in advance by clustering a second sample image containing at least one commodity, or by pre-marking each commodity in the second sample image as foreground or background, and then clustering according to the characteristics of each commodity. Some of the initial cluster centers among the initial cluster centers represent that the commodity characteristics belong to the foreground, and another part of the initial cluster centers represent that the commodity characteristics belong to the background, or some initial cluster centers all represent that the commodity characteristics belong to the foreground or the background. In other words, the foreground can correspond to one or more initial cluster centers, and the background can also correspond to one or more initial cluster centers. If the distance between the characteristics of the commodity and the initial cluster center corresponding to the foreground meets the conditions, the commodity is very likely to be the foreground.

[0045] Due to the different positional relationships between the product and the hand, the extracted product features may be different. For example, the features extracted from the same product when it is held in different hands are different from the features extracted when it is not held by the hand. By pre-establishing different initial clustering centers, the same product in different holding situations can be well distinguished.

[0046] Exemplarily, the initial cluster center a represents that product b belongs to the foreground (that is, product b is held by hand), and the initial cluster center c represents that product b belongs to the background (that is, product b is not held by hand). The distances between product b and the initial cluster center a and the initial cluster center c in the first sample image are calculated respectively, and then the target cluster center of product b in the first sample image can be determined based on the distance.

[0047] Step S14: adjusting the parameters of the target detection model according to the distance between the characteristics of each commodity and each target cluster center.

[0048] Exemplarily, the loss can be calculated based on the distance between the features of each product and each target cluster center, and then the parameters of the target detection model can be adjusted based on the loss so that the adjusted target model extracts features closer to the same type of products. Continuing with the above example, if the target cluster center of product b in the first sample image is calculated to be the initial cluster center a, it means that product b is likely to be the foreground in the first sample image. If the features of product b are far from the initial cluster center a, it means that the extracted features may not be accurate enough, and the parameters need to be adjusted so that the features of subsequent product b are closer to the initial cluster center a. This solution uses the target cluster center as a label, so that a large amount of annotation is not required, which can reduce the number of manual annotations.

[0049] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0050] In some embodiments, at least some of the second sample images contain goods of different categories or the goods are held in different situations. The holding situation includes the way the goods are held or not held. The method of obtaining several initial cluster centers may be: extracting features from each second sample image to obtain features of goods of different categories. Clustering the features of goods of different categories to obtain several initial cluster centers, and different cluster centers correspond to features of goods of different categories. In other embodiments, feature extraction is performed on each second sample image to obtain features of different goods in different holding situations. Clustering the features of different goods in different holding situations to obtain several initial cluster centers, and different cluster centers correspond to features of each good in different holding situations.

[0051] Exemplarily, there are multiple second sample images for each product category. After obtaining the features of the products in each second sample image, the features of the products in the same category are clustered to obtain the initial cluster center of the category, thereby obtaining the initial cluster centers of products in different categories. In other embodiments, clustering can also be performed by a clustering algorithm.

[0052] In the above scheme, by clustering the features of goods in different categories, the initial cluster centers of goods in each category can be obtained, which facilitates the subsequent target detection model to extract greater differences in the features of goods in different categories and smaller differences in the features of goods in the same category.

[0053] In some embodiments, see Figure 2 , Figure 2 FIG. 1 is another flow chart of an embodiment of a training method for a target detection model of the present application. Figure 2 As shown, the training method may further include the following steps:

[0054] Step S21: In response to the fact that the distances between the features of the products and the initial cluster centers do not satisfy the preset conditions, the features of the products are used as candidate cluster centers.

[0055] For example, if the distances between the features of the product and each initial cluster center are greater than a preset distance, it is determined that the distances between the features of the product and each initial cluster center do not meet the preset condition. The preset distance can be set according to demand.

[0056] Step S22: in response to the presence of other product features whose distances from the candidate cluster center satisfy a preset condition in at least one iteration, the candidate cluster center is updated using the other product features.

[0057] The time of at least one iteration is after the time of acquiring several first sample images. The at least one iteration can be one or more iterations. The training of the target detection model is generally iterative training. The current iteration may generate candidate cluster centers, and the next iteration may have other product features to update the candidate cluster centers. The updating method can be to re-cluster the features of the products corresponding to the candidate cluster centers with the features of other products to obtain new cluster centers, or to fine-tune the candidate cluster centers using the features of other products to obtain new cluster centers, such as clustering the features of other products to obtain cluster centers of other products, and averaging the cluster centers of other products with the candidate cluster centers to obtain new cluster centers.

[0058] Step S23: adding the updated candidate cluster centers to a number of initial cluster centers.

[0059] In the above scheme, when obtaining the initial cluster center, it may not cover all the goods or the holding conditions of all the goods. If there are other product features that are close to the candidate cluster center during the subsequent iterative training process, it is very likely that these goods are of the same category or the holding conditions of the same goods and are different from the categories of other goods or the holding conditions of the same goods in the initial cluster center. Therefore, an initial cluster center can be added to several initial cluster centers so that the product or the holding condition of the product can still be identified later.

[0060] See also Figure 3 , the above step S12 may include the following steps:

[0061] Step S121: using the first sub-model to determine the target image region in each first sample image.

[0062] The target image area includes the hand and the commodity. That is, the target image area is the area of ​​the hand + the commodity.

[0063] In some embodiments, step S121 may specifically include: using the first sub-model to determine the first image region where the hand is located and the second image region where each commodity is located in each first sample image. Exemplarily, the first sub-model may include a feature extraction module, a regression module, and a classification module. The features extracted by the feature extraction module are input into the regression module to obtain possible hand and possible commodity regions. The classification module obtains the confidence that each region belongs to the hand, commodity or background, and determines the image region where the hand and each commodity are located from each possible region according to the confidence. Optionally, the first sub-model and the second sub-model of the present application are trained independently. For example, the first sub-model may be trained first. When the output result of the first sub-model is relatively accurate, the parameters of the first sub-model are fixed and the second sub-model is trained. This scheme takes the first sub-model as an example where the training of the first sub-model is completed and only the second sub-model is trained. That is, the parameters of the target detection model adjusted in step S14 may specifically be the parameters of the second sub-model alone.

[0064] For each first sample image, the overlapping areas between the first image area in the first sample image and each second image area are determined, and a target overlapping area is determined. The target overlapping area is an overlapping area that meets a preset selection condition.

[0065] Then, the area where the first image area and the second image area corresponding to the target overlapping area in each first sample image are located is used as the target image area in each first sample image.

[0066] Among them, the preset selection conditions include that the ratio of the overlapping area to the first image area is greater than or equal to the first preset ratio, or the area of ​​the overlapping area is the largest and the ratio of the overlapping area to the first sample image is greater than or equal to the second preset ratio. The first preset ratio can be set according to demand. For example, if the first preset ratio is fifty percent, and the overlapping area accounts for fifty percent of the first image area, it can be determined that the overlapping area is the target overlapping area. Because the user may pick up multiple items at the same time, there may be multiple target overlapping areas in the same first sample image. In the above scheme, a large overlapping area indicates that the hand is likely to pick up the item, so the item is likely to be the foreground, and feature extraction and analysis can be used to determine whether the item is the foreground.

[0067] Step S122: extracting each target image region from each first sample image to obtain a plurality of local images.

[0068] The target image region may be a combination of the first image region and the second image region, or a regular image region including the first image region and the second image region. For example, the upper and lower boundaries or the left and right boundaries of the first image region and the second image region are not on the same line, and a regular image region may be determined as the target image region based on the boundaries of the two regions. In some embodiments, each extracted target image region is scaled proportionally so that its size is between 160 and 320 pixels.

[0069] Step S123: Use the second sub-model to extract features from each local image to obtain features of each product.

[0070] Among them, the second sub-model can be a retinanet network, which is a single network composed of a backbone network and two branches with specific tasks. The backbone network is responsible for calculating convolution features on the entire image to obtain the features of each product. The first branch performs image classification tasks on the output of the backbone network, and the second branch is responsible for convolutional border regression. The image classification task can specifically output the confidence that each detection box is foreground or background. The border regression can be the position of each product in the image. In this scheme, the retinanet network can have a third branch, and the third branch is used in the above step S13. In other embodiments, after the target detection model is deployed in the smart refrigerator, the first branch in the second sub-model can still determine whether the product belongs to the foreground or background according to clustering after inputting the features of the product, and then the category of the product is output by the output end of the first branch, or the confidence that the product belongs to the foreground or background is output. In other words, after the object detection model is deployed, the predicted product category is not a certain semantics specified by humans, but can be a cluster number obtained by clustering regional features by the model, such as the initial cluster center a, and then the category of the product in the image is obtained according to whether the cluster number belongs to the foreground or the background. That is, the first branch does not need to use the traditional single fully connected layer (FC) during classification, but can use clustering to determine whether each product belongs to the foreground or the background.

[0071] In the above scheme, if there is no overlapping area between the hand and the product or the overlapping area is too small, it means that the hand may not have taken the product, and the product can be excluded from target detection. Selecting the product that needs to be detected by the overlapping area between the hand and the product can make the subsequent target detection results more accurate.

[0072] In some embodiments, the training method may further include the following steps: obtaining a number of third sample images, in which the category label and position label of each commodity in the image are annotated, and the category label is whether the commodity belongs to the foreground category or the background category. Optionally, the third sample image may be an image taken by a camera or other image acquisition component on the smart refrigerator. After obtaining the images taken by each smart refrigerator, at least part of the images can be annotated with a human hand + commodity frame and a commodity frame, and the boundary of the hand + commodity frame can be perturbed according to a normal distribution, and at least 60% of the commodities in the hand frame are used as positive samples. At the same time, the error samples obtained by the hand + commodity detection network are also used as training data.

[0073] Optionally, part of the third sample image includes the commodity corresponding to the candidate cluster center. For example, the second sample image is an image including commodity A, commodity B, and commodity C, and the initial cluster center of commodity A, the initial cluster center of commodity B, and the initial cluster of commodity C are obtained by clustering, while the first sample image includes the image of commodity D, and the feature clustering of commodity D obtains the candidate cluster center, and part of the third sample image may include commodity D, so that the user can recognize commodity D as the foreground after taking commodity D from the refrigerator.

[0074] The target detection model is used to extract features from each third sample image to obtain features of the commodities in each third sample image. The specific method of feature extraction can be referred to above and will not be described in detail here.

[0075] Based on the features of each commodity in each third sample image, an object detection result of each commodity in the third sample image is determined, and the object detection result includes a predicted category of each commodity and a predicted image position of each commodity in each third sample image.

[0076] Based on the difference between the predicted category and the category label of each commodity in each third sample image and the difference between the preset image position and the position label of each commodity in each third sample image, the parameters in the target detection model are adjusted.

[0077] Optionally, the target detection model is trained using a semi-supervised method. After the detection model extracts features, a number of cluster centers is set, and a core closest to the feature is selected as the target class, and the loss and gradient are calculated for it. That is, for a sample, the distance of its feature from each cluster core is calculated, the closest class is used as the label, and the cross entropy loss is calculated. In this way, the features of the trained model will be around multiple cluster centers instead of a fixed point. This can improve the accuracy of the model.

[0078] In the above scheme, the first sub-model is smaller than the conventional model and is used to detect the rough area of ​​hand + product. The second sub-model performs fine detection on the rough area and uses clustering method to improve the model accuracy when predicting the prospect.

[0079] See also Figure 4 The training method of the target detection model provided in this application may also include the following steps:

[0080] Step S41: Acquire the image to be detected captured by the image acquisition device.

[0081] The trained object detection model can be deployed in the smart refrigerator. After the image acquisition component in the smart refrigerator captures the image to be detected, the object detection model detects the position of the goods taken by hand in the image to be detected, so that the subsequent object recognition model can determine the category of the goods at a specific position in the image to be detected based on the output of the object detection model, thereby determining the amount to be paid.

[0082] Step S42: Use the trained target detection model to perform target detection on the image to be detected, and obtain the target detection result of the image to be detected.

[0083] Training completion may be when the number of iterations of the target detection model reaches a preset number of times, or the accuracy of the output result of the target detection model is greater than or equal to a preset accuracy, or other custom training completion criteria.

[0084] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0085] In some embodiments, the above step S42 may include the following steps:

[0086] The target detection model is used to extract features from the image to be detected to obtain the features of the goods in the image to be detected. The features of each product are clustered to determine the distance between the features of each product and multiple initial cluster centers. For the features of each product, the initial cluster center whose distance meets the preset conditions is used as the target cluster center, and the category of the target cluster center is used as the category of the product, which can be a foreground category or a background category. The category corresponding to each initial cluster center can be a foreground category or a background category. In other embodiments, the last layer of the target detection model is a fully connected layer, and the category of each product is obtained by inputting the features of each product into the fully connected layer.

[0087] As mentioned above, the target detection model may include a first sub-model and a second sub-model. In the application scenario, the first sub-model can also be used to determine the first area where the hand is located in the image to be detected and the second area where each commodity is located; the overlapping area between the first area and each second area, and determine the overlapping area to be detected, the overlapping area to be detected is the overlapping area that meets the preset selection conditions; the area where the first area and the second area corresponding to the overlapping area to be detected in the image to be detected are located is used as the target area in the image to be detected. The target area is extracted from the image to be detected to obtain a local image; the second sub-model is used to extract features from each local image to obtain the features of each commodity. The method for determining the target area can refer to the above-mentioned method for determining the target image area, which will not be repeated here. In other embodiments, other methods such as non-maximum suppression can also be used to determine the target area.

[0088] In some embodiments, several initial cluster centers after the target detection model training is completed can also be deployed to the smart refrigerator. A display area is provided on the smart refrigerator, and the user can log in to the management program on the smart refrigerator through the administrator merchant. The user uploads an image containing a new product in the management program, and the management program can extract features of the image of the new product and cluster it to obtain a new initial cluster center and add it to several initial cluster centers. In addition, the label containing the new product image output by the user can also be received in the display area. The second sub-model is trained according to the above-mentioned training method through the label and several initial cluster centers, thereby realizing the user's own training of the target detection model.

[0089] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0090] See also Figure 5 , Figure 5It is a structural schematic diagram of an embodiment of a training device for a target detection model of the present application. The training device 50 for a target detection model includes an image acquisition module 51, a feature extraction module 52, a clustering module 53, and a parameter adjustment module 54. The image acquisition module 51 is used to acquire a plurality of first sample images, at least some of which are images containing commodities; the feature extraction module 52 is used to extract features from each first sample image using a target detection model to obtain the features of the commodities in each first sample image; the clustering module 53 is used to cluster the features of each commodity and determine the distance between each commodity and the features of a plurality of initial clustering centers, wherein, for each commodity feature, the clustering center whose distance with the commodity feature meets a preset condition is used as the target clustering center corresponding to the commodity feature, and the plurality of initial clustering centers are obtained in advance by clustering a second sample image containing at least one commodity; the parameter adjustment module 54 is used to adjust the parameters of the target detection model according to the distance between the features of each commodity and each target clustering center.

[0091] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0092] In some embodiments, the clustering module 53 is also used to: in response to the existence of product features whose distances from each initial cluster center do not meet the preset conditions, use the product features as candidate cluster centers; in response to the existence of other product features whose distances from the candidate cluster centers meet the preset conditions in at least one iteration, update the candidate cluster centers using other product features, and the time of at least one iteration is after the time of obtaining several first sample images; and add the updated candidate cluster centers to several initial cluster centers.

[0093] In the above scheme, when obtaining the initial cluster center, it may not cover all the goods or the holding conditions of all the goods. If there are other product features that are close to the candidate cluster center during the subsequent iterative training process, it is very likely that these goods are of the same category or the holding conditions of the same goods and are different from the categories of other goods or the holding conditions of the same goods in the initial cluster center. Therefore, an initial cluster center can be added to several initial cluster centers so that the product or the holding condition of the product can still be identified later.

[0094] In some embodiments, at least some of the second sample images contain commodities of different categories or are held in different situations, and the clustering module 53 is further used to: extract features from each second sample image to obtain features of commodities of different categories or features of commodities of different holding situations; cluster the features of commodities of different categories or cluster the features of commodities of different holding situations to obtain a number of initial clustering centers, and different clustering centers correspond to features of commodities of different categories or features of commodities of different holding situations.

[0095] In the above scheme, by clustering the features of goods of different categories, the initial cluster center of each category of goods can be obtained, which is convenient for the subsequent target detection model to extract that the features of goods of different categories are more different, and the differences of goods of the same category are smaller. By clustering the features of each product in different holding conditions, the target detection model can better identify the holding conditions of the goods in the future, and thus better identify whether each product is foreground or background.

[0096] In some embodiments, the target detection model includes a first sub-model and a second sub-model, and the feature extraction module 52 uses the target detection model to extract features from each first sample image to obtain features of the goods in each first sample image, including: using the first sub-model to determine the target image area in each first sample image, the target image area includes hands and goods; extracting each target image area from each first sample image to obtain a number of local images; using the second sub-model to extract features from each local image to obtain features of each product.

[0097] In the above solution, by first using the first sub-model to determine the hand+product area of ​​the first sample image, irrelevant noise in the first sample image can be removed, reducing interference in the feature extraction process of the second sub-model.

[0098] In some embodiments, the feature extraction module 52 uses the first sub-model to determine the target image area in each first sample image, including: using the first sub-model to determine the first image area where the hand is located and the second image area where each commodity is located in each first sample image; for each first sample image, determining the overlapping areas between the first image area in the first sample image and each second image area, and determining the target overlapping area, the target overlapping area is the overlapping area that meets the preset selection conditions; using the first image area and the area where the second image area corresponding to the target overlapping area in each first sample image are located as the target image area in each first sample image.

[0099] In the above scheme, if there is no overlapping area between the hand and the product or the overlapping area is too small, it means that the hand may not have taken the product, and the product can be excluded from target detection. Selecting the product that needs to be detected by the overlapping area between the hand and the product can make the subsequent target detection results more accurate.

[0100] In some embodiments, the preset selection condition includes that the ratio of the overlapping area to the first image area is greater than or equal to a first preset ratio, or the area of ​​the overlapping area is the largest and the ratio of the overlapping area to the first sample image is greater than or equal to a second preset ratio.

[0101] In the above scheme, a larger overlapping area indicates that the product is likely to be held by hand, so the product is likely to be the foreground. By extracting and analyzing the features, it can be determined whether the product is the foreground.

[0102] In some embodiments, each module in the training device 50 of the target detection model is also used for: the image acquisition module 51 is also used to acquire a number of third sample images, and the category label and position label of each commodity in the image are marked in the number of third sample images, and the category label is whether the commodity belongs to the foreground category or the background category; the feature extraction module 52 is also used to use the target detection model to extract features from each third sample image to obtain the features of the commodity in each third sample image; the training device 50 of the target detection model also includes a classification module (not shown in the figure), and the classification module is used to determine the target detection result of each commodity in the third sample image based on the features of each commodity in each third sample image, and the target detection result includes the predicted category of each commodity and the predicted image position of each commodity in each third sample image; the parameter adjustment module 54 is also used to adjust the parameters in the target detection model based on the difference between the predicted category and the category label of each commodity in each third sample image, and the difference between the preset image position and the position label of each commodity in each third sample image.

[0103] In the above solution, by training the target detection model in combination with the third sample image with a label, the target detection result obtained in the subsequent application of the target detection model can be more accurate.

[0104] In some embodiments, the training device 50 of the target detection model includes an image acquisition module (not shown) and a target detection module (not shown). The image acquisition module is used to obtain the image to be detected captured by the image acquisition device in the refrigerator; the target detection module is used to perform target detection on the image to be detected using the trained target detection model to obtain the target detection result of the image to be detected.

[0105] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0106] In some embodiments, the object detection model training method may also be performed by the object detection model training device 60. Figure 6 The target detection model training device 60 provided in the present application includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in the above-mentioned training method embodiment of any target detection model.

[0107] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of the training method embodiment of any target detection model described above. The processor 62 may also be referred to as a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip having signal processing capabilities. The processor 62 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 62 may also be any conventional processor, etc. In addition, the processor 62 may be implemented by an integrated circuit chip.

[0108] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0109] See also Figure 7 , Figure 7The computer-readable storage medium 70 stores program instructions 701 that can be executed by a processor, and the program instructions 701 are used to implement the steps in any of the above-mentioned training method embodiments of the target detection model.

[0110] In the above scheme, multiple initial clustering centers are obtained in advance according to the feature clustering of the commodities in multiple second sample images, and then the corresponding target clustering center can be determined according to the clustering results of the features of each commodity in the first sample image. Then, the parameters in the target detection model can be adjusted according to the distance between the target clustering center and the target detection model, so that the trained target detection model can better distinguish different commodities, so that all commodities that need to be detected can be identified in the subsequent application process of the target detection model.

[0111] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0112] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0116] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A method for training a target detection model, characterized in that: include: Acquire a plurality of first sample images, at least some of which are images containing commodities; Using the target detection model to extract features from each of the first sample images, to obtain features of the commodities in each of the first sample images; Clustering the features of each of the commodities, determining the distance between the features of each of the commodities and a number of initial cluster centers, wherein, for each of the features of the commodities, a cluster center whose distance from the features of the commodities satisfies a preset condition is used as a target cluster center corresponding to the features of the commodities, and the number of initial cluster centers are obtained in advance by clustering a second sample image containing at least one commodity; Adjusting the parameters of the target detection model according to the distances between the characteristics of each of the commodities and the target cluster centers; The target detection model includes a first sub-model and a second sub-model, and the feature extraction of each of the first sample images using the target detection model to obtain the features of the commodity in each of the first sample images includes: Determining a target image region in each of the first sample images using the first sub-model; Extracting each of the target image regions from each of the first sample images to obtain a plurality of local images; The second sub-model is used to extract features from each of the local images to obtain features of each of the commodities.

2. The method according to claim 1, characterized in that The method further comprises: In response to the fact that the distances between the feature of the product and each of the initial cluster centers do not satisfy a preset condition, taking the feature of the product as a candidate cluster center; In response to the presence of other product features whose distances from the candidate cluster center satisfy the preset condition in at least one iteration, the candidate cluster center is updated using the other product features, wherein the time of the at least one iteration is after the time of acquiring the plurality of first sample images; The updated candidate cluster centers are added to the several initial cluster centers.

3. The method according to claim 1, characterized in that The commodities included in at least some of the second sample images are of different categories or held in different conditions, and the method further includes: Extracting features from each of the second sample images to obtain features of commodities of different categories or features in different holding situations; The features of the commodities of different categories are clustered or the features of the commodities in different holding situations are clustered to obtain the several initial cluster centers, wherein different cluster centers correspond to the features of commodities of different categories or the features of the commodities in different holding situations.

4. The method according to any one of claims 1 to 3, characterized in that The target image area includes hands and commodities.

5. The method according to claim 4, characterized in that The determining the target image region in each of the first sample images by using the first sub-model includes: Determine, by using the first sub-model, a first image region where the hand is located and a second image region where each commodity is located in each of the first sample images; For each of the first sample images, determining overlapping areas between the first image area in the first sample image and each of the second image areas, and determining a target overlapping area, where the target overlapping area is an overlapping area that meets a preset selection condition; The area where the first image area and the second image area corresponding to the target overlapping area in each of the first sample images are located is used as the target image area in each of the first sample images.

6. The method according to claim 5, characterized in that The preset selection condition includes that the ratio of the overlapping area to the first image area is greater than or equal to a first preset ratio, or the area of ​​the overlapping area is the largest and the ratio of the overlapping area to the first sample image is greater than or equal to a second preset ratio.

7. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Acquire a plurality of third sample images, wherein the plurality of third sample images are annotated with a category label and a position label of each commodity in the image, wherein the category label indicates whether the commodity belongs to a foreground category or a background category; Using the target detection model to extract features from each of the third sample images, to obtain features of the commodities in each of the third sample images; Determine, based on the features of each of the commodities in each of the third sample images, a target detection result for each of the commodities in the third sample images, wherein the target detection result includes a predicted category of each of the commodities and a predicted image position of each of the commodities in each of the third sample images; Based on the difference between the predicted category and the category label of each commodity in each third sample image and the difference between the preset image position and the position label of each commodity in each third sample image, the parameters in the target detection model are adjusted.

8. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Acquire an image to be detected captured by an image acquisition device; The trained target detection model is used to perform target detection on the image to be detected to obtain a target detection result of the image to be detected.

9. A training device for a target detection model, characterized in that: include: An image acquisition module, used to acquire a plurality of first sample images, at least some of which are images containing commodities; A feature extraction module, used to extract features from each of the first sample images using a target detection model to obtain features of commodities in each of the first sample images; A clustering module, used to cluster the features of each of the commodities, and determine the distance between the features of each of the commodities and a plurality of initial cluster centers, wherein, for each of the features of the commodities, a cluster center whose distance from the features of the commodities satisfies a preset condition is used as a target cluster center corresponding to the features of the commodities, and the plurality of initial cluster centers are obtained in advance by clustering a second sample image containing at least one commodity; A parameter adjustment module, used to adjust the parameters of the target detection model according to the distance between the characteristics of each of the commodities and each of the target cluster centers; The target detection model includes a first sub-model and a second sub-model, and the feature extraction module uses the target detection model to extract features from each of the first sample images. The method of obtaining the features of the commodity in each of the first sample images includes: Determining a target image region in each of the first sample images using the first sub-model; Extracting each of the target image regions from each of the first sample images to obtain a plurality of local images; The second sub-model is used to extract features from each of the local images to obtain features of each of the commodities.

10. A training device for a target detection model, characterized in that: It includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the training method of the target detection model described in any one of claims 1 to 8.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the training method of the target detection model described in any one of claims 1 to 8 is implemented.