Intelligent cabinet difference commodity detection classification method

Abstract

The application provides a difference commodity detection and classification method of an intelligent cabinet, acquires two commodity images captured by a camera arranged above a shelf of the intelligent cabinet at different times, detects and processes commodities in the two commodity images by using a pre-trained commodity difference detection model to obtain a difference commodity detection result, wherein the difference commodity detection result comprises difference commodity detection frame coordinates and image information of the difference commodity, identifies the detected difference commodity according to the difference commodity detection result to obtain difference commodity category information. The application directly detects the image position of the difference commodity on two intelligent cabinet scene images before and after purchase by a consumer, and then identifies the commodity in the image position of the difference commodity to determine the difference commodity category, thereby solving the problems of high labeling cost, high updating cost and high deployment cost of a current full-supervised target detection model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a method for detecting and classifying differential goods in a smart vending machine. Background Technology

[0002] E-commerce has developed rapidly in the internet age, but after a period of high-speed growth, traditional e-commerce has entered a bottleneck period. People have increasingly higher demands for convenience and timeliness in consumption, and traditional e-commerce is struggling to meet these needs for a convenient and high-quality life. Under the concept of "new retail," traditional e-commerce is attempting to combine with offline sales channels, and smart vending machines have become an important development direction for new retail.

[0003] Smart vending machine solutions can be divided into two main categories: visual and non-visual. Non-visual solutions once dominated the market due to their advantages of simple principles, convenient deployment, and high accuracy. However, with the progress and development of deep neural networks in the field of computer vision, visual solutions based on deep neural networks have become the research focus of smart vending machine solutions. Summary of the Invention

[0004] This invention provides a method for detecting and classifying different products in smart vending machines. It can directly detect the location of different products in two smart vending machine scene images before and after a consumer's purchase, and then use a target recognition model to classify the different products, thereby completing functions such as automatic product settlement and inventory counting.

[0005] This invention provides a method for detecting and classifying differential goods in smart vending machines, comprising:

[0006] Two product images captured at different times by a camera configured above the smart vending machine shelf; wherein the two product images are obtained by the camera taking a top-down view;

[0007] The product difference detection model is pre-trained to detect products in the two product images. The detection process includes feature extraction, feature fusion, and target regression to obtain the product difference detection result. The product difference detection result includes the coordinates of the product difference detection box and the image information of the product difference.

[0008] Based on the detection results of the differentiated goods, the detected differentiated goods are identified to obtain differentiated goods category information.

[0009] In an optional embodiment, the product difference detection model has two weight-sharing feature extractors, the outputs of which are connected to a feature fusion operator, and the output of which is connected to a regression network.

[0010] Accordingly, the pre-trained product difference detection model is used to detect products in the two product images. This detection process includes feature extraction, feature fusion, and target regression to obtain the product difference detection result, including:

[0011] The two weight-shared feature extractors are used to extract features from the two product images respectively, resulting in first image features and second image features;

[0012] The feature fusion operator is used to calculate the difference between the first image features and the second image features to obtain the fused image features;

[0013] The regression network is used to identify the features of the fused image to obtain the difference in product detection results.

[0014] Furthermore, the two weight-sharing feature extractors employ ResNet-18 with the last fully connected layer removed.

[0015] Furthermore, the regression network contains a spatial attention module and a channel attention module.

[0016] In an optional embodiment, the step of classifying the detected differential products according to the differential product detection results to obtain differential product category information includes:

[0017] The location information of the differential product is determined based on the coordinates of the differential product detection box and the image information to which the differential product belongs.

[0018] A pre-trained product recognition model is used to identify products at the locations of different products and determine the category information of the different products.

[0019] In an optional embodiment, before acquiring two product images taken at different times by a camera configured above the smart vending machine shelf, the method further includes:

[0020] Collect multiple sets of differentiated product images; wherein each set of differentiated product images includes two product images taken from the same perspective at different times;

[0021] The differential products in the multiple sets of differential product images are bounded together to obtain the bounding box information corresponding to each set of differential product images, and the label information of each set of differential product images is generated based on the bounding box information.

[0022] Data augmentation processing is performed on the multiple sets of differentiated product images to obtain multiple sets of differentiated product training images;

[0023] The constructed product difference detection model is trained using the multiple sets of differentiated product training images and label information to obtain the pre-trained product difference detection model.

[0024] Further, the step of generating label information for each group of differentiated product images based on the annotation box information includes:

[0025] The multiple sets of differentiated product training images are divided into grids according to a preset grid size to obtain at least one grid region;

[0026] Based on the bounding box information corresponding to each set of training images of different products, generate label information corresponding to each grid region of each set of training images of different products; wherein, the label information includes the positional relationship between the grid and the center point of the bounding box, the positional relationship between the two images of different products and the center point of the bounding box, the horizontal and vertical coordinates of the center point of the bounding box, and the length and width of the bounding box.

[0027] Furthermore, the data augmentation processing of the multiple sets of differentiated product images includes at least one of the following:

[0028] The positions of the two different product images in each group of training images for different products are randomly interchanged;

[0029] For each set of training images of different products, two images of different products are randomly cropped and / or randomly filled.

[0030] For each set of training images of different products, randomly mirror and flip the two images of different products.

[0031] Enhance the contrast and / or brightness and / or saturation of the two different product images in each set of training images for different products.

[0032] Furthermore, training the constructed product difference detection model using the multiple sets of differentiated product training images and label information includes:

[0033] Differential product detection is performed on each grid region in the multiple sets of differential product training images, and the preset anchor frame in each grid region is adjusted according to the detection results to obtain differential product prediction information for each grid region.

[0034] The difference product detection loss value is calculated based on the difference product prediction information and label information of each grid area, and the difference product detection loss value is backpropagated to each layer of the product difference detection model to update the weight parameters of each layer according to the difference product detection loss value.

[0035] Repeat the above training steps until the product difference detection model converges.

[0036] This invention provides a method for detecting and classifying discrepancies in smart vending machines. The method involves acquiring two product images taken at different times by a camera positioned above the smart vending machine shelf; wherein the two product images are obtained from a top-down view by the camera. A pre-trained product difference detection model is used to detect the products in the two product images. The detection process includes feature extraction, feature fusion, and target regression to obtain a discrepancy detection result. The discrepancy detection result includes the coordinates of the discrepancy detection box and the image information to which the discrepancy product belongs. Based on the discrepancy detection result, the detected discrepancy products are identified to obtain discrepancy product category information. Compared with existing technologies, this solution directly detects the image locations of discrepancies in the smart vending machine scene images before and after a consumer's purchase, and then uses a target recognition model to identify the products at the image locations of the discrepancies to obtain the discrepancy product category, thereby achieving automatic checkout and intelligent management of goods. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of a scenario architecture upon which this disclosure is based;

[0039] Figure 2 A flowchart illustrating a method for detecting and classifying differential goods in an intelligent vending machine, as provided in an embodiment of this disclosure;

[0040] Figure 3 This is a schematic diagram of the structure of a commodity difference detection model provided in an embodiment of the present disclosure;

[0041] Figure 4 This is a flowchart illustrating a product difference detection model training method provided in an embodiment of this disclosure. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Currently, smart vending machines mainly use visual recognition algorithms to identify the category of goods taken by consumers and automatically settle the bill. The main principle is to use a camera to take an image of the goods purchased by the user, use a trained target recognition model to automatically identify the category of the purchased goods, and then settle the bill based on the category of the goods purchased by the user.

[0044] However, this method is based on a fully supervised object detection model to achieve product identification and calculation, which has problems such as high labeling cost, high update cost, and high deployment cost.

[0045] Figure 1 This is a schematic diagram of a scenario architecture upon which this disclosure is based, such as... Figure 1 As shown, the scenario architecture based on this disclosure may include a smart vending machine 1, a differential commodity detection and classification device 2, and a camera 3.

[0046] The differential product detection and classification device 2 is hardware or software that can interact with the camera 3 via a network, and it can be used to execute the differential product detection and classification methods described in the following embodiments.

[0047] When the differential product detection and classification device 2 is hardware, it can be an electronic device with computing capabilities. When the differential product detection and classification device 2 is software, it can be installed in an electronic device with computing capabilities. Such electronic devices include, but are not limited to, servers, smart boxes, and desktop computers.

[0048] Specifically, camera 3 can be a hardware device integrated on smart vending machine 1 that can capture images of a wide range of items at close range.

[0049] In practical scenarios, the differential product detection and classification device 2 can be a server integrated or installed on the smart vending machine 1. The device can run on the smart vending machine 1, or it can be integrated or installed in a backend server that processes product images, providing product detection and classification services to the smart vending machine 1. Specifically, the differential product detection and classification device 2 acquires two product images taken by the camera 3 before and after the consumer purchases the product. The device 2 uses the method shown in the following embodiment to detect the two product images before and after the consumer's purchase, determine the location of the differential product within the smart vending machine 1, and identify the category of the product at the location of the differential product for automatic settlement.

[0050] The following will further explain the differential product detection and classification method for smart vending machines provided in this application:

[0051] Figure 2 This is a flowchart illustrating a method for detecting and classifying differential goods in a smart vending machine, as provided in an embodiment of this disclosure. Figure 2As shown in the embodiments of this disclosure, a method for detecting and classifying differential goods in a smart vending machine includes:

[0052] S21. Obtain two product images taken at different times by a camera configured above the smart vending machine shelf; wherein the two product images are obtained by the camera taking a top-down view.

[0053] Among them, the camera can capture a wide range of product images at close range.

[0054] In this embodiment, since consumers take goods from the smart vending machine shelves when purchasing goods, in order to accurately identify the goods taken by the user, the camera can be configured to capture all the goods on the shelf. Therefore, by using the camera to capture an overhead view of all the goods on the shelf, the product recognition error caused by blind spots can be avoided.

[0055] S22. The product difference detection model is pre-trained to detect products in the two product images. The detection process includes feature extraction, feature fusion and target regression to obtain the product difference detection result. The product difference detection result includes the coordinates of the product difference detection box and the image information of the product difference.

[0056] In this embodiment, unlike ordinary object detection models that detect a single input product image, the product difference detection model in this embodiment takes a pair of product images as input. The product difference detection model sequentially performs feature extraction, feature fusion, and target regression on the two product images to obtain the coordinates of the difference product detection box and the image information to which the difference product belongs.

[0057] Specifically, such as Figure 3 As shown, the product difference detection model consists of two weight-shared feature extractors, a feature fusion operator, and a regression network. The product difference detection process includes: extracting features from the two product images using the two weight-shared feature extractors to obtain first image features and second image features; calculating the difference between the first image features and the second image features using the feature fusion operator to obtain fused image features; and identifying the fused image features using the regression network to obtain the product difference detection result.

[0058] Furthermore, the feature extractor in this embodiment uses a ResNet-18 with the last fully connected layer removed to extract the depth features of the two input product images. The initialization parameters of the ResNet-18 were pre-trained on the ImageNet dataset. The regression network in this embodiment contains a spatial attention module and a channel attention module.

[0059] The advantage of this setup is that it uses two feature extractors with shared weights to extract features from the input product images. Since the parameters of the two feature extractors are shared, the two input product images are mapped to the same feature space, which is beneficial for obtaining the spatial differences between the two input product images. The key to this algorithm is to use a feature fusion operator to fuse the features extracted by the two feature extractors to obtain difference information. The difference is calculated on the features of the two input product images to obtain fused image features containing difference information. Based on the fused image features, the difference product information can be regressed. In order to better analyze the spatial difference of the fused image features and the dependency relationship between global features, two types of attention modules, one spatial dimension and one channel dimension, are added to the regression network to simulate the semantic interdependence in the spatial and channel dimensions, respectively, to better analyze the spatial difference features. The regression network in this embodiment is a regression network composed of four convolutional layers. It divides the feature map into S×S grids, regresses K prediction boxes for each grid, and finally uses a non-maximum suppression method to remove redundant prediction boxes.

[0060] S23. Based on the detection results of the differentiated goods, identify the detected differentiated goods to obtain differentiated goods category information.

[0061] In this embodiment, based on the coordinates of the differential product detection box and the image to which the differential product belongs, the product recognition model can be used to identify the differential product in the image to which the differential product belongs, and determine the category of the differential product as the category of the product purchased by the consumer. Then, the payment can be settled according to the category of the product purchased by the consumer, thus realizing intelligent management of the products in the smart vending machine.

[0062] Specifically, the location information of the differential product is determined based on the coordinates of the differential product detection box and the image information to which the differential product belongs; the product at the location of the differential product is identified using a pre-trained product recognition model to determine the category information of the differential product.

[0063] The advantage of this setup is that the product difference detection model only detects the location of the different products and does not perform product classification. Product classification is done by a dedicated product recognition model. Therefore, when new products are launched, only the product recognition model needs to be updated, and there is no need to retrain the product difference detection model, which reduces the difficulty of model training and saves a lot of model update costs.

[0064] This embodiment provides a product recognition method for smart vending machines. It acquires two product images taken at different times by a camera positioned above the smart vending machine shelf; wherein the two product images are obtained from a top-down view by the camera. A pre-trained product difference detection model is used to detect products in the two product images. The detection process includes feature extraction, feature fusion, and target regression to obtain a difference product detection result. The difference product detection result includes the coordinates of the difference product detection box and the image information to which the difference product belongs. Based on the difference product detection result, the detected difference products are identified to obtain difference product category information. By adopting the technical solution provided in this embodiment, the image positions of difference products in two smart vending machine scene images before and after a consumer's purchase are directly detected. Then, a target recognition model is used to identify the products at the image positions of the difference products to obtain the difference product category, solving problems such as high annotation costs, high update costs, and high deployment costs.

[0065] Based on the above embodiments, Figure 4 This is a flowchart illustrating a product difference detection model training method provided in an embodiment of this disclosure. Before obtaining two product images taken at different times by a camera configured above the smart vending machine shelf in step S21, a product difference detection model training stage is also included, as shown below. Figure 4 As shown, it includes:

[0066] S41. Collect multiple sets of differentiated product images; wherein each set of differentiated product images includes two product images taken from the same perspective at different times.

[0067] In this embodiment, since there is no publicly available dataset for direct training, the dataset used to train the model needs to be collected and labeled according to the actual application scenario. Image pairs taken inside the vending machine can be collected. Each image pair simulates the scene inside the vending machine before and after a consumer's purchase. The images are taken from the same angle at different times by a camera inside the vending machine. The time interval between the images is short, thus ensuring that the two images are basically under the same lighting and background. However, the products in the images are different. This shooting condition is set to simulate the situation of a consumer making a purchase in the smart vending machine. Usually, the time of a purchase is not too long and the number of products purchased is not large, so the changes in products brought about by the purchase are not particularly large.

[0068] S42. Mark the different products in the multiple sets of different product images to obtain the annotation box information corresponding to each set of different product images, and generate the label information of each set of different product images based on the annotation box information.

[0069] In this embodiment, the bounding box of each set of differentiated product images represents the location of the differentiated product in each set of differentiated product images. The bounding box information can be saved as an XML file. The bounding box information records the bounding boxes of the differentiated products on the two images of each set of differentiated product images. The position and size of the differentiated products can be recorded using the coordinates of the upper left corner and the lower right corner of the bounding box. During training, the bounding box information of each set of differentiated product images is parsed according to specific rules to obtain the label information of each set of differentiated product images for model training.

[0070] Specifically, after drawing bounding boxes to label the different products in each group of different product images, the bounding box information is parsed, including: dividing the multiple groups of different product training images into grids according to a preset grid size to obtain at least one grid region; generating label information corresponding to each grid region of each group of different product training images based on the bounding box information corresponding to each group of different product training images; wherein, the label information includes the positional relationship between the grid and the center point of the bounding box, the positional relationship between the two different product images and the center point of the bounding box, the horizontal and vertical coordinates of the center point of the bounding box, and the length and width of the bounding box.

[0071] For example, each set of differentiated product images includes Image A and Image B. Each set of differentiated product images is divided into S×S grids. Each grid corresponds to a label represented by a vector of the following form: [P(Obj), P(A|Obj), P(B|Obj), midx, midy, w, h]. P(Obj) indicates whether the center point of the label box falls within the grid. P(A|Obj) and P(B|Obj) indicate whether the center point of the label box falls on Image A or Image B, respectively. midx and midy represent the horizontal and vertical coordinates of the center point of the label box, and w and h represent the length and width of the label box. If the center point of the bounding box falls within the current grid cell, P(Obj) is set to 1; otherwise, it is set to 0. If the bounding box is on image A, P(A|Obj) is set to 1; otherwise, it is set to 0. The setting of P(B|Obj) is similar. If there are two different bounding boxes on images A and B respectively, and their center points fall exactly within the same grid cell, both P(A|Obj) and P(B|Obj) are set to 1. The size and position information of the bounding box are recorded using (midx, midy, w, h). Following these rules, a label vector of size S×S×7 can be obtained for each group of differentiated product images.

[0072] S43. Perform data augmentation processing on the multiple sets of differentiated product images to obtain multiple sets of differentiated product training images.

[0073] In this embodiment, due to insufficient training sample data and the irregular distribution of different objects on each group of different product images, data augmentation is used to process the initial training sample data to expand the type and quantity of training sample data.

[0074] Specifically, data augmentation processing includes at least one of the following: randomly perposing the positions of two different product images in each set of training images for different products; randomly cropping and / or randomly padding two different product images in each set of training images for different products; randomly mirroring two different product images in each set of training images for different products; and enhancing the contrast and / or brightness and / or saturation of two different product images in each set of training images for different products.

[0075] For example, to mitigate the negative impact of uneven distribution of differentiated products in images, a random permutation data augmentation strategy was designed. The order of images A and B, along with their corresponding annotations, was permuted with a 50% probability. This mitigates the negative impact of the fixed input order of images A and B on the learning process, while balancing the number of differentiated products distributed in images A and B. To increase the diversity of training samples, two differentiated product images in each group of training images were randomly cropped or randomly filled to obtain samples with richer dimensions. Then, the contrast, brightness, and saturation of the images were randomly adjusted. Finally, images A and B were randomly mirrored with a 50% probability. When performing the above image augmentation processing on each group of differentiated product training images, the processing methods for images A and B must be exactly the same.

[0076] S44. The constructed product difference detection model is trained using the multiple sets of differentiated product training images and label information to obtain the pre-trained product difference detection model.

[0077] In this embodiment, multiple sets of training images of differentiated products and label information are input into the constructed product difference detection model. The product difference detection model detects differentiated products in each set of training images to obtain differentiated product prediction information. Based on the differentiated product prediction information and label information, the differentiated product detection loss value is calculated and backpropagated to each layer of the product difference detection model to update the weight parameters of each layer according to the differentiated product detection loss value. The above training steps are repeated until the product difference detection model converges.

[0078] Specifically, differential product detection is performed on each grid region in the multiple sets of differential product training images, and the preset anchor boxes in each grid region are adjusted according to the detection results to obtain differential product prediction information for each grid region; differential product detection loss value is calculated based on the differential product prediction information and label information of each grid region, and the differential product detection loss value is backpropagated to each layer of the product difference detection model to update the weight parameters of each layer according to the differential product detection loss value; the above training steps are repeated until the product difference detection model converges.

[0079] For example, predicting the center coordinates and width / length offsets of samples based on predefined anchor boxes is much simpler than directly regressing coordinates. This simplifies the regression problem and makes the network easier to train. The predefined anchor boxes tile the feature map using convolutions, making the position of each anchor box relative to its corresponding grid fixed. For each predicted box, the network predicts: the probability that an object is present in the predicted box, the probability that the object in the predicted box falls on both images, and the position of the predicted box. For example, with K predefined anchor boxes for each grid, and predicting 1 object score, 2 position scores, and 4 offsets relative to the anchor box for each anchor box, 7K filters are applied around each grid in the feature map. The model models the task of detecting discrepancies as a regression problem. First, the image is divided into an S×S grid, and K bounding boxes are predicted for each grid. The entire loss function is divided into two parts: prior box loss and regression loss. The regression loss includes object loss, coordinate loss and category loss. In the commodity difference detection model, the category refers to whether the object falls in graph A or graph B in space. It does not represent the type of commodity. It is spatial information. In this algorithm, we only identify which graph the object falls in, and do not identify the category to which the object belongs. The prior box loss is shown in Formula 1.

[0080]

[0081] Where t represents the total number of training samples. This indicates that the current number of training samples is less than the preset number of iterations T0. The prior box loss is only calculated when this condition is met, because L prior The design is simply to allow the model to learn the preset anchor boxes more quickly in the early stages.

[0082] The definition of regression loss is as follows:

[0083]

[0084]

[0085]

[0086]

[0087] Specifically, if the intersection-union ratio (IoU) between the predicted bounding box obtained from the j-th anchor point corresponding to the i-th grid and the labeled bounding box falling into that grid is less than a preset threshold Thresh, then it is considered that there is no target object within the predicted bounding box, and L is calculated for the predicted bounding box. noojb When the j-th anchor box corresponding to the i-th grid matches a label box falling into that grid, then L is calculated for that predicted box. obj L crood and L classTo reflect the different impacts of the same prediction bias on large-scale and small-scale bounding boxes, we added a factor to the error loss. This parameter reduces the penalty for prediction bias on large-scale boxes, while increasing the penalty for prediction bias on small-scale boxes. and These are the normalized values ​​of the annotation box's length and width relative to the current grid, ranging from 0 to 1.

[0088] The loss function is obtained by adding the above loss functions with different weights, as shown in Formula 6.

[0089] L t =λ prior *L prior +λ coord *L coord +λ noobj *L noobj +λ obj *L obj +λ class *L class Formula Six

[0090] In practice, the weight is set to λ. prior =0.01,λ noobj =0.5,λ obj =5,λ crd =2,λ class =1.

[0091] As can be seen, the loss function in this embodiment calculates the loss value for each part of each grid cell, and its meaning is not entirely the same as the loss function of a typical object detection model. First, the independent variable of the loss function is a fused feature pair of input images, representing information different from that in a typical object detection model. Second, the semantics of the category are completely different from those in a typical object detection model. In this loss function, the category indicates which image the current predicted bounding box belongs to, representing spatial semantic information, while the category in a typical object detection model refers to the specific category to which the object within the predicted bounding box belongs.

[0092] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for detecting and classifying differential goods in a smart vending machine, characterized in that, include: Collect multiple sets of images of different products; each set of images includes two images of the product taken from the same perspective at different times. The differential products in the multiple sets of differential product images are bounded together to obtain the bounding box information corresponding to each set of differential product images, and the label information of each set of differential product images is generated based on the bounding box information. Data augmentation processing is performed on the multiple sets of differentiated product images to obtain multiple sets of differentiated product training images; The constructed product difference detection model is trained using the multiple sets of differentiated product training images and label information to obtain a pre-trained product difference detection model. The product difference detection model is configured with two weight-shared feature extractors. The outputs of the two weight-shared feature extractors are connected to a feature fusion operator. The output of the feature fusion operator is connected to a regression network. Two product images captured at different times by a camera configured above the smart vending machine shelf; wherein the two product images are obtained by the camera taking a top-down view; The product difference detection model is pre-trained to detect products in the two product images. The detection process includes feature extraction, feature fusion, and target regression to obtain the product difference detection result. The product difference detection result includes the coordinates of the product difference detection box and the image information of the product difference. Based on the detection results of the differentiated goods, the detected differentiated goods are identified to obtain differentiated goods category information; The step of generating label information for each group of differentiated product images based on the annotation box information includes: The multiple sets of differentiated product training images are divided into grids according to a preset grid size to obtain at least one grid region; Based on the bounding box information corresponding to each set of training images of different products, generate label information corresponding to each grid region of each set of training images of different products; wherein, the label information includes the positional relationship between the grid and the center point of the bounding box, the positional relationship between the two images of different products and the center point of the bounding box, the horizontal and vertical coordinates of the center point of the bounding box, and the length and width of the bounding box; The data augmentation process performed on the multiple sets of differentiated product images includes at least one of the following: The positions of the two different product images in each group of training images for different products are randomly interchanged; For each set of training images of different products, two images of different products are randomly cropped and / or randomly filled. For each set of training images of different products, randomly mirror and flip the two images of different products. Enhance the contrast and / or brightness and / or saturation of the two different product images in each set of training images for different products; The step of training the constructed product difference detection model using the multiple sets of differentiated product training images and label information includes: Differential product detection is performed on each grid region in the multiple sets of differential product training images, and the preset anchor frame in each grid region is adjusted according to the detection results to obtain differential product prediction information for each grid region. The difference product detection loss value is calculated based on the difference product prediction information and label information of each grid area, and the difference product detection loss value is backpropagated to each layer of the product difference detection model to update the weight parameters of each layer according to the difference product detection loss value. Repeat the above training steps until the product difference detection model converges; The pre-trained product difference detection model is used to detect products in the two product images. The detection process includes feature extraction, feature fusion, and target regression to obtain the product difference detection results, including: The two weight-shared feature extractors are used to extract features from the two product images respectively, resulting in first image features and second image features; The feature fusion operator is used to calculate the difference between the first image features and the second image features to obtain the fused image features; The regression network is used to identify the features of the fused image to obtain the difference in product detection results.

2. The method for detecting and classifying differential goods in a smart vending machine according to claim 1, characterized in that, The two weight-sharing feature extractors use ResNet-18 with the last fully connected layer removed.

3. The method for detecting and classifying differential goods in a smart vending machine according to claim 1, characterized in that, The regression network contains a spatial attention module and a channel attention module.

4. The method for detecting and classifying differential goods in a smart vending machine according to claim 1, characterized in that, The step of classifying the detected differential products based on the differential product detection results to obtain differential product category information includes: The location information of the differential product is determined based on the coordinates of the differential product detection box and the image information to which the differential product belongs. A pre-trained product recognition model is used to identify products at the locations of different products and determine the category information of the different products.

Images