Commodity identification method, storage medium and unmanned retail equipment

Abstract

The embodiment of the invention provides a commodity identification method, a storage medium and unmanned retail equipment, and the method comprises the steps: obtaining a first commodity taking image when it is detected that a user takes a commodity, obtaining a second commodity returning image when it is detected that a commodity returning operation exists after the user takes the commodity, and carrying out the image analysis, a first commodity identification feature, a first surface geometric feature, a second commodity identification feature, and a second surface geometric feature are extracted. And then, on the basis of feature comparative analysis on the two aspects of the commodity identifier and the commodity surface geometric structure, whether the second commodity put back by the user is the first commodity originally taken by the user or not is determined, and an alarm signal is generated when it is determined that the second commodity and the first commodity are different, so that the effects of preventing theft detection and improving the accuracy of theft detection are achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a commodity identification method, storage medium, and unmanned retail equipment. Background Art

[0002] With the widespread adoption of unmanned retail devices, the technology used to detect theft of goods from these devices is becoming increasingly mature. Current theft detection technologies typically rely on videos of users taking goods and whether or not they check out to determine theft. While this technology can detect theft by determining the presence of goods and whether or not a purchase has been made, if a user replaces the goods they actually took with expired or counterfeit goods, the technology will mistakenly determine that the user did not purchase the goods, and thus miss the user's theft. Therefore, current theft detection technologies for unmanned retail devices cannot effectively cover this type of theft, resulting in missed theft detections. Summary of the Invention

[0003] Based on the above problems, in order to solve the problem of missed theft detection in current unmanned retail equipment, the embodiments of the present application provide a commodity identification method, a storage medium and an unmanned retail equipment.

[0004] The embodiments of this application disclose the following technical solutions: In a first aspect, an embodiment of the present application provides a commodity identification method, which is applied to an unmanned retail device, and the method includes: Acquire a first product taking image when a user is detected taking a product, and acquire a second product returning image when the user is detected returning the product after taking the product; the second product is the product returned by the user after taking the first product; Performing image analysis on the first product picking-up image and the second product returning image to extract a first product identification feature and a first surface geometric feature of the first product, and a second product identification feature and a second surface geometric feature of the second product; performing a feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature to determine whether the second product is the first product taken by the user; If it is determined that the second commodity is not the first commodity, an alarm signal is generated.

[0005] In a possible implementation, performing feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature includes: performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree; When the product identification matching degree is less than a preset first threshold and greater than a preset second threshold, performing geometric feature matching based on the first surface geometric feature and the second surface geometric feature to determine the geometric feature matching degree; the preset first threshold is greater than the preset second threshold; Determine whether the second product is the first product taken by the user based on the geometric feature matching degree.

[0006] In a possible implementation, determining whether the second product is the first product taken by the user based on the geometric feature matching degree includes: If the geometric feature matching degree is not less than a preset third threshold, determining that the second product is the first product; When the geometric feature matching degree is less than the preset third threshold, performing a feature comparison analysis on the second surface geometric feature and the first surface geometric feature using a first preset neural network model to determine a human feature determination result for the second surface geometric feature; the human feature determination result is used to indicate whether the feature difference between the second surface geometric feature and the first geometric feature is caused by the user's product picking behavior or product returning behavior; If the result of the human feature determination is yes, determining that the second product is the first product; If the result of the human feature determination is negative, it is determined that the second product is not the first product.

[0007] In a possible implementation, after performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree, the method further includes: If the product identification matching degree is less than the preset second threshold, performing identification occlusion analysis on the first product picking image and the second product returning image using a second preset neural network model to generate an identification occlusion analysis result; the identification occlusion analysis result is used to determine whether the difference in product identification between the first product and the second product is caused by product identification occlusion; Determine whether the second product is the first product taken by the user based on the logo occlusion analysis result.

[0008] In a possible implementation, determining whether the second product is the first product taken by the user according to the logo occlusion analysis result includes: If the identification occlusion analysis result is yes, extracting the third product identification feature of the first product and the fourth product identification feature of the second product; the third product identification feature and the fourth product identification feature are not blocked; performing identification feature matching based on the third product identification feature and the fourth product identification feature to obtain an optimized product identification matching degree, and determining whether the second product is the first product taken by the user based on the optimized product identification matching degree; If the result of the logo obstruction analysis is negative, it is determined that the second product is not the first product.

[0009] In a possible implementation, after performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree, the method further includes: If the matching degree of the product identifier is less than the preset second threshold, determining that the second product is not the first product; When the matching degree of the product identification is greater than the preset first threshold, the second product is determined to be the first product.

[0010] In one possible implementation, the unmanned retail device includes at least one horizontal camera and at least one vertical camera; when the unmanned retail device includes multiple horizontal cameras and multiple vertical cameras, the horizontal cameras are arranged based on a preset first spacing, and the vertical cameras are arranged based on a preset second spacing; The acquiring of a first product-picking image when detecting that a user picks up a product, and acquiring of a second product-returning image when detecting that the user returns the product after picking up the product, includes: When it is detected that the user takes and returns a commodity, the commodities taken and returned by the user are photographed by the horizontal camera and the vertical camera to obtain the first commodity taking image and the second commodity returning image.

[0011] In a possible implementation, obtaining the first product picking image and the second product returning image includes: When detecting that the door of the unmanned retail device is open, acquiring in real time a first video frame image of the user taking the first product, and calculating the image clarity of multiple frames of the first video frame image; determining the first video frame image with the highest image definition as the first product picking image; When detecting that the door of the unmanned retail device is closed, obtaining a second video frame image of the second commodity based on a preset duration, and calculating image clarity of multiple frames of the second video frame image; The second video frame image with the highest image definition is determined as the second product return image.

[0012] In a second aspect, an embodiment of the present application provides an unmanned retail device, the device comprising: An image acquisition module is configured to acquire a first product-taking image when a user is detected taking a product, and to acquire a second product-returning image when a user is detected returning a product after taking the product; the second product is the product returned by the user after taking the first product; an image analysis module, configured to perform image analysis on the first product picking-up image and the second product returning image to extract a first product identification feature and a first surface geometric feature of the first product, and a second product identification feature and a second surface geometric feature of the second product; a comparison and analysis module, configured to perform feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature to determine whether the second product is the first product taken by the user; The alarm module is configured to generate an alarm signal when it is determined that the second commodity is not the first commodity.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any possible commodity identification method in the first aspect.

[0014] Compared to the prior art, the present application has the following advantages: The embodiments of the present application provide a product identification method, storage medium, and unmanned retail equipment. In the method, when a user is detected to have taken a product, a first product picking image is acquired. When a user is detected to have returned a product after taking the product, a second product returning image is acquired. The image analysis is then performed to extract the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature. Subsequently, based on feature comparison analysis at the product identification and product surface geometric structure levels, it is determined whether the second product returned by the user is the first product they originally took. If the two are determined to be different, an alarm signal is generated, thereby achieving the effect of preventing theft detection. Based on this method, when a user is detected to have returned a product, the product identification features and surface geometric features of the product taken and returned by the user can be used to accurately determine whether the product returned by the user is the first product they took. This prevents the user from replacing the product with expired or counterfeit products in the retail equipment. This effectively solves the problem of missed theft detection in unmanned retail equipment and improves the accuracy of theft detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 A flowchart of a commodity identification method provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a feature comparison and analysis method provided in an embodiment of the present application; Figure 3 A flowchart of a method for determining product theft based on geometric feature matching provided in an embodiment of the present application; Figure 4 A flowchart of an occlusion analysis method provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an unmanned retail device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of this application more clearly understood, the application is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be noted that the embodiments described in the embodiments of this application are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by people with ordinary skills in the field to which this application belongs. The words "first", "second" and similar terms used in the embodiments of this application do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, but do not exclude other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] As described above, current theft detection technologies typically rely on videos of users picking up items and whether or not they check out to determine theft. While this technology can detect theft by determining the presence of items and whether or not items have been checked out, if a user replaces an item they actually took with expired or counterfeit goods, the technology will misjudge that the user did not purchase the item, thereby missing the user's theft. Therefore, current theft detection technologies for unmanned retail devices cannot effectively cover this type of theft, and there is a problem of missed theft detection. Therefore, how to solve the problem in existing technologies of being unable to issue dangerous anomaly alerts based on the temperature of the vehicle's interior environment has become a technical problem that those skilled in the art urgently need to solve.

[0020] To address the above-mentioned issues, embodiments of the present application provide a product identification method, storage medium, and unmanned retail device. In the method, a first product picking image is acquired when a user is detected picking up a product, and a second product returning image is acquired when a user is detected returning a product after picking up the product. Image analysis is then performed to extract first product identification features, first surface geometric features, second product identification features, and second surface geometric features. Subsequently, feature comparison analysis is performed at both the product identification and product surface geometric structure levels to determine whether the second product returned by the user is the first product they originally picked up. If the two are determined to be different, an alarm signal is generated, thereby achieving the effect of preventing theft detection. Based on this method, when a user is detected returning a product, the product identification features and surface geometric features of the product picked up and returned by the user can be used to accurately determine whether the product returned by the user is the first product they originally picked up. This prevents users from replacing expired or counterfeit products in the retail device. This effectively addresses the current problem of missed theft detection in unmanned retail devices and improves the accuracy of theft detection.

[0021] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0022] See also Figure 1 , which is a flow chart of a commodity identification method provided in an embodiment of the present application, specifically comprising the following steps: S101: Acquire a first product taking image when a user is detected taking a product, and acquire a second product returning image when a product returning operation is detected after the user takes the product; the second product is the product returned by the user after taking the first product.

[0023] In this embodiment, at least one horizontal camera and at least one vertical camera are installed on the surface of the unmanned retail device, excluding the cabinet door. This ensures that detailed features of products can be captured from multiple dimensions when users pick up or put back items. For example, using the beverage section as an example, a horizontal camera (looking from the side) installed on the vertical side wall of the device can clearly capture product identification features such as product labels / barcodes / production dates. Meanwhile, a vertical camera (looking from above) installed on the horizontal side wall can clearly capture geometric features of the beverage surface, such as the indentations on the bottle and the wrinkles on the packaging. This combination of horizontal and vertical viewing angles captures product feature information, facilitating subsequent multi-angle image comparison to determine the characteristics of the products picked up and put back by the user. This improves the accuracy of product verification and comparison, preventing theft and missed detection.

[0024] In one possible implementation, multiple horizontal cameras and multiple vertical cameras can be set on the side wall of the device, and the arrangement spacing between the multiple horizontal cameras and the multiple vertical cameras can be limited by a preset first spacing and a preset second spacing, respectively, so as to achieve product shooting without blind spots and redundant identification. Among them, the preset first spacing defined for the multiple horizontal cameras can ensure that there are no overlapping blind spots in the horizontal viewing angle of the device, and the preset second spacing defined for the multiple vertical cameras can ensure continuous coverage of the device in the vertical viewing angle. Arranging the horizontal cameras and the vertical cameras in this way can achieve multi-angle cross-shooting of any product on the device shelf. Even if a camera is blocked or malfunctions, the adjacent camera can still capture the complete image; at the same time, this arrangement can also improve the accuracy of subsequent product feature matching through multi-source image comparison, which is particularly suitable for high-density display shelf scenarios.

[0025] The first product picking image is used to represent a clear image of the first product when the user picks up the first product. The second product picking image is used to represent an image of a product that the user can return after picking up the first product.

[0026] Specifically, the process of capturing the image of the first product being picked up and the second product depends on the device's detection of the cabinet door being opened or closed. During the actual operation of the device, the device needs to continuously detect the cabinet door's open or closed state. When the cabinet door is detected to be open, the horizontal camera and the vertical camera will be awakened to capture real-time video of the user picking up the first product. The video of the user picking up the first product is divided into images for each frame, and multiple continuous first video frames of the user picking up the first product can be obtained. Accordingly, in order to ensure the clarity of the image of the first product being picked up, in the process of acquiring the first video frame image, it is necessary to analyze the clarity of each frame of the image frame by frame using a real-time clarity evaluation algorithm (such as Laplacian gradient calculation), and determine the image with the highest clarity as the first product being picked up image for the user.

[0027] The same principle applies to capturing the image of the second product being returned. The second product is a product that the user returns to the shelf within a certain period of time after taking the first product. It may be the first product that the user originally took, or it may be a counterfeit or inferior product that the user uses to replace the first product. Therefore, to ensure the accuracy of capturing the image of the second product being returned, the capture of the image of the second product returning needs to be performed when the cabinet door is closed. When the cabinet door is closed, it indicates that the user has finished selecting the product, and there is no interference from the user's hand or other objects in the device. At this time, the second product is also in a static position, thus ensuring the accuracy of capturing the image of the second product returning. Similar to the clarity screening logic when acquiring the image of the first product being taken, in the process of acquiring the image of the second product returning, it is necessary to capture a video of the second product within a period of time after the cabinet door is closed based on a preset time length, and obtain multiple frames of the second video frame image. Filter the video frame image with the highest image clarity from the multiple frames of the second video frame image to obtain the second product returning image that can clearly show the characteristics of the second product.

[0028] Furthermore, in addition to real-time camera detection, the determination of a user's pick-up-and-return operation can also be made using gravity sensors installed on the shelves. These sensors capture the gravity of the items on the shelves in real time and monitor changes in gravity. If the gravity value decreases and then increases again within a short period of time, and if there is a slight difference between the two values, it can be determined that the user has taken and returned the item, and the process of capturing the first image of the item being taken and the second image of the item being returned can be initiated.

[0029] S102: Perform image analysis on the first commodity picking-up image and the second commodity returning image to extract a first commodity identification feature and a first surface geometric feature of the first commodity, and a second commodity identification feature and a second surface geometric feature of the second commodity.

[0030] After obtaining an image of a user taking a first item and an image of a user returning a second item, image analysis is performed on the two images to extract product features of the first and second items. This allows for comparison of the product features to determine whether the second item returned by the user is the first item the user originally took.

[0031] In this embodiment, the product features obtained by image analysis are divided into two types: product identification features and surface geometric features. Product identification features are often unique product identifiers for a product, such as barcodes, production dates, product numbers, trademarks, etc. Surface geometric features are used to characterize the structural features of the product surface. Taking beverage products as an example, the surface geometric features of beverage products can be obvious depressions left on the bottle body, or creases on the outer packaging caused by long-term squeezing, etc. Product identification features exist in every product and are unique to each product, while surface geometric features are not unique and can be copied to other products.

[0032] In one possible implementation, the product identification features can be quickly read through barcode or QR code scanning technology to read coded information such as QR codes, barcodes, etc. on the product packaging. In addition, radio frequency identification technology can also be used to read electronic tag information in the product packaging, such as production date, expiration date, unique identifier, etc. The surface geometric features of the product can be extracted based on multimodal image analysis technology, by identifying the edge features and line features of the geometric structure in the image, and calculating the depth value and three-dimensional relief angle of the edge depression or crease through binocular vision or structured light three-dimensional reconstruction technology, so as to obtain the surface geometric features of the product. This embodiment does not limit the method of obtaining product identification features and product surface geometric features.

[0033] S103: Performing feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature to determine whether the second product is the first product taken by the user; S104: If it is determined that the second commodity is not the first commodity, an alarm signal is generated.

[0034] Finally, a feature comparison analysis is performed based on the product identification features and surface geometric features of the first and second products (i.e., the first product identification feature, the second product identification feature, the first surface geometric feature, and the second surface geometric feature). By analyzing the degree of feature matching between these features, the device determines whether the second product returned by the user is the original first product. If the device determines that the second product returned by the user is not the original first product, it generates an alarm signal, thus accurately detecting theft.

[0035] Next, the process of step S103 will be introduced with reference to the accompanying drawings of specific embodiments.

[0036] See also Figure 2 , which is a flow chart of a feature comparison and analysis method provided in an embodiment of the present application, specifically comprising the following steps: S1031: performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree; S1032: If the product identification matching degree is less than a preset first threshold and greater than a preset second threshold, performing geometric feature matching based on the first surface geometric feature and the second surface geometric feature to determine the geometric feature matching degree; the preset first threshold is greater than the preset second threshold; S1033: Determine, based on the geometric feature matching degree, whether the second product is the first product taken by the user.

[0037] As can be seen from the previous description of product identification features, product identification features are used to characterize unique identification features on products, such as barcodes, QR codes, trademarks, etc., and product identification features have strong singleness and independence. Therefore, for the comparison and judgment between the first and second products, it is first necessary to carry out the comparison based on the product identification features of the two, that is, to calculate the product identification matching degree between the first product identification feature and the second product identification feature. If the product identification matching degree between the two is very high (greater than the preset first threshold), it indicates that the specific identification features of the first and second products have a high matching degree. At this time, it can be directly determined that the second product belongs to the first product originally taken by the user, without further reference to the geometric feature comparison between the two. If the product identification matching degree between the two is very small (less than the preset second threshold), it can be directly determined that the second product is not the first product originally taken by the user, without the need to proceed with the subsequent geometric feature comparison process.

[0038] Similarly, if the product identification matching degree between the two is in a relatively vague range, that is, the preset second threshold < product identification matching degree < preset first threshold, it indicates that the product identification matching analysis between the first product and the second product may be affected by external obstructions or other factors. At this time, it is necessary to further calculate the geometric feature matching degree between the two to further determine whether the second product belongs to the same product as the first product by comparing the geometric features of the product appearance.

[0039] See also Figure 3 This figure is a flow chart of a method for determining product theft based on geometric feature matching according to an embodiment of the present application. The figure shows the process of determining whether a second product is the same as a first product based on geometric feature matching according to an embodiment of the present application, which specifically includes the following steps: S1034: Determine whether the geometric feature matching degree is less than a preset third threshold; S1035: When the geometric feature matching degree is less than a preset third threshold, a feature comparison analysis is performed on the second surface geometric feature and the first surface geometric feature through a first preset neural network model to determine an artificial feature determination result for the second surface geometric feature.

[0040] S1036: Determine whether the result of the human feature determination is yes; S1037: If the result of the human feature determination is yes, the second product is determined to be the first product; S1038: If the result of the human feature determination is negative, the second product is determined not to be the first product; S1039: If the geometric feature matching degree is not less than a preset third threshold, determine that the second product is the first product.

[0041] The determination of the geometric features between the first and second items is based on the relationship between the geometric feature matching degree and a preset third threshold. When the geometric feature matching degree is not less than the preset third threshold, it indicates that the two items are highly similar in appearance, and the second item returned by the user can be determined to be the first item originally taken. When the geometric feature matching degree is less than the preset third threshold, it indicates that the geometric surface matching degree between the second item and the first item is low. However, in actual product purchases, when a user takes the first item, they may manually restore a dent in the item or forcefully remove it, causing the item's geometric features to change significantly over a short period of time. Therefore, to avoid such misjudgments of user theft, when the geometric feature matching degree between the two items is less than the preset third threshold, a first preset neural network model is used to perform feature analysis on the surface geometric features of the two items to determine whether the difference in geometric features between the second and first surfaces was caused by the user's behavior when taking or returning the item, thereby determining a human-induced feature determination result.

[0042] The result of the artificial feature determination is used to determine whether the characteristic difference between the second surface geometric feature and the first geometric feature is caused by the user's behavior when taking or putting back the first product. If the artificial feature determination result is yes, it indicates that the characteristic difference between the second surface geometric feature and the first geometric surface feature is caused by the user's normal behavior when taking or putting back the product, which is a normal purchasing behavior. In this case, it is determined that the second product returned by the user is the first product originally taken by the user. Conversely, if the artificial feature determination result is no, it indicates that the difference in geometric surface features between the two products is essentially the difference in appearance between the two different products. In this case, it is determined that the second product returned by the user is not the first product originally taken, thereby improving the accuracy of product theft determination.

[0043] In addition, in one possible implementation, the first preset neural network model can adopt a dual-branch structure based on a Siamese Network, which can parallel encode the geometric features of the product before and after the user operation, quantify the differences through a distance metric between feature vectors (such as Euclidean distance or contrast loss), and combine a gating mechanism to filter environmental noise. It is suitable for determining whether the difference in surface geometric features between the first and second products is caused by the picking or returning of the products.

[0044] As mentioned above, when the product identification matching degree is low, it may be because the user's hand or clothing obscured the product identification area during the process of picking up the product, resulting in a low product identification matching degree. Therefore, to avoid such situations causing misjudgment in the determination of product identification matching degree, when the product identification matching degree is less than the preset second threshold, further occlusion analysis is required on the first product picking image and the second product returning image to determine whether the difference in product identification matching degree is due to occlusion.

[0045] See also Figure 4 , which is a flow chart of an occlusion analysis method provided in an embodiment of the present application, specifically comprising the following steps: S201: When the product identification matching degree is less than a preset second threshold, a second preset neural network model is used to perform identification occlusion analysis on the first product picking image and the second product returning image to generate an identification occlusion analysis result.

[0046] The results of the product identification occlusion analysis are used to determine whether the difference in product identification between the first and second products is due to occlusion. If the product identification match is less than a preset second threshold, a second preset neural network model is used to analyze the first product removal image and the second product return image to confirm whether there is an occlusion affecting the product identification analysis.

[0047] In one possible implementation, the second neural network model can be designed as an occlusion-aware network based on multi-task learning. The model uses the dual-channel input of the first product picking image and the second product returning image as the data source, extracts multi-scale features through the pre-trained ResNet-50 backbone network, and adopts the attention mechanism module to enhance the focusing ability of the identification area; for occlusion analysis, the model performs pixel-level semantic segmentation and occlusion degree quantification in parallel, and combines the contrastive learning branch to calculate the dynamic difference of the occlusion area between the two images, so as to analyze whether there are external occluding objects (such as hands, clothing, outer packaging of other products, etc.) in the product identification area of ​​the first product picking image and the second product returning image.

[0048] S202: Determine whether the result of the logo occlusion analysis is yes; S203: If the result of the identification occlusion analysis is yes, extracting the third product identification feature of the first product and the fourth product identification feature of the second product; neither the third product identification feature nor the fourth product identification feature is occluded; S204: performing identification feature matching based on the third product identification feature and the fourth product identification feature to obtain an optimized product identification matching degree, and determining whether the second product is the first product taken by the user based on the optimized product identification matching degree; S205: If the result of the logo occlusion analysis is negative, it is determined that the second product is not the first product.

[0049] If the result of the identification occlusion analysis is yes, it indicates that the difference in product identification between the two is caused by an obstruction, and the calculated product identification matching degree is inaccurate. At this time, it is necessary to re-extract the third product identification feature in the first product and the fourth product identification feature in the second product. Among them, the third product identification feature and the fourth product identification feature are the identification features that are not obscured in both products. By re-extracting the identification features that are not obscured and recalculating the product identification matching degree, an optimized and more accurate product identification degree can be obtained, thereby re-determining whether the second product is the first product based on the optimized product identification matching degree.

[0050] Correspondingly, if the result of the logo obstruction analysis is negative, it indicates that the difference in the product logos between the two is not caused by an obstruction. At this time, it can be determined that the second product returned by the user is not the first product he originally took.

[0051] The embodiments of the present application provide a product identification method, storage medium, and unmanned retail equipment. In the method, a first product picking image is acquired when a user is detected picking up a product, and a second product returning image is acquired when a user is detected returning a product after picking up the product. Image analysis is performed on the image to extract the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature. Subsequently, based on feature comparison analysis at the product identification and product surface geometric structure levels, it is determined whether the second product returned by the user is the first product they originally picked up, and an alarm signal is generated when it is determined that the two are different, thereby achieving the effect of preventing theft detection. Based on this method, when a user is detected returning a product, the product identification features and surface geometric features of the user's picked up product and the returned product can be used to accurately determine whether the product returned by the user is the first product they picked up, thereby preventing the user from re-inserting expired or counterfeit products into the retail equipment to replace them. This effectively solves the problem of missed theft detection in unmanned retail equipment and improves the accuracy of theft detection.

[0052] An unmanned retail device provided in an embodiment of the present application is introduced below. The unmanned retail device described below and the commodity identification method described above can be referenced to each other.

[0053] See also Figure 5 , which is a schematic diagram of the structure of an unmanned retail device provided in an embodiment of the present application, specifically including the following modules: The image acquisition module 100 is configured to acquire a first product-taking image when a user is detected taking a product, and to acquire a second product-returning image when a user is detected returning a product after taking the product; the second product is the product returned by the user after taking the first product; An image analysis module 200 is configured to perform image analysis on the first product removal image and the second product return image to extract a first product identification feature and a first surface geometric feature of the first product, and a second product identification feature and a second surface geometric feature of the second product; a comparison and analysis module 300 for performing feature comparison and analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature to determine whether the second product is the first product taken by the user; The alarm module 400 is configured to generate an alarm signal when it is determined that the second commodity is not the first commodity.

[0054] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, an embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the commodity identification method described in any of the above embodiments.

[0055] The computer-readable media of the embodiments of the present application include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0056] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the commodity identification method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0057] It should be noted that the various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for methods, devices and media, since they are basically similar to the method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments. The methods, devices and media described above are merely schematic. The units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement them without expending creative work.

[0058] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A commodity identification method, characterized in that: Applied to unmanned retail equipment, the method includes: Acquire a first product taking image when a user is detected taking a product, and acquire a second product returning image when the user is detected returning the product after taking the product; the second product is the product returned by the user after taking the first product; Performing image analysis on the first product picking-up image and the second product returning image to extract a first product identification feature and a first surface geometric feature of the first product, and a second product identification feature and a second surface geometric feature of the second product; performing a feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature to determine whether the second product is the first product taken by the user; If it is determined that the second commodity is not the first commodity, an alarm signal is generated.

2. The method according to claim 1, characterized in that The performing feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature includes: performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree; When the product identification matching degree is less than a preset first threshold and greater than a preset second threshold, performing geometric feature matching based on the first surface geometric feature and the second surface geometric feature to determine the geometric feature matching degree; the preset first threshold is greater than the preset second threshold; Determine whether the second product is the first product taken by the user based on the geometric feature matching degree.

3. The method according to claim 2, characterized in that The determining, based on the geometric feature matching degree, whether the second product is the first product taken by the user includes: If the geometric feature matching degree is not less than a preset third threshold, determining that the second product is the first product; When the geometric feature matching degree is less than the preset third threshold, performing a feature comparison analysis on the second surface geometric feature and the first surface geometric feature using a first preset neural network model to determine a human feature determination result for the second surface geometric feature; the human feature determination result is used to indicate whether the feature difference between the second surface geometric feature and the first geometric feature is caused by the user's product picking behavior or product returning behavior; If the result of the human feature determination is yes, determining that the second product is the first product; If the result of the human feature determination is negative, it is determined that the second product is not the first product.

4. The method according to claim 2, characterized in that After performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree, the method further includes: If the product identification matching degree is less than the preset second threshold, performing identification occlusion analysis on the first product picking image and the second product returning image using a second preset neural network model to generate an identification occlusion analysis result; the identification occlusion analysis result is used to determine whether the difference in product identification between the first product and the second product is caused by product identification occlusion; Determine whether the second product is the first product taken by the user based on the logo occlusion analysis result.

5. The method according to claim 4, characterized in that The determining, based on the result of the logo occlusion analysis, whether the second product is the first product taken by the user includes: If the identification occlusion analysis result is yes, extracting the third product identification feature of the first product and the fourth product identification feature of the second product; the third product identification feature and the fourth product identification feature are not blocked; performing identification feature matching based on the third product identification feature and the fourth product identification feature to obtain an optimized product identification matching degree, and determining whether the second product is the first product taken by the user based on the optimized product identification matching degree; If the result of the logo obstruction analysis is negative, it is determined that the second product is not the first product.

6. The method according to claim 2, characterized in that After performing identification feature matching based on the first product identification feature and the second product identification feature to determine a product identification matching degree, the method further includes: If the matching degree of the product identifier is less than the preset second threshold, determining that the second product is not the first product; When the matching degree of the product identification is greater than the preset first threshold, the second product is determined to be the first product.

7. The method according to claim 1, characterized in that The unmanned retail device includes at least one horizontal camera and at least one vertical camera; when the unmanned retail device includes multiple horizontal cameras and multiple vertical cameras, the horizontal cameras are arranged based on a preset first spacing, and the vertical cameras are arranged based on a preset second spacing; The acquiring of a first product-picking image when detecting that a user picks up a product, and acquiring of a second product-returning image when detecting that the user returns the product after picking up the product, includes: When it is detected that the user takes and returns a commodity, the commodities taken and returned by the user are photographed by the horizontal camera and the vertical camera to obtain the first commodity taking image and the second commodity returning image.

8. The method according to claim 1, characterized in that The acquiring of the first product taking image and the second product returning image includes: When detecting that the door of the unmanned retail device is open, acquiring in real time a first video frame image of the user taking the first product, and calculating the image clarity of multiple frames of the first video frame image; determining the first video frame image with the highest image definition as the first product picking image; When detecting that the door of the unmanned retail device is closed, obtaining a second video frame image of the second commodity based on a preset duration, and calculating image clarity of multiple frames of the second video frame image; The second video frame image with the highest image definition is determined as the second product return image.

9. An unmanned retail device, characterized in that: The device comprises: An image acquisition module is configured to acquire a first product-taking image when a user is detected taking a product, and to acquire a second product-returning image when a user is detected returning a product after taking the product; the second product is the product returned by the user after taking the first product; an image analysis module, configured to perform image analysis on the first product picking-up image and the second product returning image to extract a first product identification feature and a first surface geometric feature of the first product, and a second product identification feature and a second surface geometric feature of the second product; a comparison and analysis module, configured to perform feature comparison analysis based on the first product identification feature, the first surface geometric feature, the second product identification feature, and the second surface geometric feature to determine whether the second product is the first product taken by the user; The alarm module is configured to generate an alarm signal when it is determined that the second commodity is not the first commodity.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the commodity identification method according to any one of claims 1 to 8 is implemented.

Images