Goods perception system and goods perception method based on image monitoring

Abstract

The present invention provides a product sensing system and method based on image monitoring. The system includes a sample collection unit, a model training unit, a real-time image collection unit, and a product category acquisition unit. The method includes sample collection, model training, real-time image collection, and product category acquisition steps. The present invention has the beneficial effect of determining the placement and removal status of products on a shelf, and can identify the specific type of product removed or returned based on real-time images of the space in front of the shelf.

Description

Technical Field

[0001] The present invention relates to a product sensing technology for the retail industry, and more particularly to a product sensing system and method based on image monitoring. Background Art

[0002] Traditional retail requires dedicated sales and cashier staff in every supermarket or convenience store, resulting in high labor costs. With the advancement of electronic payment, identity recognition, and cloud computing technologies, unmanned supermarkets are becoming increasingly feasible. A fundamental challenge facing unmanned supermarkets is determining and recording user purchases. Specifically, the server must accurately determine the type, quantity, and price of items removed from the shelf, enabling automated payment.

[0003] Existing technologies use RFID technology to detect the items users purchase. This approach requires attaching RFID tags to each item and installing RFID readers at the entrance. However, its drawbacks are: first, the hardware cost is high, with each RFID tag costing approximately 0.5-1 yuan. This increases the cost of each item, reducing the supermarket's competitiveness. For an item costing 5 yuan, adding an RFID tag can increase the cost by 10-20%. Second, there's a risk that items can be blocked or removed from the store, potentially allowing users to deceive the RFID reader and cause item loss. Third, this solution only allows for payment at the supermarket entrance. If a user consumes edible items before leaving the store and leaves the packaging behind, the RFID reader will be unable to detect and determine the user's actual purchase amount. This approach relies heavily on the self-discipline and moral integrity of individual consumers, rather than on technical constraints. This creates significant risks in the operation of unmanned supermarkets. Summary of the Invention

[0004] The purpose of the present invention is to provide a product perception technology based on image monitoring, which can solve the technical problems existing in the prior art of poor perception accuracy of the product placement status, large monitoring errors, high costs, and easy loss of goods.

[0005] To achieve the above-mentioned objectives, the present invention provides a product perception system based on image monitoring, comprising: a sample acquisition unit for acquiring multiple groups of image samples, each group of image samples including multiple sample images of a type of product at multiple angles; a group of image samples of the same type of product are provided with the same group identifier, and the group identifier is the type of product corresponding to the group of image samples; a model training unit for training a convolutional neural network model based on each sample image and its group identifier in the multiple groups of image samples to obtain a product recognition model; a real-time image acquisition unit for continuously acquiring at least one real-time image of the space in front of the shelf, each real-time image including part or all of the product image; and a product category acquisition unit for acquiring the type of product displayed in the real-time image based on the real-time image and the product recognition model.

[0006] Furthermore, in different embodiments, the product perception system based on image monitoring also includes a first camera, connected to the sample acquisition unit, for taking pictures of each product at multiple angles and multiple distances; a second camera, connected to the real-time picture acquisition unit, for taking real-time pictures of the space in front of a shelf; wherein the field of view of the lens of the second camera covers the space in front of the shelf.

[0007] Furthermore, in different embodiments, the number of the second cameras is two or four; the lens of the second camera is facing the middle of the space area in front of the shelf; the angle between the central axis of the lens of the second camera and the horizontal plane is 30-60 degrees; and / or, the distance between the lens of the second camera and the upper or lower end of the shelf is 0.8-1.2 meters; and / or, the distance between the lens of the second camera and one side of the shelf is 0.8-1.2 meters.

[0008] Furthermore, in different embodiments, the model training unit includes a grouping model training unit, which inputs each sample image of multiple groups of training samples and their group identification into a convolutional neural network model, and obtains a grouping model after training; wherein, the training samples are all or part of the multiple groups of image samples, and the grouping model is a product recognition model.

[0009] Furthermore, in different embodiments, the model training unit includes a sample classification unit for randomly dividing multiple groups of image samples into two categories, namely training samples and test samples; a grouping model training unit for inputting each sample image of multiple groups of training samples and their group identification into a convolutional neural network model to obtain a grouping model after training; and a cross-validation unit for verifying the grouping model based on each image of multiple groups of test samples and the group identification of each group of test samples to calculate the model accuracy; when the accuracy of the grouping model is less than a preset threshold, it is returned to the sample classification unit; when the accuracy of the grouping model is greater than or equal to the preset threshold, the grouping model is a product identification model.

[0010] Furthermore, in different embodiments, the grouping model training unit includes a feature extraction unit, which is used to input the picture of each training sample into the convolutional neural network, perform feature extraction, and obtain a feature image; a candidate area generation unit, which is used to input the picture of each training sample into the candidate area network to generate multiple candidate areas; a candidate area mapping unit, which is used to map the picture candidate area of ​​each training sample to the feature image of the last convolution layer of the convolutional neural network; and a classifier generation unit, which is used to collect the picture feature images and candidate areas of multiple training samples, calculate the feature images of the candidate areas, and send them into the classifier network to generate a classifier.

[0011] Furthermore, in different embodiments, the cross-validation unit includes: a test sample input unit, used to input each image of multiple test samples into the grouping model to obtain test group identifiers of multiple test samples; and an accuracy calculation unit, used to compare the test group identifiers of multiple test samples with the group identifier of the test sample, and calculate the ratio of the number of identical identifiers to the number of test samples, which is the accuracy of the primary data model.

[0012] Furthermore, in different embodiments, the product category acquisition unit also includes a group identification acquisition unit, which is used to input multiple real-time images into the product identification model to obtain multiple group identifications corresponding to the multiple real-time images as possibility conclusions; and an identification credibility calculation unit, which is used to calculate the ratio of the number of each group identification in the possibility conclusion to the total number of all group identifications in the possibility conclusion. This ratio is the credibility of each group identification, and the type of goods corresponding to the group identification with the greatest credibility is the type of goods displayed on the real-time image.

[0013] Furthermore, in different embodiments, the group identification acquisition unit includes: a second feature extraction unit, used to input a real-time image into a convolutional neural network, perform feature extraction, and obtain a feature image; a second candidate area generation unit, used to input the real-time image into the candidate area network to generate multiple candidate areas; a second candidate area mapping unit, used to map the candidate area of ​​the real-time image to the feature image of the last layer of the convolutional neural network; and a group acquisition unit, used to collect the feature images and candidate areas of multiple images, calculate the feature images of the candidate areas, and send them into the classifier network to obtain the group identification corresponding to the real-time image.

[0014] To achieve the above-mentioned purpose, the present invention also provides a product perception method based on image monitoring, comprising the following steps: a sample collection step for collecting multiple groups of image samples, each group of image samples including multiple sample images of a product at multiple angles; a group of image samples of the same type of product are provided with the same group identifier, and the group identifier is the type of product corresponding to the group of image samples; a model training step for training a convolutional neural network model based on each sample image and its group identifier in the multiple groups of image samples to obtain a product recognition model; a real-time image collection step for continuously collecting at least one real-time image of the space in front of the shelf, each real-time image including a part or all of the product image; and a product category acquisition step for acquiring the type of product displayed in the real-time image based on the real-time image and the product recognition model.

[0015] Furthermore, in different embodiments, the model training step includes the following steps: a grouping model training step, inputting each sample image and its group identifier of multiple groups of training samples into a convolutional neural network model, and obtaining a grouping model after training; wherein, the training samples are all or part of the multiple groups of image samples, and the grouping model is a product recognition model.

[0016] Furthermore, in different embodiments, the model training step includes the following steps: a sample classification step, used to randomly divide multiple groups of image samples into two categories, namely training samples and test samples; a grouping model training step, used to input each sample image of multiple groups of training samples and their group identification into the convolutional neural network model, and obtain the grouping model after training; and a cross-validation step, used to verify the grouping model based on each image of multiple groups of test samples and the group identification of each group of test samples, and calculate the model accuracy; when the accuracy of the grouping model is less than a preset threshold, return to the sample classification step; when the accuracy of the grouping model is greater than or equal to the preset threshold, the grouping model is a product identification model.

[0017] Furthermore, in different embodiments, the grouping model training step includes the following steps: a feature extraction step, which is used to input the picture of each training sample into the convolutional neural network, perform feature extraction, and obtain a feature image; a candidate area generation step, which is used to input the picture of each training sample into the candidate area network to generate multiple candidate areas; a candidate area mapping step, which is used to map the picture candidate area of ​​each training sample to the feature image of the last convolution layer of the convolutional neural network; a classifier generation step, which is used to collect the picture feature images and candidate areas of multiple training samples, calculate the feature images of the candidate areas, and send them into the classifier network to generate a classifier.

[0018] Furthermore, in different embodiments, the cross-validation step includes the following steps: a test sample input step for inputting each image of multiple test samples into the grouping model to obtain test group identifiers of multiple test samples; and an accuracy calculation step for comparing the test group identifiers of multiple test samples with the group identifier of the test sample, and calculating the ratio of the number of identical identifiers to the number of test samples, which is the accuracy of the primary data model.

[0019] Furthermore, in different embodiments, the product category acquisition step also includes the following steps: a group identification acquisition step, which is used to input multiple real-time images into the product identification model to obtain multiple group identifications corresponding to the multiple real-time images as possibility conclusions; and an identification credibility calculation step, which is used to calculate the ratio of the number of each group identification in the possibility conclusion to the total number of all group identifications in the possibility conclusion. This ratio is the credibility of each group identification, and the type and / or quantity of goods corresponding to the group identification with the greatest credibility is the type and / or quantity of goods displayed on the real-time image.

[0020] Furthermore, in different embodiments, the group identification acquisition step includes the following steps: a second feature extraction step, used to input a real-time image into a convolutional neural network, perform feature extraction, and obtain a feature image; a second candidate area generation step, used to input the real-time image into the candidate area network to generate multiple candidate areas; a second candidate area mapping step, used to map the candidate area of ​​the real-time image to the feature image of the last layer of the convolutional neural network; and a group acquisition step, used to collect the feature images and candidate areas of multiple images, calculate the feature images of the candidate areas, and send them into the classifier network to obtain the group identification corresponding to the real-time image.

[0021] The present invention provides a product sensing system and method based on image monitoring, which can determine the placement and removal status of products on shelves and identify the specific types of products removed or returned based on real-time images of the space in front of the shelves. If combined with a product sensing system based on weight monitoring, the quantity of products can also be accurately determined. If combined with user identification and location tracking technologies, the identity of the user who removed or returned the products can also be accurately determined, thereby accurately adding or deleting shopping records from the user's shopping database, allowing for automatic settlement after the user completes their purchase. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A top view of an unmanned supermarket according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the overall structure of the shelf according to an embodiment of the present invention;

[0024] Figure 3 This is a structural block diagram of the user identity recognition system according to an embodiment of the present invention;

[0025] Figure 4 This is a structural block diagram of the user positioning system according to an embodiment of the present invention;

[0026] Figure 5 A distribution diagram of the image sensors in a closed space according to an embodiment of the present invention;

[0027] Figure 6 This is a structural block diagram of a product perception system based on image monitoring according to an embodiment of the present invention;

[0028] Figure 7 A diagram showing the positional relationship between the second camera and the shelf in an embodiment of the present invention;

[0029] Figure 8 This is a structural block diagram of the shopping user judgment system according to an embodiment of the present invention;

[0030] Figure 9 This is a structural block diagram of the shopping database system according to an embodiment of the present invention;

[0031] Figure 10 This is a structural block diagram of the settlement system according to an embodiment of the present invention;

[0032] Figure 11 This is a flow chart of a method for sensing goods based on image monitoring according to an embodiment of the present invention;

[0033] Figure 12 This is a flowchart of the model training steps described in an embodiment of the present invention;

[0034] Figure 13This is a flow chart of the grouping model training steps described in an embodiment of the present invention;

[0035] Figure 14 This is a flow chart of the cross-validation steps described in an embodiment of the present invention;

[0036] Figure 15 This is a flow chart of the type determination steps described in an embodiment of the present invention;

[0037] Figure 16 This is a flow chart of the steps for obtaining a group identifier according to an embodiment of the present invention.

[0038] The components in the figure are numbered as follows:

[0039] 1 closed space, 2 shelves, 3 racks, 4 pallets, 5 racks, 7 data processing equipment;

[0040] 100 user identification system, 101 access control device, 102 identification device; 1021 code scanning device, 1022 identity acquisition unit, 103 user entrance, 104 user exit;

[0041] 200 target positioning system, 201 three-dimensional image acquisition device, 202 target coordinate acquisition unit;

[0042] 2011 image sensor, 2012 depth image sensor, 2013 RGB image sensor, 2014 3D image integration unit;

[0043] 2021 coordinate system establishment unit, 2022 parameter acquisition unit, 2023 background removal unit, 2024 target object coordinate calculation unit; 20221 sensor coordinate acquisition unit, 20222 relative coordinate acquisition unit, 20223 coordinate correction unit;

[0044] 400 product perception system based on image monitoring, 401 sample collection unit, 402 model training unit, 403 real-time image collection unit, 404 product category acquisition unit, 405 first camera, 406 second camera; 407 placed product confirmation unit; 408 taken product confirmation unit;

[0045] 4021 sample classification unit, 4022 grouping model training unit, 4023 cross validation unit;

[0046] 40221 feature extraction unit, 40222 candidate region generation unit, 40223 candidate region mapping unit, 40224 classifier generation unit; 40231 test sample input unit, 40232 accuracy calculation unit;

[0047] 4041 group identification acquisition unit, 4042 identification credibility calculation unit;

[0048] 40411 a second feature extraction unit, 40412 a second candidate region generation unit, 40413 a second candidate region mapping unit, 40414 a group acquisition unit;

[0049] 500 shopping user judgment system, 501 product information storage unit, 502 shelf coordinate storage unit, 503 shelf and user matching judgment unit, 504 product and user matching judgment unit;

[0050] 600 shopping database system, 601 shopping database generating unit, 602 shopping database updating unit;

[0051] 700 settlement system, 701 total amount calculation unit, 702 payment unit. DETAILED DESCRIPTION

[0052] The following fully describes the preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and its protection scope is not limited to the embodiments mentioned herein.

[0053] In the drawings, components with the same structure are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. Directional terms used in the present invention, such as up, down, front, back, left, right, inside, outside, upper surface, lower surface, side, top, bottom, front end, rear end, and end, are merely directions in the drawings and are used only to explain and illustrate the present invention, and are not intended to limit the scope of protection of the present invention.

[0054] When certain components are described as being “on” another component, the component may be directly placed on the other component; there may also be an intermediate component with the component placed on the intermediate component, and the intermediate component placed on the other component. When a component is described as being “mounted to” or “connected to” another component, both can be understood to be directly “mounted” or “connected” or indirectly “mounted to” or “connected to” the other component through an intermediate component.

[0055] This embodiment relates to a product perception system based on image monitoring, which is a part of an unmanned vending system used in unmanned supermarkets. Figures 1 and 2As shown, the unmanned vending system includes a closed space 1, in which a plurality of shelves 2 are provided. Each shelf 2 includes a bracket 3 and a plurality of trays 4 detachably mounted on the bracket 3. The plurality of trays 4 are parallel to each other at different heights or flush with each other at the same height. Each tray 4 is provided with a plurality of shelves 5 arranged in parallel, and each shelf 5 is placed with at least one product. In this embodiment, the products placed on the shelf 5 need to be easy for the user to take away or put back, so the end of the shelf 5 facing the user is used as the front end of the shelf 5. In this embodiment, each shelf 5 is an open box body, which can be placed with one or more products. The products are standard products, and the appearance and weight of the same type of products are the same or similar.

[0056] This embodiment also includes a data processing device 7, such as a server or a computer. The data processing device 7 is internally provided with multiple data processing software and has multiple functional modules. It can be connected to multiple hardware via a data cable to realize multiple functions in a combination of software and hardware.

[0057] like Figure 1 、 Figure 3 As shown, this embodiment also includes a user identification system 100 for identifying each user's identity information. User identification system 100 includes an access control device 101 and an identification device 102. The enclosed space 1 described in this embodiment is not an absolutely sealed space, but rather a relatively closed space. Enclosed space 1 has one or more entrances and exits, preferably a user entrance 103 and a user exit 104. All users enter enclosed space 1 through user entrance 103 and leave enclosed space 1 through user exit 104.

[0058] Each entrance and exit of the enclosed space 1 is equipped with an access control device 101, preferably an automatic gate. The identity recognition device 102 is used to obtain the user's identity information and includes a code scanning device 1021 connected to the data processing device 7 and an identity acquisition unit 1022 within the data processing device 7. The code scanning device 1021 is installed inside or outside the access control device 101 at the user entrance 103, preferably on the outer surface of the automatic gate, and is used to scan an identity recognition code, preferably a QR code; the identity acquisition unit 1022 is a functional module in the data processing device 7 that can obtain the user's identity information based on the identity recognition code. The access control device 101 at the user exit 104 does not need to be equipped with an identity recognition device 102.

[0059] In this embodiment, each user downloads a dedicated application (APP) for use with the unmanned supermarket to a mobile communication terminal (such as a mobile phone or tablet), registers an account in the application, and links it to a payment software. Alternatively, each user downloads a payment software (such as WeChat / Alipay) to a mobile communication terminal, embeds a mini-program for use with the unmanned supermarket in the payment software, and registers an account in the payment software. The dedicated application (APP) or payment software contains user registration information and electronic payment information, including user identity information, bank account information, payment password, etc. After registration is completed, the user identity information is stored in the user database of the data processing device 7.

[0060] The application software (APP) in the mobile communication terminal can generate a QR code, which stores the user's identity information, etc. When a user needs to enter the closed space 1 from the outside, the QR code generated by the application software is facing the scanning end of the scanning device 1021. After scanning, the scanning device 1021 decodes the QR code and transmits the decoding result to the data processing device 7. If the QR code is recognizable and the recognized identity information matches the identity information pre-stored in the user database, it is determined that the user's identity is legal, and the access control device 101 opens to allow the user to enter the closed space 1. The access control device 101 at the user entrance 103 is provided with a sensing device, such as an infrared sensor. After the user enters the closed space 1, the access control device 101 senses that someone has walked through the access control and then automatically closes. When the user finishes shopping and needs to leave the enclosed space 1, the access control device 101 at the user exit 104 senses someone approaching the access control device 101 from inside the enclosed space 1, and the access control device will automatically open. After the user leaves the enclosed space 1, the access control device 101 senses someone walking through the access control and then automatically closes.

[0061] After the identity acquisition unit 1022 obtains the user's identity information based on the identification code, the data processing device 7 can generate a shopping database for the user. During the user's shopping process, the data processing device 7 can update the shopping database by acquiring shopping information based on each purchase made by the user. Because the user's mobile communication terminal exchanges data with the data processing device 7 in real time via an application (APP), the user's shopping database can also be displayed in the application (APP) on the mobile communication terminal, forming a shopping cart interface, allowing the user to view their shopping history and facilitate subsequent settlement.

[0062] like Figure 4As shown, this embodiment also includes a target positioning system 200 for obtaining the real-time position of each target in the enclosed space 1. The target is the entire or partial user and its extensions, and is used to obtain the coordinate set of the entire user or a partial user (such as the head or hand). The target positioning system 200 includes a three-dimensional image acquisition device 201 connected to the data processing device 7 and a target coordinate acquisition unit 202 located within the data processing device 7.

[0063] like Figure 5 As shown, the 3D image acquisition device 201 includes at least one image sensor 2011 for capturing at least one frame of 3D image in real time. Image sensors 2011 are evenly distributed across the top of the enclosed space 1, with their lenses facing downward. The central axis of the lens can be perpendicular to the horizontal plane or at a certain angle. The field of view of image sensor 2011 covers the entire bottom surface of the enclosed space 1. When a user moves around or shops in the unmanned supermarket, they are always under the surveillance of the image sensor. The 3D image captured by the image sensor includes the user's image, which refers to the entire or partial user's body and its extensions. If the enclosed space is unoccupied, the 3D image at each moment is identical to the previous one, and the 3D image at that moment can be determined to be the background, not including any user image.

[0064] Each image sensor 2011 includes a depth image sensor 2012 and an RGB image sensor 2013 arranged in parallel, and a 3D image integration unit 2014. The depth image sensor 2012 continuously captures multiple frames of depth images, and the RGB image sensor 2013 continuously captures multiple frames of RGB images. The 3D image integration unit 2014 combines a depth image frame and an RGB image frame captured at the same time into a 3D image frame.

[0065] The above two sensors collect data synchronously (collect data simultaneously) and the collection frequency is 30 frames per second. The image sensor 2011 can obtain 30 frames of RGB images and 30 frames of depth images per second. The three-dimensional image integration unit 2014 can continuously obtain 30 frames of three-dimensional images per second and transmit them to the target object coordinate acquisition unit 202 of the data processing device 7.

[0066] The target coordinate acquisition unit 202 is a functional module within the data processing device 7. It establishes a three-dimensional coordinate system within the enclosed space and acquires the user's coordinate set or coordinates within the three-dimensional coordinate system in real time based on multiple consecutive frames of three-dimensional images including the user's image. The target coordinate acquisition unit 202 includes a coordinate system establishment unit 2021, a parameter acquisition unit 2022, a background removal unit 2023, and a target coordinate calculation unit 2024. The coordinate system establishment unit 2021 establishes the three-dimensional coordinate system within the enclosed space. Preferably, the center point of the enclosed space bottom surface (the floor of the unmanned supermarket) is selected as the coordinate system origin, with the X and Y axes set horizontally and the Z axis set vertically.

[0067] Since the entire or partial body of the user occupies a large space in the three-dimensional coordinate system, a coordinate set can be used to represent the user's position. If precise position control and calculation convenience are taken into account, the coordinates of a specific point in the coordinate set can also be used to represent the user's position. For example, the coordinates of the highest point in the user's coordinate set (the point with the largest Z-axis value) can be used to represent the user's position.

[0068] The parameter acquisition unit 2022 processes multiple consecutive frames of 3D images, including the user image, to obtain the position and color parameters of each pixel in each frame. The position parameters are x, y, and z, representing the position coordinates of the pixel in the 3D coordinate system; the color parameters are r, g, and b, representing the intensities of the three primary colors of the pixel. When a user moves into the field of view of any image sensor, the data processing device 7 can acquire multiple frames of 3D images at a rate of 30 frames per second. Each frame includes the user image and the background image, and each pixel may be part of either the user or the background.

[0069] In three-dimensional images captured by different image sensors, pixels at the same location on the user's body and its extensions have the same color parameters r, g, and b. Because the distances between the image sensors at different locations and the user vary, the primary position parameters directly captured by each image sensor are the coordinates of a point on the user's body and its extensions relative to that image sensor. Therefore, a coordinate transformation is required to convert the primary position parameters captured by the image sensors at different locations into position parameters within the three-dimensional coordinate system established within the enclosed space.

[0070] The parameter acquisition unit 2022 includes a sensor coordinate acquisition unit 20221, a relative coordinate acquisition unit 20222, and a coordinate correction unit 20223. The sensor coordinate acquisition unit 20221 acquires the coordinates of the center point of the image sensor that captures the frame of three-dimensional image (i.e., the midpoint of the line connecting the center points of the lenses of the parallel depth image sensor 2012 and the RGB image sensor 2013) in the three-dimensional coordinate system established in the closed space. The relative coordinate acquisition unit 20222 establishes a second three-dimensional coordinate system with the center point of the image sensor as a second origin, with the directions of its X-axis, Y-axis, and Z-axis being the same as those of the three-dimensional coordinate system, and acquires the coordinates of each pixel point in the three-dimensional image in the second three-dimensional coordinate system. The coordinate correction unit 20223 is configured to calculate and correct the coordinates of each pixel point in the three-dimensional image in the three-dimensional coordinate system based on the coordinates of the center point of the image sensor in the three-dimensional coordinate system and the coordinates of each pixel point in the three-dimensional image in the second three-dimensional coordinate system, thereby obtaining the position parameters of each pixel point of the user and its extension.

[0071] In M consecutively acquired 3D image frames, each of which includes only one user's image, if N pixels belonging to different 3D images and having the same position parameters have the same color parameters, and N is greater than 0.9*M and less than or equal to M, the background removal unit 2023 determines that these N pixels are background pixels and removes these N background pixels from the M 3D image frames to obtain M frames of background-free 3D images, which are the user's images. In consecutively acquired 3D images, if pixels belonging to different 3D images and having the same position parameters have the same color parameters, or a majority (e.g., 90%) of the same color parameters, the pixel's location is considered background, and the pixel can be removed from the corresponding 3D image.

[0072] In the target object coordinate calculation unit 2024, if the target object is the user and its entire extension, the set of position parameters of all pixels in the M frames of background-free 3D images is the complete coordinate set of the user and its entire extension. Within this coordinate set, the position parameter of the pixel with the largest parameter z is defined as the coordinates of the user. In the continuously acquired 3D images, after removing background pixels, the remaining pixels represent the user's entire trajectory. If each of the M frames of continuously acquired 3D images includes images of multiple users, it is necessary to first capture a 3D image containing all or part of only one user from each of the M frames.

[0073] If the target object is a part of the user and its extension, the coordinate set of the user's part can be obtained, such as the head, shoulders, elbows, wrists, hands, etc. The depth image sensor 2012 and the RGB image sensor 2013 are each equipped with a lens. If the central axis of the two lenses is set perpendicular to the horizontal plane, the two lenses will look down at the goods and users in the enclosed space. Under normal circumstances, the two lenses can capture the position coordinate set of the user's head and shoulders. When the user extends his hand, the position coordinate set of the user's arm, elbow, wrist, and hand can also be captured. If the user's head, shoulders, elbows, wrists, and hands at a certain moment are connected into a broken line or curve, a correspondence can be established between the user's hand and head positions. In other words, the position of a hand can be obtained in real time, and it can also be determined which user the hand belongs to.

[0074] Furthermore, the field of view of image sensor 2011 can also cover part of the space outside the entrance or exit. When a user is outside the entrance or exit, the user's image can be captured by image sensor 2011. The entire process of a user using the unmanned vending system, including identity recognition at the entrance or exit, entering enclosed space 1, walking or staying within enclosed space 1, and leaving enclosed space 1, is monitored by image sensor 2011. This allows real-time monitoring of the real-time location of a user with a known identity and a part of their body within enclosed space 1. When the code scanning device 1021 reads the user's QR code, data processing device 7 can obtain their identity information. Image sensor 2011 begins locating and tracking the user's position in real time from the moment the code scanning device 1021 reads the code, monitoring whether the user matches a specific shelf. If image sensor 2011 is unable to obtain a real-time three-dimensional image of the user, it can be determined that the user has completed their shopping and settlement can be performed.

[0075] like Figure 6 As shown, this embodiment relates to a product perception system 400 based on image monitoring, which includes a sample collection unit 401, a model training unit 402, a real-time image collection unit 403 and a product category acquisition unit 404. The above four units are functional modules in the data processing device 7. The product perception system 400 based on image monitoring can monitor the real-time image of the space in front of the shelf and determine the type of goods taken away or put back.

[0076] The product perception system 400 based on image monitoring also includes a first camera 405 and a second camera 406. The first camera 405 is connected to the sample acquisition unit 401 in the data processing device 7 and is used to capture images of each product at multiple angles and distances. The second camera 406 is connected to the real-time image acquisition unit 403 in the data processing device 7 and is used to capture real-time images of the space in front of a shelf.

[0077] like Figure 7 As shown, preferably, there are two or four second cameras 406, which are arranged on the outside of the shelf 2, with each second camera 406 facing a corner of the shelf 2. The front ends of the multiple shelf plates 5 of the shelf 2 are located on the same plane, which is called the shelf plane. The second camera 406 is provided with a lens, and the field of view of the lens covers the space in front of the shelf. When goods are removed from the shelf or placed on the shelf, the image of the goods being removed or placed back is captured by the second camera. The space in front of the shelf refers to the spatial area in front of the shelf corresponding to the shelf plane. The space in front of the shelf generally refers to an area 30 to 50 cm wide in front of the shelf. The lens of each second camera 406 faces the center area of ​​the space in front of the shelf.

[0078] Preferably, the angle between the central axis of the lens of the second camera 406 and the horizontal plane is 30-60 degrees; and / or, the distance between the lens of the second camera 406 and the upper or lower end of the shelf 2 is 0.8-1.2 meters; and / or, the distance between the lens of the second camera 406 and one side of the shelf 2 is 0.8-1.2 meters, ensuring that the field of view of the second camera 406 can completely cover the space in front of the shelf, when the goods are removed from the shelf 2 or placed on the shelf 2, the image of the removal process or the placement process is captured by the second camera 406.

[0079] like Figure 6 As shown, the sample collection unit 401 is used to collect at least one set of image samples, each set of image samples including multiple sample images of a single product at multiple angles. A set of image samples of the same type of product is assigned the same group identifier, which represents the type of product to which the set of image samples corresponds. Preferably, the first camera 405 captures 3,000 to 5,000 images of each product on shelf 2 at different angles and distances, and transmits these images to the sample collection unit 401 of the data processing device 7. Some of these images are of the product in isolation, some are of the product with a background, some are of the product being held in a person's hand, and some are of multiple similar products stacked together. Since the products sold in the unmanned supermarket involved in this embodiment are standard products, the six-sided images of the same type of product are identical or similar. Therefore, simply selecting one or several products from the same type and photographing them multiple times can complete the sampling of training samples for that type of product.

[0080] The model training unit 402 is used to train a convolutional neural network (CNN) model based on each sample image and its corresponding group identifier in multiple sets of image samples to obtain a product recognition model. A convolutional neural network (CNN) is a feedforward neural network that excels at processing large images. Preferably, the CNN model in this embodiment is the Faster RCNN network model, which currently offers the lowest computational complexity and fastest response speed. This model has a maximum response speed of approximately 0.2 seconds, enabling accurate identification of the type of item in an image in a very short time. If the model is trained on a small number of samples or the sample clarity is low, a grouping model obtained from a single training run may have significant errors in determining the type of product displayed in an image. Therefore, a cross-validation process is recommended to obtain a more accurate model. The model training unit 402 includes a sample classification unit 4021, a grouping model training unit 4022, and a cross-validation unit 4023.

[0081] The sample classification unit 4021 randomly divides the multiple sets of image samples into two categories: training samples and test samples. The grouping model training unit 4022 inputs each sample image and its group identifier from the multiple sets of training samples into a convolutional neural network model to obtain a grouping model after training. The cross-validation unit 4023 verifies the grouping model based on each image and group identifier from the multiple sets of test samples and calculates the model accuracy. When the accuracy of the grouping model is less than a preset threshold (e.g., 90%), the model is returned to the sample classification unit. When the accuracy of the grouping model is greater than or equal to the preset threshold (e.g., 90%), the grouping model is considered a product recognition model. If the model is trained with a small number of samples or the samples are of low clarity, the grouping model obtained from only one training session may have large errors when determining the type of product displayed in the image. Therefore, it is best to include a cross-validation unit.

[0082] If the model training samples are sufficient and the samples are of high definition, a single training run can be performed using the Faster RCNN network model to train a grouping model that can effectively identify the types of goods displayed in the image. In other embodiments, the model training unit 402 may include only a grouping model training unit 4022, which uses all or part of the multiple sets of image samples collected by the sample collection unit 401 as training samples, inputs each sample image and its group identifier into the convolutional neural network model, and obtains a grouping model after training. This grouping model is the product recognition model.

[0083] The grouping model training unit 4022 includes a feature extraction unit 40221, a candidate region generation unit 40222, a candidate region mapping unit 40223, and a classifier generation unit 40224. The feature extraction unit 40221 inputs each training sample image into a convolutional neural network to extract features and obtain a feature map. This feature map corresponds to the region in the training image that displays all or part of the product. The candidate region generation unit 40222 inputs each training sample image into a candidate region network (RPN) to generate multiple candidate regions. The candidate region mapping unit 40223 maps each training sample image candidate region onto the feature image of the last convolutional layer of the convolutional neural network. The classifier generation unit 40224 collects the feature images and candidate regions of multiple training sample images, calculates the feature images of the candidate regions, and inputs them into the classifier network to generate a classifier. In the classifier network, the feature images of all training sample images corresponding to the same product are associated with the group identifier of the product. The group identifier is the type of product corresponding to the group of image samples.

[0084] The cross-validation unit 4023 includes a test sample input unit 40231 and an accuracy calculation unit 40232. The test sample input unit 40231 inputs each image of multiple test samples into the grouping model to obtain the test group identifiers of the multiple test samples; the accuracy calculation unit 40232 compares the test group identifiers of the multiple test samples with the group identifier of the test sample, and calculates the ratio of the number of identical identifiers to the number of test samples, which is the accuracy of the primary data model. Preferably, the preset threshold is set to 90%. When the accuracy of the grouping model is less than 90%, the sample classification unit 4021 regroups and retrains the samples; when the accuracy of the grouping model is greater than or equal to 90%, it can be considered that the grouping model is a product recognition model.

[0085] The real-time image acquisition unit 403 is used to continuously capture at least one real-time image of the space in front of the shelf, and each real-time image includes part or all of one or more pictures of goods. The real-time image acquisition unit 403 is connected to multiple second cameras 406, which are arranged at the edge of the shelf to take pictures of the area in front of the shelf in real time. When a user takes goods from a shelf, or when a user places goods or items on a shelf, the second camera 406 can take a picture of the goods in front of the shelf, which includes a full or partial photo of the goods, showing the shape, pattern and color of the goods. Preferably, the shooting speed of the second camera is 30 frames per second, which quickly obtains multiple real-time pictures of the goods and transmits them to the real-time image acquisition unit 403.

[0086] In the multiple frames of images acquired continuously, the distance between the goods and the edge of the shelf in each frame is different. The data processing device 7 can calculate the change in the distance to determine the change in the distance between the goods and the shelf. If the distance becomes larger, it can be considered that the goods are taken away from the shelf. If the distance becomes smaller, it can be considered that the goods are placed on the shelf.

[0087] Assume that a user holds an item in front of a shelf for one second. The two second cameras 406 capture images at 30 frames per second, capturing 60 real-time images from different angles within one second. Each image may include the user's hand and a portion of the item. If the real-time images do not contain any item content, for example, if some items are small, only the user's hand may be shown in some images, or only the background image may be shown. Such images can be filtered out. This embodiment is not suitable for extremely small items. If the item is completely covered by the user's hand, the item type cannot be identified.

[0088] The second camera 406 can be a device that runs continuously for a long time, or an infrared switch can be set on the shelf board and connected to the second camera 406. It is usually in a power-off state. When a user walks near the shelf, the infrared switch senses the heat and connects, and the second camera 406 is powered on and started; when the user leaves, the infrared switch no longer senses the heat and is disconnected, and the second camera 406 is powered off and turned off.

[0089] The product category acquisition unit 404 is used to acquire the type of the product displayed in the real-time image based on the real-time image and the product recognition model. The product category acquisition unit 404 includes a group identification acquisition unit 4041 and an identification credibility calculation unit 4042.

[0090] The group identification acquisition unit 4041 obtains multiple real-time images containing a particular product and inputs them into the product recognition model. The model then outputs multiple group identifications corresponding to the multiple real-time images as a likelihood conclusion. The group identification acquisition unit 4041 includes a second feature extraction unit 40411, a second candidate region generation unit 40412, a second candidate region mapping unit 40413, and a second group acquisition unit 40414. The second feature extraction unit 40411 inputs a real-time image into a convolutional neural network to extract features and obtain a feature image. The second candidate region generation unit 40412 inputs the real-time image into the candidate region network to generate multiple candidate regions. The second candidate region mapping unit 40413 maps the candidate regions of the real-time image onto the feature image of the last layer of the convolutional neural network. The group acquisition unit 40414 collects the feature images and candidate regions of the multiple images, calculates the feature images of the candidate regions, and inputs them into the classifier network to obtain the group identification corresponding to the real-time image. After each real-time image is input into the product recognition model, the group identifier corresponding to the image can be determined. If the two second cameras capture 60 images at a speed of 30 frames per second, and each image shows a certain product, and it is input into the product recognition model, 60 group identifiers can be obtained.

[0091] The identifier credibility calculation unit 4042 calculates the ratio of the number of each group identifier in the likelihood conclusion to the total number of group identifiers in the likelihood conclusion. This ratio is the credibility of each group identifier. The type and / or quantity of the product corresponding to the group identifier with the highest credibility is the type and / or quantity of the product displayed in the real-time image. Comparing the 60 group identifiers described above, if, among the 60 product categories corresponding to the 60 group identifiers, product A appears 36 times, product B appears 18 times, and product C appears 6 times, then the credibility of these three is 60%, 30%, and 10%, respectively. Therefore, the product type displayed in the real-time image can be determined to be product A with the highest credibility. If the image samples collected by the sample collection unit 401 include multiple photos of similar products stacked together, the product category acquisition unit 404 can further determine the quantity of the products.

[0092] The beneficial effect of the product perception system 400 based on image monitoring is that it can monitor the image of the space in front of the shelf in real time, determine whether any goods have been taken away from the shelf or put back on the shelf, use the convolution algorithm in machine learning to infer the possible conclusions about the type of goods, and select the result with the highest credibility as the final conclusion.

[0093] like Figure 8As shown, this embodiment also includes a shopping user identification system 500, which is a functional module in the data processing device 7. When any type of goods is taken away or returned, the identity of the user who took or returned the goods is obtained based on the user's identity information and the user's real-time location. Shopping user identification system 500 includes a product information storage unit 501, a shelf coordinate storage unit 502, a shelf and user matching determination unit 503, and a product and user matching determination unit 504.

[0094] The data processing device 7 is provided with a product database, which is stored in the product information storage unit 501. The product database includes information about each product; the product information includes the product name, model, net content and unit price of each product, as well as the shelf number where the product is placed, the rack number where the product is placed and the product number.

[0095] The target object positioning system 200 establishes a three-dimensional coordinate system in the closed space. Since the positions of the shelves 2 and the shelves 5 are determined, the coordinates of each shelf 2 and each shelf 5 can be obtained after the coordinate system is established. The shelf coordinate set and the shelf coordinate set are stored in the shelf coordinate storage unit 502. The height of the shelf space above the shelf for placing goods is set (such as 30CM), and the coordinate set of the shelf space can be obtained.

[0096] The user coordinate acquisition unit 202 can obtain the real-time coordinate set of the hand of each known user. When the coordinate set of the shelf space above a shelf intersects with the coordinate set of a user's hand, the shelf and user matching judgment unit 503 determines that the shelf matches the user, and it can be considered that the user has extended his hand into the shelf space above the shelf.

[0097] The target object positioning system 200 establishes a three-dimensional coordinate system in the closed space. Since the positions of the shelves 2 and the shelves 5 are determined, the coordinates of each shelf 2 and each shelf 5 can be obtained after the coordinate system is established. The shelf coordinate set and the shelf coordinate set are stored in the shelf coordinate storage unit 502. The height of the shelf space above the shelf for placing goods is set (such as 30CM), and the coordinate set of the shelf space can be obtained.

[0098] The user coordinate acquisition unit 202 can obtain the real-time coordinate set of the hand of each known user. When the coordinate set of the shelf space above a shelf intersects with the coordinate set of a user's hand, the shelf and user matching judgment unit 503 determines that the shelf matches the user, and it can be considered that the user has extended his hand into the shelf space above the shelf.

[0099] The product sensing system 400, which uses image monitoring, monitors the real-time image of the space in front of the shelf using a second camera 406. It can determine the product's access status and the type of product being taken or returned based on the changes in the distance between the product and the shelf in multiple frames of continuously acquired images. The access status includes the product being at rest, being taken, and being returned.

[0100] When goods are taken from or placed on a shelf, and at the same time a user matches the shelf, the goods and user matching judgment unit 504 determines that the goods match the user, and the goods are taken from or placed on the shelf by the user at this moment, thereby determining the identity of the user who took the goods or put the goods back.

[0101] like Figure 9 As shown, this embodiment also includes a shopping information recording unit 600, which is a functional module in the data processing device 7. It generates at least one shopping database based on each user's identity information to record the type and quantity of at least one item taken by each user. The shopping information recording unit 600 includes a shopping database generation unit 601 and a shopping database update unit 602.

[0102] When a user's identity is recognized by the user identity recognition system 100, the identity acquisition unit 1022 obtains the user's identity information, and the shopping database generation unit 601 generates the user's shopping database in the data processing device 7 based on the user's identity information. The shopping database in the initial state does not have any shopping information.

[0103] The shopping database updating unit 602 generates a set of shopping information based on the type and quantity of the removed goods and the identity information of the user who removed the goods, and stores it in the user's shopping database. The shopping information includes the type and quantity of the goods removed at that moment, as well as product information such as the name, model, net weight, and unit price. After the user removes goods multiple times within the enclosed space 1, their shopping database will contain multiple sets of shopping information. Because the user's mobile communication terminal is connected to the data processing device 7 via wireless communication and exchanges data, the shopping information in the shopping database can also be displayed on the user's mobile communication terminal app interface, forming the user's electronic shopping cart.

[0104] When the product matches a user, if the product sensing system 400 based on image monitoring detects that an item is placed on the shelf, the type of the item can be determined.

[0105] The shopping database is queried for each shopping information to determine whether the category value of the purchased goods matches the category of the item, that is, to determine whether there are one or more purchased goods in the user's shopping database that are the same as the goods placed on the shelf.

[0106] If the product sensing system 400 based on image monitoring can also determine whether the type of the returned product is consistent with the type of products originally on the shelf, if not, an alarm signal can be optionally generated to alert the management staff or user that the product has been misplaced. If the product sensing system 400 based on image monitoring cannot determine the type of the returned product, it can be confirmed that the item returned to the shelf is not an existing product in the unmanned supermarket, and may be a user-brought item, such as an umbrella or a mobile phone. In this case, an alarm signal can be optionally generated. If necessary, the shelf number of the shelf can be displayed on a display to alert the management staff or user.

[0107] In other embodiments, the unmanned vending system may further include a product sensing system based on weight monitoring, in which only one type of product is placed on each shelf, and a weight sensor is provided inside the shelf to sense the weight changes of each shelf in real time. By working in conjunction with the product sensing system based on image monitoring in this embodiment, it can more accurately judge the status of product placement, the type and quantity of the products placed and taken.

[0108] like Figure 10 As shown, this embodiment also includes a settlement system 700, a functional module within the data processing device 7, for settling purchases based on the types and quantities of all items in the user's shopping database. After a user completes their shopping, they can exit the enclosed space 1 via the access control device at the entrance. When the image sensor 211 of the user positioning system 200 is unable to capture a real-time 3D image of the user, the user's shopping is deemed complete, and the settlement system 700 settles the purchase for the user.

[0109] The settlement system 700 includes a total amount calculation unit 701 and a payment unit 702. When the user leaves the enclosed space, the total amount calculation unit 701 calculates the total amount based on the types and quantities of all goods in the user's shopping database. Since the unit price of each type of goods is pre-stored as product information in the data processing device 7, the total amount the user needs to pay is the sum of the product of the unit price and quantity of the multiple goods. Furthermore, in other embodiments, the user can enjoy discounts on goods or use coupons, vouchers, etc., and the total amount the user needs to pay is the sum of the product of the unit price and quantity of the multiple goods minus the coupon and / or voucher amount and / or discount amount. The payment unit 702 is the payment software included with the settlement system 700 or third-party payment software, which can deduct funds from the user's bank account or electronic account, and the amount deducted is the same as the total amount the user needs to pay.

[0110] like Figure 11 As shown, this embodiment also provides a product perception method based on image monitoring, namely, a method for implementing the aforementioned product perception system 400 based on image monitoring, comprising the following steps: step S201) sample collection, step S202) model training, step S203) real-time image collection, and step S204) product category acquisition. Since the products sold in the unmanned supermarket involved in this embodiment are standard products, the six-sided images of the same product type are identical. Therefore, simply selecting one or several products from the same product type and photographing them multiple times can complete the sampling of training samples for that product type. Prior to steps S201)-S204), the product perception method based on image monitoring further includes steps S101) shelf setup, S102) first camera setup, and S102) second camera setup. Step S101) Shelf Setup: Set up at least one shelf and place at least one product on it. Step S102) First Camera Setup: Set up a first camera to capture images of each product at multiple angles and distances, including six-sided views of each product. Step S103) Second Camera Setup: Set up multiple second cameras to capture real-time images of the space in front of a shelf. The second cameras' fields of view cover the space in front of the shelf, capturing images of products being removed or replaced.

[0111] Step S201) Sample collection step, used to collect multiple sets of image samples, each set of image samples includes multiple sample images of a product at multiple angles. A set of image samples corresponding to the same type of product is assigned the same group identifier, which is the type of product corresponding to the set of image samples. During the sample collection step, images of each product are taken at multiple angles and distances at a rate of 30 frames per second. Preferably, the number of images is 3,000 to 5,000. Too many images will be too costly, while too few will result in large model errors. 5,000 images are taken of each product at different angles and distances. Some are of the product taken independently, some are of the product with a background, some are of the product being held in a person's hand, and even images of multiple similar products stacked together are taken.

[0112] Step S202) Model training step, used to train a convolutional neural network model based on each sample image and its group identifier in the multiple groups of image samples to obtain a product recognition model.

[0113] If the model is trained with fewer samples or the sample clarity is low, the grouping model obtained by training only once may have a large error in judging the type of goods shown in the picture, so it is best to have a cross-validation step. Figure 12 As shown, step S202) the model training step includes step S2021) the sample classification step, step S2022) the grouping model training step and step S2023) the cross-validation step.

[0114] Step S2021) Sample classification is used to randomly divide the multiple sets of image samples into two categories: training samples and test samples. In the aforementioned step S201) Sample collection, 4,000 images of each product are captured at different angles and distances at a rate of 30 frames per second. These 4,000 images of each product are randomly divided into two groups: training samples and test samples, each containing approximately 2,000 images.

[0115] Step S2022) The grouping model training step is used to input each sample image and the group identifier of each sample image of the multiple training samples into the convolutional neural network model, and obtain the grouping model after training. Figure 13As shown, step S2022) grouping model training includes step S20221) feature extraction, step S20222) candidate region generation, step S20223) candidate region mapping, and step S20224) classifier generation. Step S20221) feature extraction involves inputting each training sample image into a convolutional neural network (CNN) for feature extraction, resulting in a feature map corresponding to the area in the training image showing all or part of the product. For example, for 2,000 images of a particular product, a feature map related to the entire product or a portion of the product is identified in each image. For color images, each pixel's RGB primary colors correspond to a two-dimensional matrix. Each matrix is ​​convolved with a 3*3 or 5*5 convolution kernel to generate three new two-dimensional matrices, or feature maps. Step S20222) The region proposal generation step is used to input each training sample image into a region proposal network (RPN) to generate multiple region proposals. Preferably, 300 region proposals are generated for each image. The Region Proposal Network (RPN) is the proposal generation network in the Faster R-CNN model. Because the scale of objects in object detection can vary greatly, it is necessary to generate region proposals of as many different sizes as possible. Step S20223) The region proposal mapping step first converts the region proposals of different sizes into images of the same size. Then, the region proposals for each training sample image are mapped to the feature map of the last convolutional layer of the convolutional neural network. Step S20224) The classifier generation step is used to collect the feature maps and region proposals from multiple training sample images, calculate the proposal feature maps, and input them into the classifier network to generate the classifier. In the classifier network, the feature images of all training sample images corresponding to the same product are associated with the group identifier of the product. The group identifier is the type of product corresponding to the group of image samples.

[0116] Step S2023) Cross-validation step, used to verify the grouping model based on each image of multiple test samples and the group identification of each test sample group, and calculate the model accuracy. Figure 14As shown, step S2023) the cross-validation step includes step S20231) a test sample input step and step S20232) an accuracy calculation step. Step S20231) the test sample input step is used to input each image of multiple test samples into the grouping model to obtain the test group identification of multiple test samples. Step S20232) the accuracy calculation step is used to compare the test group identification of multiple test samples with the group identification of the test sample, and calculate the ratio of the number of identical identifications to the number of test samples. This ratio is the accuracy of the primary data model. When the accuracy of the grouping model is less than a preset threshold, the method returns to the sample classification step; when the accuracy of the grouping model is greater than or equal to the preset threshold, the grouping model is a product identification model. Preferably, the preset threshold is set to 90%. When the accuracy of the grouping model is less than 90%, return to the sample classification step, regroup the samples, and retrain; when the accuracy of the grouping model is greater than or equal to 90%, it can be considered that the grouping model is a product identification model.

[0117] If the model has a sufficient number of training samples and the samples are of high definition, a single training run can produce a grouping model using the Faster RCNN network model. This grouping model can effectively identify the types of products displayed in images. Step S202) Model Training can include Step S2022) Grouping Model Training. All or part of the multiple sets of image samples collected in Step S201) Sample Collection are used as training samples. Each training image and its group identifier are input into the convolutional neural network model. After training, a grouping model is obtained. This grouping model serves as the product recognition model.

[0118] Step S203) A real-time image capture step is used to continuously capture at least one real-time image, each of which includes a portion or the entire image of an item. Step S203) The real-time image capture step includes a second image capture step, capturing multiple images of each item at a rate of 30 frames per second, with the number of captures ranging from 10 to 200. A second camera 206 is positioned at each of the four corners of the shelf. Each second camera 206 has a field of view covering the area in front of the shelf, and the lens of each second camera 206 faces the center of the shelf plane. When a user reaches out to remove an item from the shelf or places it back on the shelf, the four second cameras 206 capture images of the entire item or a portion of the item from different angles. Assuming a user holds an item in front of the shelf for one second, and the four second cameras 206 capture images at 30 frames per second, the four second cameras 206 can capture 120 real-time images from different angles within one second. Each image can include the user's hand and a portion of the item. The second camera 206 can be a device that is powered on or started for a long time, or an infrared switch can be set on the shelf board and connected to the second camera 206. It is usually in a power-off state. When a user walks near the shelf, the infrared switch senses the heat and connects, and the second camera 206 is powered on and started; when the user leaves, the infrared switch no longer senses the heat and is disconnected, and the second camera 206 is powered off and turned off.

[0119] Step S204) The step of obtaining the product category is used to determine the type of product displayed in the real-time image based on the real-time image and the product recognition model. Figure 15 As shown, step S204) the category determination step also includes step S2041) a group identifier acquisition step and step S2042) an identifier credibility calculation step. Step S2041) the group identifier acquisition step is used to input multiple real-time images into the product recognition model to obtain multiple group identifiers corresponding to the multiple real-time images. Step S2042) the identifier credibility calculation step calculates the ratio of the number of each group identifier to the total number of group identifiers. This ratio represents the credibility of each group identifier. The category of the product corresponding to the group identifier with the highest credibility is the category of the product displayed in the real-time image. After step S204) is completed, the process returns to step S203) the real-time image capture step to capture images of the product the next time a product is removed or returned.

[0120] like Figure 16As shown, step S2041) group identification acquisition step includes: step S20411) second feature extraction step, step S20412) second candidate region generation step, step S20413) second candidate region mapping step, and step S20414) group acquisition step. Step S20411) second feature extraction step is used to input a real-time image into a convolutional neural network (CNN) to extract features and obtain a feature image feature map. Step S20412) second candidate region generation step is used to input the real-time image into a region candidate network (RPN) to generate multiple candidate regions (region proposals), with 300 candidate regions generated for each real-time image. Step S20413) second candidate region mapping step is used to map the candidate regions (region proposals) of the real-time image onto the feature map (feature map) of the last layer of the convolutional neural network. Step S20414) The group acquisition step is used to collect feature maps and region proposals from multiple images, calculate proposal feature maps for the candidate regions, and feed them into a classifier network to obtain the group identifier corresponding to the real-time image.

[0121] Step S2042) Calculating Identifier Credibility: This step calculates the ratio of the number of each group identifier in the likelihood conclusion to the total number of group identifiers in the likelihood conclusion. This ratio represents the credibility of each group identifier. The type and / or quantity of the product corresponding to the group identifier with the highest credibility is the type and / or quantity of the product displayed on the real-time image. Comparing the 60 group identifiers described above, if, among the 60 product categories corresponding to the 60 group identifiers, product A appears 36 times, product B appears 18 times, and product C appears 6 times, then the credibility of these three categories is 60%, 30%, and 10%, respectively. Therefore, the product type displayed on the real-time image can be determined to be product A, which has the highest credibility. If the image samples collected in step S201) include multiple photos of similar products stacked together, step S204) Acquiring Product Categories can further determine the quantity of the products.

[0122] The product perception system and product perception method based on image monitoring described above can determine the pick-up and placement status of the products on the shelf and the specific types of products taken away or put back based on the real-time picture of the space in front of the shelf. If combined with the product perception system based on weight monitoring, the quantity of products can be accurately determined. If combined with user identity recognition technology and user location tracking technology, the identity of the user of the products taken away or put back can also be accurately determined, thereby accurately adding or deleting shopping records in the user's shopping database so that the user can automatically settle the bill after the shopping is completed.

[0123] The foregoing description is merely a preferred embodiment of the present invention, which will enable those skilled in the art to more clearly understand how to practice the present invention. These embodiments are not intended to limit the scope of the present invention. Those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A product perception system based on image monitoring, characterized in that: include: enclosed spaces; At least one shelf is disposed in the enclosed space, the shelf comprising at least one shelf board, and at least one product is placed on the shelf board of the shelf; A user identification system for identifying the identity information of each user entering the enclosed space; At least one image sensor is evenly distributed on the top of the enclosed space, and the field of view of the image sensor covers the entire bottom surface of the enclosed space; the image sensor is used to capture at least one frame of three-dimensional image in real time, the three-dimensional image includes an image of the user, and the user image includes an image of the user's hand; At least one second camera, configured to continuously capture real-time images of the space in front of the shelf, wherein the lens of each second camera is directed toward the middle of the space in front of a shelf, and the field of view of at least one second camera covers the space in front of the shelf; At least one weight sensor, each weight sensor is installed in a shelf, for sensing the weight change of each shelf in real time, for determining the status of goods being taken or placed on each shelf, and the type and quantity of goods being taken or placed; Data processing equipment, including: a user coordinate acquisition unit, configured to establish a three-dimensional coordinate system in the enclosed space, and acquire in real time a coordinate set or coordinates of the user's hand in the three-dimensional coordinate system based on the at least one frame of three-dimensional image captured by the image sensor; a shelf and user matching determination unit, which determines that the shelf matches the user when a coordinate set of the shelf space above the shelf intersects with a coordinate set of a user's hand; A sample collection unit is used to collect multiple sets of image samples, each set of image samples includes multiple sample images of a product from multiple angles; a set of image samples of the same type of product is assigned the same group identifier, which is the type of product corresponding to the set of image samples; A model training unit, configured to train a convolutional neural network model based on each sample image and its group identifier in the plurality of groups of image samples to obtain a product recognition model; A real-time image acquisition unit, configured to continuously acquire at least one real-time image of the space in front of the shelf, each real-time image including part or all of the product image; and a product category acquisition unit configured to acquire the type and quantity of the product displayed in the real-time image based on the real-time image and the product recognition model; and calculate the change in the distance between the product and the shelf edge based on multiple frames of continuously acquired real-time images; if the distance increases, it is determined that the product has been removed from the shelf; if the distance decreases, it is determined that the product has been placed on the shelf; The product and user matching judgment unit determines that when a product is taken from a shelf or placed on a shelf, and a user is matched with the shelf at the same time, the product is determined to match the user and the user's identity is determined; A shopping information recording system generates at least one shopping database based on the identity information of each user, for recording the type and quantity of at least one item taken by each user; and The settlement system settles and completes payment for the user based on the user's shopping database when the image sensor cannot obtain a three-dimensional image of the user in the closed space.

2. The product perception system based on image monitoring according to claim 1, characterized in that: Also includes A first camera, connected to the sample collection unit, for taking pictures of each product at multiple angles and distances; The second camera is connected to the real-time image acquisition unit; the second camera is provided with a lens, the field of view of which covers the space in front of the shelf.

3. The product sensing system based on image monitoring according to claim 1, characterized in that: The number of the second cameras is two or four; The angle between the central axis of the lens of the second camera and the horizontal plane is 30-60 degrees; and / or, The distance between the lens of the second camera and the upper end or the lower end of the shelf is 0.8-1.2 meters; and / or, The distance between the lens of the second camera and one side of the shelf is 0.8-1.2 meters.

4. The product perception system based on image monitoring according to claim 1, characterized in that: The model training unit includes A grouping model training unit inputs each sample image and its group identifier of multiple groups of training samples into a convolutional neural network model, and obtains a grouping model after training; The training samples are all or part of the multiple groups of image samples, and the grouping model is a product recognition model.

5. The product perception system based on image monitoring according to claim 4, characterized in that: The model training unit includes A sample classification unit, configured to randomly divide the plurality of image samples into two categories, namely, training samples and test samples; A grouping model training unit is used to input each sample image and its group identifier of multiple groups of training samples into a convolutional neural network model to obtain a grouping model after training; as well as A cross-validation unit is used to verify the grouping model based on each image of multiple groups of test samples and the group identifier of each group of test samples, and calculate the model accuracy; when the accuracy of the grouping model is less than a preset threshold, it returns to the sample classification unit; When the accuracy of the grouping model is greater than or equal to the preset threshold, the grouping model is a product identification model.

6. The product perception system based on image monitoring according to claim 5, characterized in that: The group model training unit includes A feature extraction unit is used to input the image of each training sample into the convolutional neural network to extract features and obtain a feature image; A candidate region generation unit is used to input the image of each training sample into the candidate region network to generate multiple candidate regions; The candidate region mapping unit is used to map the candidate region of each training sample image to the feature image of the last convolutional layer of the convolutional neural network; The classifier generation unit is used to collect the image feature images and candidate regions of multiple training samples, calculate the feature images of the candidate regions, and send them to the classifier network to generate a classifier.

7. The product sensing system based on image monitoring according to claim 6, characterized in that: The cross-validation unit includes: a test sample input unit, configured to input each image of a plurality of test samples into the grouping model to obtain test group identifiers of the plurality of test samples; and The accuracy calculation unit is used to compare the test group identifiers of multiple test samples with the group identifier of the test sample, and calculate the ratio of the number of identical identifiers to the number of the test samples. The ratio is the accuracy of the grouping model.

8. The product perception system based on image monitoring according to claim 1, characterized in that: The commodity type acquisition unit also includes a group identification acquisition unit, configured to input a plurality of real-time images into the product identification model and acquire a plurality of group identifications corresponding to the plurality of real-time images as a possibility conclusion; as well as The identification credibility calculation unit is used to calculate the ratio of the number of each group identification in the possibility conclusion to the total number of all group identifications in the possibility conclusion. This ratio is the credibility of each group identification. The type and / or quantity of goods corresponding to the group identification with the highest credibility is the type and / or quantity of goods displayed on the real-time image.

9. The product perception system based on image monitoring according to claim 8, characterized in that: The group identification obtaining unit includes A second feature extraction unit is used to input a real-time image into a convolutional neural network to perform feature extraction and obtain a feature image; a second candidate region generating unit, configured to input the real-time image into a candidate region network to generate a plurality of candidate regions; A second candidate region mapping unit is used to map the candidate region of the real-time image to the feature image of the last layer of the convolutional neural network; as well as The group acquisition unit is used to collect feature images and candidate areas of multiple pictures, calculate the feature images of the candidate areas, and send them to the classifier network to obtain the group identification corresponding to the real-time picture.

10. A method for sensing goods based on image monitoring, characterized in that: The steps include: A shelf setting step is to set up at least one shelf in the enclosed space, including at least one shelf board, each shelf board is provided with a weight sensor, and at least one product is placed on the shelf board of the shelf; an image sensor arrangement step, wherein image sensors are evenly distributed on the top of the enclosed space, with the field of view of the image sensors covering the entire bottom surface of the enclosed space; the image sensors are configured to capture at least one frame of three-dimensional image in real time, the three-dimensional image including an image of the user, and the user image including an image of the user's hand; a camera setting step of setting two or more second cameras to continuously capture real-time images of the space in front of the shelf, wherein the lens of each second camera is directed toward the middle of the space in front of a shelf, and the field of view of at least one second camera covers the space in front of the shelf; A sample collection step is used to collect multiple sets of image samples, each set of image samples including multiple sample images of a product at multiple angles; a set of image samples of the same type of product is assigned the same group identifier, which is the type of product corresponding to the set of image samples; A model training step for training a convolutional neural network model based on each sample image and its group identifier in the multiple groups of image samples to obtain a product recognition model; A user identification step for identifying the identity information of each user entering the enclosed space; A user coordinate acquisition step for establishing a three-dimensional coordinate system in the closed space, and acquiring in real time a coordinate set or coordinates of the user's hand in the three-dimensional coordinate system based on the at least one frame of three-dimensional image captured by the image sensor; a step of determining whether a shelf and a user are matched, wherein when a coordinate set of a shelf space above a shelf intersects a coordinate set of a user's hand, the shelf is determined to be matched with the user; a real-time image capturing step for continuously capturing at least one real-time image of the space in front of the shelf. When a product passes through the space in front of a shelf, the second camera continuously captures multiple frames of real-time images, each frame of the real-time image showing all or part of the product and the edge of the shelf; as well as a commodity type acquisition step for acquiring the type and quantity of the commodity displayed in the real-time image based on the real-time image and the commodity recognition model; The step of determining the state of goods to be taken and placed is as follows: when a product passes through the space in front of a shelf, the change in the distance between the product and the edge of the shelf is determined based on the multiple frames of real-time images acquired continuously; If it becomes larger, it is determined that the product has been taken off the shelf; if it becomes smaller, it is determined that the product has been placed on the shelf; The product and user matching determination step: when a product is taken from a shelf or placed on a shelf, and a user is matched with the shelf at the same time, the product is determined to match the user and the user's identity is determined; A shopping information recording step, generating at least one shopping database based on the identity information of each user, for recording the type and quantity of at least one item taken by each user; as well as In the settlement step, when the image sensor cannot obtain a three-dimensional image of a user in a closed space, the user is settled and the payment is completed according to the user's shopping database.

11. The product perception method based on image monitoring according to claim 10, characterized in that: The model training step includes the following steps: In the group model training step, each sample image and its group identifier of the multiple groups of training samples are input into the convolutional neural network model, and the group model is obtained after training; The training samples are all or part of the multiple groups of image samples, and the grouping model is a product recognition model.

12. The method for sensing goods based on image monitoring according to claim 10, wherein: The model training step includes the following steps: A sample classification step for randomly dividing the plurality of image samples into two categories, namely training samples and test samples; A grouping model training step, for inputting each sample image and its group identifier of the multiple groups of training samples into a convolutional neural network model, and obtaining a grouping model after training; and The cross-validation step is used to verify the grouping model based on each image of multiple groups of test samples and the group identification of each group of test samples, and calculate the model accuracy; when the accuracy of the grouping model is less than the preset threshold, return to the sample classification step; when the accuracy of the grouping model is greater than or equal to the preset threshold, the grouping model is a product identification model.

13. The method for product perception based on image monitoring according to claim 12, wherein: The group model training step includes the following steps: The feature extraction step is used to input the image of each training sample into the convolutional neural network to extract features and obtain feature images; The candidate region generation step is used to input the image of each training sample into the candidate region network to generate multiple candidate regions; The candidate region mapping step is used to map the candidate region of each training sample image to the feature image of the last convolutional layer of the convolutional neural network; as well as The classifier generation step is used to collect the image feature images and candidate regions of multiple training samples, calculate the feature images of the candidate regions, and send them to the classifier network to generate a classifier.

14. The method for product perception based on image monitoring according to claim 12, wherein: The cross-validation step comprises the following steps: A test sample input step, for inputting each image of a plurality of test samples into the grouping model to obtain a test group identifier of the plurality of test samples; as well as The accuracy calculation step is used to compare the test group identifiers of multiple test samples with the group identifier of the test sample, and calculate the ratio of the number of identical identifiers to the number of the test samples. The ratio is the accuracy of the grouping model.

15. The method for product perception based on image monitoring according to claim 10, wherein: The step of obtaining the product category further includes the following steps: A group identification acquisition step is used to input multiple real-time images into the product recognition model to obtain multiple group identifications corresponding to the multiple real-time images as possibility conclusions; as well as The identification credibility calculation step is used to calculate the ratio of the number of each group identification in the possibility conclusion to the total number of all group identifications in the possibility conclusion. This ratio is the credibility of each group identification. The type and / or quantity of goods corresponding to the group identification with the highest credibility is the type and / or quantity of goods displayed on the real-time image.

16. The method for product perception based on image monitoring according to claim 10, wherein: The group identification acquisition step includes the following steps: The second feature extraction step is used to input a real-time image into the convolutional neural network to perform feature extraction and obtain a feature image; A second candidate region generating step is used to input the real-time image into a candidate region network to generate multiple candidate regions; A second candidate region mapping step is used to map the candidate region of the real-time image to the feature image of the last layer of the convolutional neural network; as well as The group acquisition step is used to collect feature images and candidate areas of multiple images, calculate the feature images of the candidate areas, and send them to the classifier network to obtain the group identification corresponding to the real-time image.

Images