Goods Identification Method and Device

Abstract

A method and device for identifying goods. The method includes: obtaining a first current frame image and a second current frame image respectively collected by a first camera and a second camera for a goods placement area, where the first camera and the second camera are respectively assembled at different positions outside the goods placement area; respectively identifying a first good in the first current frame image and a second good in the second current frame image, and determining a target good that is moved at the current moment according to the identification results of the first good and the second good; in the case of determining the target good type and the target movement mode of the target good, updating the quantity of the goods corresponding to the target good type and the target movement mode in a goods count list according to the quantity of the target good, where the target movement mode includes taking out from or putting into the goods placement area. This method can improve the accuracy of goods identification at a relatively low hardware cost.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and in particular, to a method and device for identifying goods. Background Art

[0002] With the rapid development of the new retail business, the vending technology based on automatic goods recognition has gradually matured. Taking the intelligent vending cabinet as an example, when the staff shelves new goods or consumers take goods, the vending cabinet needs to automatically detect the goods being moved. For this, the related technology mainly uses the following methods for automatic goods recognition:

[0003] Using RFID (Radio Frequency Identification) for recognition, this method requires pasting disposable RFID radio frequency tags on each of the goods to be sold one by one. Not only the labor cost and the tag cost are relatively high, but also the RFID radio frequency tags are easily interfered by metals, and the recognition accuracy is relatively low. Using the commodity recognition technology of static images, the static images of the goods are collected by a single-channel camera installed in the vending cabinet, and the goods put in or taken out are detected based on the image recognition technology. However, the recognition effect of this method depends on the placement method of the goods and the recognition algorithm of the camera, and it is easy to lose the goods during the tracking process due to hand occlusion, and the accuracy is still relatively low. By calculating the goods based on the increase or decrease of the weight of the goods, this method requires installing several gravity sensors under the shelves. Not only the hardware cost is relatively high, but also in the case where the weights of different types of goods are similar or in multiples, it is impossible to accurately calculate the goods put in or taken out. Not only the calculation accuracy is relatively low, but also the types of goods that the vending cabinet can place are limited.

[0004] It can be seen that the above methods of the related technology generally have the problems of high cost and low recognition accuracy, and urgent improvement is needed. Summary of the Invention

[0005] In view of this, the embodiments of the present disclosure propose a method and device for identifying goods to solve the deficiencies in the related technology.

[0006] According to the first aspect of the embodiments of the present disclosure, a method for identifying goods is proposed, including:

[0007] Obtaining a first current frame image and a second current frame image respectively collected by a first camera and a second camera for a goods placement area, where the first camera and the second camera are respectively assembled at different positions outside the goods placement area;

[0008] Identifying a first good in the first current frame image and a second good in the second current frame image respectively, and determining a target good being moved at the current moment according to the recognition results of the first good and the second good;

[0009] When the target goods type and the target moving mode of the target goods are determined, update the quantity of the goods corresponding to the target goods type and the target moving mode in the goods counting list according to the quantity of the target goods, where the target moving mode includes taking out from or putting into the goods placement area.

[0010] According to a second aspect of the embodiments of the present disclosure, a goods identification device is provided. The device includes one or more processors, and the processors are configured to:

[0011] Obtain a first current frame image and a second current frame image respectively collected by a first camera and a second camera for a goods placement area, where the first camera and the second camera are respectively assembled at different positions outside the goods placement area;

[0012] Identify a first good in the first current frame image and a second good in the second current frame image respectively, and determine the target good being moved at the current moment according to the identification results of the first good and the second good;

[0013] When the target goods type and the target moving mode of the target goods are determined, update the quantity of the goods corresponding to the target goods type and the target moving mode in the goods counting list according to the quantity of the target goods, where the target moving mode includes taking out from or putting into the goods placement area.

[0014] According to a third aspect of the embodiments of the present disclosure, an intelligent storage cabinet is provided, including:

[0015] A goods placement component corresponding to the goods placement area for placing goods; a first camera for collecting a first current frame image for the goods placement area; a second camera for collecting a second current frame image for the goods placement area; a processor; and a memory for storing instructions executable by the processor. Wherein, the processor is configured to implement the goods identification method described in the first aspect above.

[0016] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps in the goods identification method described in the first aspect above are implemented.

[0017] According to an embodiment of the present disclosure, an intelligent vending cabinet obtains a first current frame image and a second current frame image respectively captured by a first camera and a second camera for a goods placement area, where the first camera and the second camera are respectively installed at different positions outside the goods placement area; then respectively identify a first good in the first current frame image and a second good in the second current frame image, and determine a target good that is moved at the current moment according to the identification results of the first good and the second good; when the target good type and the target movement mode of the target good are determined, update the quantity of the goods corresponding to the target good type and the target movement mode in the goods count list according to the quantity of the target good, and the target movement mode includes taking out from or putting into the goods placement area.

[0018] It can be seen that this solution uses two cameras (i.e., the first camera and the second camera) to respectively capture videos for the goods placement area, and comprehensively identify the first good and the second good based on the respectively captured current frame images, and then determine the target good that is moved based on the identification results of the goods in the two images. It can be understood that, on the one hand, using two cameras with lower cost can effectively reduce the hardware cost of the intelligent vending cabinet compared with gravity sensors and disposable RFID wireless radio frequency tags; on the other hand, since the first camera and the second camera are respectively installed at different positions outside the goods placement area, the video shooting angles of the two cameras are different, so it can be avoided as much as possible the loss of goods caused by occlusion by hands or other items during the goods tracking process (the possibility of both cameras losing simultaneously is extremely low); and by comprehensively determining the target good based on the identification results of the first good and the second good by the two cameras, it helps to improve the identification accuracy of the target good.

[0019] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions described in the embodiments of the present disclosure, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 is a flowchart of a goods identification method shown according to an embodiment of the present disclosure.

[0022] Figure 2 is a schematic diagram of an intelligent vending cabinet shown according to an embodiment of the present disclosure.

[0023] Figure 3 It is a schematic diagram of a current frame image shown according to an embodiment of the present disclosure.

[0024] Figure 4 It is a flowchart of another method for identifying goods shown according to an embodiment of the present disclosure.

[0025] Figure 5 It is a flowchart of a method for identifying and tracking goods in a current frame image shown according to an embodiment of the present disclosure.

[0026] Figure 6 It is a flowchart of a method for determining the movement type of currently unlost goods shown according to an embodiment of the present disclosure.

[0027] Figure 7 It is a flowchart of a method for identifying the goods label of currently unlost goods shown according to an embodiment of the present disclosure.

[0028] Figure 8 It is a flowchart of a method for fusing the content of two current frame lists shown according to an embodiment of the present disclosure.

[0029] Figure 9 It is a flowchart of a method for fusing two recognition results shown according to an embodiment of the present disclosure.

[0030] Figure 10 It is a flowchart of a method for updating the current counting result according to the current states of both associated parties shown according to an embodiment of the present disclosure.

[0031] Figure 11 It is a flowchart of a method for updating the current goods counting result shown according to an embodiment of the present disclosure.

[0032] Figure 12 It is a schematic block diagram of an intelligent storage cabinet shown according to an embodiment of the present disclosure. Detailed implementation manners

[0033] Next, the technical solutions in the embodiments of the present disclosure will be clearly and completely described in conjunction with the accompanying drawings of the present disclosure. Apparently, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0034] In the goods identification solutions proposed in the related art, there are generally problems of high hardware costs and low identification accuracy. To solve the above problems existing in the related technical solutions, the present disclosure proposes a goods identification solution that simultaneously tracks goods through two cameras respectively, and comprehensively determines the target goods being moved based on the independent identification results of the two cameras, so as to improve the accuracy of goods tracking and identification while reducing hardware costs. The goods identification solution of the present disclosure will be described in detail below with reference to the accompanying drawings and corresponding embodiments.

[0035] Figure 1 is a flowchart of a goods identification method shown in an exemplary embodiment of the present disclosure. As Figure 1 shown, the method is applied to an intelligent container and may include the following steps 102-106.

[0036] Step 102, obtain a first current frame image and a second current frame image respectively collected by a first camera and a second camera for a goods placement area, where the first camera and the second camera are respectively installed at different positions outside the goods placement area.

[0037] In the intelligent container of the present disclosure, there may be installed a goods placement component corresponding to the goods placement area. This component may specifically be a cross beam, a baffle, a frame, etc. Correspondingly, the area for placing goods inside or on the surface of the goods placement component is the goods placement area described in the present disclosure. In addition, a first camera and a second camera are respectively installed at different positions outside the goods placement area in the intelligent container. The first camera and the second camera may be two independent video acquisition systems controlled by the intelligent container, and any one of the cameras can independently complete corresponding functions such as video shooting and storage, so that the intelligent container (actually the controller of the intelligent container) can identify and track goods based on the above videos.

[0038] As Figure 2 shown in the schematic diagram of the intelligent container, in the intelligent container 201, a plurality of cross beams 202 are installed in the vertical direction. Any cross beam can be used to place goods 203. The areas of each cross beam 202 that can place goods together constitute the goods placement area inside the intelligent container 201. Two cameras, namely a first camera 2041 and a second camera 2042, are also respectively installed on the left and right sides above the intelligent container 201. In addition to as Figure 2In addition to installing the first camera 2041 and the second camera 2042 on the upper left and right sides above the goods placement area as shown, they can also be installed on both sides in the middle, both sides at the bottom, both the upper and lower sides on the left, both the upper and lower sides on the right, the upper left and the lower right, etc. of the goods placement area. In the implementation of the solution, the installation positions of the above two cameras can be adjusted according to the actual situation such as the internal structure of the intelligent container 201 and the law of goods placement. The embodiments of the present disclosure do not limit this. However, the above two cameras should be set at different positions and even their respective shooting angles can be adjusted (such as rotatable or movable, etc.) so that the two cameras can respectively shoot the goods placement area from different angles to avoid the loss of goods during the goods tracking process caused by the occlusion of goods in the video captured by one of the cameras.

[0039] For example Figure 2 taking the first camera 2041 shown as an example, the video frame obtained by shooting the lower goods placement area can be as Figure 3 shown: The video frame is divided by the edge of the container into the goods placement area 301 inside the intelligent container 201 and its external area 302. In order to effectively distinguish different areas, an edge marking frame can be preset for the goods placement area within the visual field corresponding to the image. Among them, the edge marking frame can be single-layer; or it can also be multi-layer, such as Figure 3 the inner marking frame 3031 and the outer marking frame 3032 shown constitute a double-layer edge marking frame. And the edge marking frame can be a Figure 3 closed marking frame as shown (that is, the inner marking frame 3031 and the outer marking frame 3032 respectively form a closed polygon), or it can also be an open marking frame, such as Figure 3 the parts of the edge marking frame connecting the image edge on the left, below and on the right shown can be empty (that is, only the edge marking frame is set between the goods placement area and the external area). The embodiments of the present disclosure do not limit this. From this perspective, if the user moves (that is, takes, including putting in or taking out) the goods, it can be reflected in the video captured by the camera: such as Figure 3 shown, the picture includes the user's limbs (hand + arm) and the goods (cola bottle) put in or taken out. Of course, the video frame image collected by the second camera 2042 is similar to Figure 3 and will not be elaborated.

[0040] In addition, the intelligent container 201 can be an open container to facilitate the user (such as the salesperson putting in the goods or the consumer taking out the goods, etc.) to put in or take out the goods; or, in order to control the hygiene and operations such as putting in and taking out of the placed goods, the intelligent container 201 can also be equipped with a blocking device, such as Figure 2The shown cargo door 205 can be used to dust-proof or control the temperature of the goods in the intelligent cargo cabinet 201 through its setting. In the solution of the present disclosure, the above-mentioned blocking device can be assembled at the edge of the goods placement area to effectively control the entry and exit of goods. For example, when the user needs to put goods into or take goods out of the goods placement area with the cargo door 205 open.

[0041] In an embodiment, the blocking device assembled at the edge of the goods placement area for controlling the entry and exit of goods can also be used to control the opening and closing of two cameras, that is, the opening and closing of the blocking device are used as the trigger conditions for the start and stop of the goods recognition solution described in this solution. For example, the intelligent cargo cabinet can send opening messages to the first camera and the second camera respectively in response to detecting the opening of the blocking device to trigger the first camera and the second camera to start recording the first video and the second video respectively. Among them, the first current frame image belongs to the first video, and the second current frame image belongs to the second video. Thereafter, the intelligent cargo cabinet can detect and identify the first current frame image and the second current frame image respectively in the manner described below to determine the target goods being moved. After the user finishes putting in or taking out the goods, the user can close the blocking device, such as closing Figure 2 the shown cargo door 205. Correspondingly, the intelligent cargo cabinet can send closing messages to the first camera and the second camera respectively in response to detecting the closing of the blocking device to trigger the first camera and the second camera to stop recording the first video and the second video respectively. It can be understood that when the blocking device is closed, the user cannot move the goods in the goods placement area. Therefore, at this time, the intelligent cargo cabinet has no need for goods recognition. Naturally, the first camera and the second camera do not need to shoot the first video and the second video respectively. Through the above method, the start and stop of video shooting of two cameras can be controlled based on the opening and closing of the blocking device, so that when the blocking device is closed, neither of the two cameras shoots invalid videos, reducing the workload of video acquisition and also helping to extend the service life of the two cameras.

[0042] Step 104: Identify the first goods in the first current frame image and the second goods in the second current frame image respectively, and determine the target goods being moved at the current moment according to the recognition results of the first goods and the second goods.

[0043] Based on the first current frame image and the second current frame image respectively captured by the first camera and the second camera, the intelligent vending cabinet can respectively identify the first item in the first current frame image and the second item in the second current frame image, and comprehensively determine the target item being moved at the current moment according to the recognition results of the two. It should be noted that the first current frame image may be the first video frame image of the first video captured by the first camera; or, the first current frame image may not be the first video frame image of the first video. In this case, on the time axis of the first video, there is at least one historical frame image before the first current frame image. The second video is similar and will not be elaborated below. In the following embodiments, any one of the first current frame image and the second current frame image is taken as an example to illustrate the item recognition process:

[0044] In one embodiment, the intelligent vending cabinet can first detect the items contained in the any current frame image and their current coordinates, and then, when the current coordinates of the item are different from its historical coordinates, determine the item as the item being moved, where the historical coordinates are the position coordinates of the item in the historical frame image, and the historical frame image is the video frame image before the any current frame image in the video to which the any current frame image belongs; further, the item type of the item being moved can be recognized. It should be noted that multiple items may be detected in the any current frame image, such as at least one item placed in the item placement area and / or at least one item moved by the user, etc. Any item is represented as an item image area in the any current frame image, and this image area can be reflected by a detection box such as a horizontal rectangular box or a rotated rectangular box. Taking the horizontal rectangular box as an example, its coordinates can be represented in various forms such as the four vertex coordinates (left_x, left_y, right_x, right_y), a certain vertex and the rectangle length and width (left_x, left_y, w, h), the rectangle center point and the length and width (center x, center y, w, h), etc. Based on this, the coordinates of a certain preset point in the detection box can be used as the current coordinates of the item corresponding to the detection box, where the preset point can be the center point or a certain vertex of the detection box, etc., and the embodiments of the present disclosure do not limit this.

[0045] Taking the first current frame image as an example, when the first current frame image is the first video frame image of the first video, the intelligent vending cabinet can directly record only the detection results of the goods and their current coordinates in the first current frame image; while when the first current frame image is a non-first video frame image of the first video, the current coordinates are compared with the historical coordinates of the goods in the historical frame image to determine whether the goods have been moved. Taking the historical frame image as the previous frame image of the first current frame image as an example, if the historical coordinates of any goods in the previous image are different from their current coordinates in the first current frame image, that is, the goods are in different positions in the two frame images, it indicates that the goods have moved during the time interval between the shooting of the two frame images. Therefore, the goods can be determined as the first goods detected from the first current frame image. It can be seen that the first goods and the second goods detected in this way are all goods that have moved, rather than the static (not moved) goods placed in the goods placement area, realizing the dynamic and static screening of goods, which helps to reduce the number of goods to be processed in the subsequent processing process, avoid ineffective processing of static goods that have not moved, and improve the goods recognition efficiency.

[0046] Among them, the intelligent vending cabinet can use a pre-trained goods detection model and a goods recognition model to detect and identify the goods in the current frame image respectively. For example, the intelligent vending cabinet can input any current frame image into the goods detection model to detect the goods and their position information contained therein by this model, and obtain the position information of the detected goods in the any current frame image output by the goods detection model. Among them, the above position information includes both the current coordinates and can also include information such as the size of the corresponding detection frame (corresponding to the size of the goods in the any current frame image), which will not be elaborated here. Similarly, the intelligent vending cabinet can input the any current frame image into the goods recognition model to identify the goods category of the goods therein by this model, and obtain the type label used to characterize the goods type of the goods output by the goods recognition model. Among them, the goods recognition model can also output the similarity between different type labels and the goods, so that the intelligent vending cabinet can determine the type label of the goods according to the similarity. The specific process can refer to the description of the following embodiments and will not be elaborated here for the time being. Of course, in order to improve the recognition accuracy of the goods recognition model, the detected goods in the any current frame image can also be cropped first to obtain a cropped image, and then the cropped image is input into the goods recognition model to obtain the type label of the goods type of the goods output by it. Among them, the type label of any goods can be used to characterize the goods type of the goods, and this label can be accurate to the SKU (Stock Keeping Unit) granularity to more accurately determine which goods have been moved.

[0047] In fact, the foregoing goods detection model and / or the goods recognition model can be deployed locally on the intelligent container, so as to accelerate the speed of goods detection and recognition by bringing the model forward. Moreover, the intelligent container can even operate offline, which helps to flexibly select the installation location of the intelligent container according to business requirements. Of course, considering that the data volume of the above models (especially the goods recognition model) and the resources required for their operation may be large, the above models can also be deployed on a remote server (such as the cloud) to reduce the local computing pressure on the intelligent container. As a result, fewer local resources such as storage and computing can be set for the intelligent container, further reducing the software and hardware costs and operating costs of the intelligent container.

[0048] In the case where a consumer takes out a good from the goods placement area, only the goods taken out of the goods placement area (i.e., outside the edge marking frame) are likely to be purchased. Therefore, identifying the type of the goods when it is determined that the goods are taken out of the goods placement area or are outside the goods placement area helps to avoid ineffective identification. For example, if a user takes a good but puts it back without taking it out of the goods placement area, there is no need to identify the type of the good at this time. To this end, an edge marking frame located at the edge of the goods placement area can be set in each video frame image included in the video to which the any current frame image belongs. The marking frame can be used as the demarcation line between the goods placement area and other areas in the any current frame image, so that it can be determined whether the goods are located in the goods placement area based on the marking frame. Based on the edge marking frame, the intelligent container can identify the type of the moved good when it is determined that the moved good is located outside the edge marking frame at the current moment; or, when it is determined that the moved good is moved from the inside of the edge marking frame to the outside, identify the type of the moved good. In this way, the intelligent container can further identify the type of the detected good only when necessary (i.e., when it is determined that the goods are moved out of the goods placement area or are outside the goods placement area), which helps to avoid ineffective identification and improve the efficiency of the goods recognition solution to a certain extent.

[0049] In addition, the process of the user moving the goods may be relatively slow, or there may be jitters during the movement, or the goods may even be taken out and put in repeatedly, resulting in the goods appearing near the edge marking frame of their own multiple times. In this scenario, various measures can be taken to avoid jumps in the judgment result of whether the goods are inside or outside the goods placement area, so as to ensure the accuracy and effectiveness of the recognition result. For example, if the moved goods detected in a continuous preset number of video frame images including any of the current frame images are on the same side of the edge marking frame, it can be determined that the goods are on this side of the edge marking frame at the current moment. The above preset number can be set according to actual situations such as the usage scenario and video bit rate. For example, it can be set to 5 frames, 3 frames, 10 frames, etc. The embodiments of the present disclosure do not limit this. Taking the first video as an example, if a certain good is inside the edge marking frame in 5 consecutive video frame images including the current frame image (i.e., the first current frame image and the 4 consecutive historical frame images before it), it can be determined that the good is inside the goods placement area at the current moment; or, if a certain good is outside the edge marking frame in 10 consecutive video frame images including the current frame image (i.e., the first current frame image and the 9 consecutive historical frame images before it), it can be determined that the good is outside the goods placement area at the current moment.

[0050] For another example, in the case where the edge marking frame includes an inner marking frame and an outer marking frame with a preset interval, the intelligent storage cabinet can determine the position of the moved goods relative to the edge marking frame at any moment in the following way: if the moved goods are within the inner marking frame at any moment, it can be determined that the moved goods are inside the edge marking frame at the any moment; and, if the moved goods are outside the outer marking frame at any moment, it can be determined that the moved goods are outside the edge marking frame at the any moment. Taking the first video in the scenario shown as an example, if a certain good is within the inner marking frame 3031 at any moment, it can be determined that the good is inside the edge marking frame at this time; or, if a certain good is outside the outer marking frame 3032 at any moment, it can be determined that the good is outside the edge marking frame at this time. Among them, the preset interval between the above inner marking frame and outer marking frame can be set according to the actual situation. For example, it can be set to an interval of 10 pixel points, or an interval of 2 mm, etc. The embodiments of the present disclosure do not limit this. If a certain good is outside the inner marking frame and inside the outer marking frame (such as in the area between the inner marking frame 3031 and the outer marking frame 3032) at any moment, it can be determined that the position of the good at this time is still the position at the previous moment, that is, the relative position of the good to the edge marking frame remains unchanged until the good crosses the inner marking frame or the outer marking frame again. Figure 3 Taking the first video in the scenario shown as an example, if a certain good is within the inner marking frame 3031 at any moment, it can be determined that the good is inside the edge marking frame at this time; or, if a certain good is outside the outer marking frame 3032 at any moment, it can be determined that the good is outside the edge marking frame at this time. Among them, the preset interval between the above inner marking frame and outer marking frame can be set according to the actual situation. For example, it can be set to an interval of 10 pixel points, or an interval of 2 mm, etc. The embodiments of the present disclosure do not limit this. If a certain good is outside the inner marking frame and inside the outer marking frame (such as in the area between the inner marking frame 3031 and the outer marking frame 3032) at any moment, it can be determined that the position of the good at this time is still the position at the previous moment, that is, the relative position of the good to the edge marking frame remains unchanged until the good crosses the inner marking frame or the outer marking frame again.

[0051] Through the foregoing method, it is possible to determine whether the moved goods have crossed the goods placement area, such as whether the goods have moved from the inside of the edge marking frame to the outside or from the outside to the inside. Based on this, for any of the first current frame image and the second current frame image, when the intelligent cabinet indicates that the moved goods have crossed the edge of the goods placement area in the any current frame image, the intelligent cabinet can also identify the moving manner of the moved goods. The moving manner may be taking out from the goods placement area (i.e., moving from the inside of the goods placement area to the outside) or putting into the goods placement area (i.e., moving from the outside of the goods placement area to the inside). By this means, the specific moving manner of the moved goods can be determined, which is convenient for accurately counting the goods corresponding to different moving manners subsequently.

[0052] In an embodiment, when the first goods is detected in the first current frame image, the intelligent cabinet can also record the goods information of the first goods in the first current frame list; and when the second goods is detected in the second current frame image, record the goods information of the second goods in the second current frame list. Further, the intelligent cabinet can determine the target goods moved at the current moment according to the goods information respectively recorded in the first current frame list and the second current frame list. The goods information of any goods recorded in any of the foregoing current frame lists may include any one of the following: the goods identifier (i.e., goods id) assigned to the goods, the current coordinates of the goods, the moment corresponding to the goods (i.e., the current moment), and the goods label of the goods (if the goods label is recognized, record the recognized goods label; if the goods label is not recognized, a default label such as "foreign object" or "unknow" etc. may be recorded).

[0053] It can be understood that the process of determining the position coordinates and the moving process of the same goods in different frame images in the foregoing embodiment is the process of tracking the goods. In the implementation of the solution, the above tracking process can use the Kalman filter algorithm to predict the target position of the goods: use the previous state value and the current state measurement value (the detection box BBox of the detected goods) to predict the estimated value of the next state, so as to realize the prediction of the target position; and match the prediction result with the target detection result at the next moment by using the Hungarian algorithm, calculate the distance between the two BBoxes with IOU (Intersection over Union), and use the Hungarian algorithm to select the optimal association result to realize the association between the predicted BBox and the detected BBox. Then use the corresponding detected BBox to represent the successfully tracked BBox result to realize the tracking of the goods. The specific implementation of the above process can refer to the records in the related technology and will not be elaborated here.

[0054] Based on the above tracking process, for any one of the first video to which the first current frame image belongs and the second video to which the second current frame image belongs, there may also be corresponding moving goods lists respectively (i.e., the first video has a first moving goods list, and the second video has a second moving goods list). The moving goods list is used to record the tracking information of the goods identified from any one of the videos (i.e., the tracked goods) with the goods identifier as the index. Among them, the tracking information of any one goods may include one of the following:

[0055] The goods identifier assigned to this goods (i.e., goods id);

[0056] The first appearance coordinates, that is, the position coordinates when first appearing in the video;

[0057] The first appearance time, that is, the time when first appearing in the video, which can be the time corresponding to the video frame image of the first appearance coordinates on the video time axis;

[0058] The latest coordinates, that is, the position coordinates corresponding to when this goods is detected most recently;

[0059] The position identifier, which is used to represent whether it is inside or outside the goods placement area at the current moment, that is, inside or outside the edge marking frame. Among them, when current_status takes the value of 1, it means that it is currently inside the goods placement area; when it takes the value of -1, it means that it is currently outside the goods placement area;

[0060] The goods label, which is used to represent the specific type of the goods. If the goods label is recognized, the recognized goods label is recorded, such as "Coke" or "Milk", etc.; if the goods label is not recognized, the default label can be recorded, such as "foreign object" or "unknow", etc.;

[0061] The tracking status; including the tracking continuous status and the tracking lost status, which can be reflected by the values of the temporary deletion flag bDelete and the real deletion flag bRealDel: when bDelete takes the value of true, it indicates that this goods is in the tracking lost status; when bDelete takes the value of false, it indicates that this goods is in the tracking continuous status; when bRealDel takes the value of true, it indicates that the tracking information of this goods should be deleted from the moving goods list; when bRealDel takes the value of false, it indicates that the tracking information of this goods does not need to be deleted from the moving goods list;

[0062] The movement type identifier, which is used to represent whether the goods is put into the goods placement area or taken out of the goods placement area during the movement process. When the value of behavior_status is 1, it means that this goods is put in; when the value is -1, it means that this goods is taken out; when the value is 0, it means that the movement type has not been determined yet;

[0063] Counting flag, used to characterize whether the goods corresponding to the taking-out or putting-in behavior have been counted in the goods counting list: when the value of bCount is true, it means that they have been counted; when the value of bCount is false, it means that they have not been counted.

[0064] Based on this, for any goods detected from the current frame image of any of the videos, the intelligent vending cabinet can determine whether the goods are newly added goods, that is, determine whether the tracking information of the goods has been recorded in the corresponding moving goods list, that is, determine whether the goods have been tracked. For example, when it is determined that any of the goods is the same as the goods represented by any of the goods identifiers in the moving goods list, it indicates that the tracking information of any of the goods has been recorded in any of the moving goods lists. At this time, the intelligent vending cabinet can update the tracking information recorded with any of the goods identifiers as the index in the moving goods list according to any of the goods. For example, when the same goods are recognized in any of the current frame images and its previous video frame image, the tracking information of the goods updated according to the previous video frame image is recorded in the moving goods list. Then, at the current moment, the tracking information can be updated according to the goods recognized in any of the current frame images to ensure that the tracking information is updated in real time along with the tracking process of the goods. For another example, when it is determined that any of the goods is different from the goods represented by each of the goods identifiers in the moving goods list, it indicates that any of the goods is a newly emerged (just started to be moved) goods that has not been tracked before. At this time, the intelligent vending cabinet can assign a new goods identifier to any of the goods and record the tracking information of any of the goods with the new goods identifier as the index in the moving goods list to add the tracking information of the newly emerged goods in the moving goods list.

[0065] The tracking information recorded in the moving goods list may include a tracking status, which may be a tracking continuous status and a tracking lost status. The tracking continuous status can be used to characterize that any of the goods is detected from the previous video frame image of the current frame image (that is, the goods are continuously tracked), and the tracking lost status can be used to characterize that any of the goods is not detected from the previous video frame image of the current frame image and any of the goods is detected from other historical video frame images before the previous video frame image (that is, the goods have been tracked but lost in the previous video frame image before the current moment). Among them, the tracking status can be identified through the aforementioned tracking algorithm and distinguished by the value of the temporary deletion flag bDelete. For example, when the value of bDelete is "true", it indicates that the tracking status is the tracking lost status and the corresponding goods are being tracked; when the value of bDelete is "false", it indicates that the tracking status is the tracking continuous status and the corresponding goods are lost during the tracking process.

[0066] Based on this, when updating the tracking information recorded with any of the item identifiers in the moving item list according to any of the items, if the tracking status corresponding to any of the item identifiers is the tracking continuous status, other tracking information except the first value recorded with any of the item identifiers can be updated - because the item is recognized in any of the current frame images and its previous video frame images, the tracking of the item is actually achieved, and at this time, it is only necessary to continue to maintain the item in the tracking continuous status. If the tracking status corresponding to any of the item identifiers is the tracking lost status, all the tracking information recorded with any of the item identifiers can be updated, and the tracking lost status is updated to the tracking continuous status - because the item is recognized in any of the current frame images but not in its previous frame image, the item that was actually lost is re-tracked, so the item can be updated from the tracking lost status to the tracking continuous status. In this way, the intelligent storage cabinet can re-match and re-track the items that are lost during the item tracking process, avoiding the counting deviation that may be caused by the loss of items during the tracking process due to abnormal reasons such as processing speed, shooting quality, or short-term occlusion, thus helping to improve the final recognition and counting accuracy.

[0067] Among them, for the intelligent storage cabinet to determine that any of the items is the same item as the item with the tracking lost status corresponding to any of the item identifiers in the moving item list, it may include at least one of the following: the appearance time of any of the items is later than the appearance time of any of the item identifiers - the loss of an item must occur before its reappearance; the item type of any of the items is the same as the item type of the item represented by any of the item identifiers - if a newly appeared item is a lost item, the item types of the two items should be the same; the distance between any of the items and the item represented by any of the item identifiers is not greater than the distance between any of the items and other items with the tracking lost status - the loss time is usually short, so the lost item closest to the newly appeared item is usually the newly appeared item.

[0068] In fact, the first item may not be recognized in the above-mentioned first current frame image, and the second item may not be recognized in the second current frame image either. In this regard, the intelligent storage cabinet can adopt different methods to determine the target item being moved at the current moment according to the recognized quantities of the first item and the second item. For example, when the quantities of both the first item and the second item are greater than zero, the intelligent storage cabinet can perform a fusion process on the recognition results of the first item and the second item to determine the target item being moved at the current moment; while when the quantity of any one of the first item and the second item is greater than zero and the quantity of the other item is zero, then the said any one item can be determined as the target item being moved at the current moment.

[0069] As mentioned above, the first video to which the first current frame image belongs may have a first moving item list, and the second video to which the second current frame image belongs may have a second moving item list. The moving item list of any video can be used to record the tracking information of the items recognized from the said any video with the item identifier as the index. In this scenario, let's assume that the quantity of the first item is greater than the quantity of the second item. At this time, the intelligent storage cabinet can determine the association relationship between each first item and each second item according to the item information, and divide the first items into first associated items and / or first non-associated items, and divide the second items into second associated items and / or second non-associated items; then traverse each first item. Among them, if any first associated item has a second associated item associated with it, then the said any first associated item can be determined as the target item being moved at the current moment, and the corresponding tracking information recorded in the first moving item list can be updated according to the said any first associated item.

[0070] Among them, for any first item and any second item, it can be determined whether there is an association between the two according to their respective tracking information. For example, if the first appearance coordinates, the first appearance time, and the item label of any first item and any second item are respectively the same, it can be determined that the two items are actually the same item being moved photographed by the first camera and the second camera respectively, so it can be determined that there is an association between the two. At this time, the said any first item can be determined as the first associated item, and the second item can be determined as the second associated item. Correspondingly, the first item that has no association with any second item is the first non-associated item, and the second item that has no association with any first item is the second non-associated item. By this method, the first item and the second item can be respectively divided into corresponding associated items and non-associated items, and updating the tracking information of the items based on the above classification can improve the judgment efficiency of whether there is an association.

[0071] Step 106, when the target goods type and the target moving mode of the target goods are determined, update the quantity of the goods corresponding to the target goods type and the target moving mode in the goods count list according to the quantity of the target goods, where the target moving mode includes taking out from or putting into the goods placement area.

[0072] After the target goods to be moved are determined in the foregoing manner, the intelligent cabinet can further determine the target goods type of the target goods and its target moving mode, and then update the quantity of the goods corresponding to the target goods type and the target moving mode according to the quantity of the target goods.

[0073] As described above, the tracking information recorded in the moving goods list corresponding to any video may include a tracking status, where the tracking status includes a tracking continuous status and a tracking lost status. The tracking continuous status is used to represent that any goods are detected in the previous video frame image of the current frame image, and the tracking lost status is used to represent that any goods are not detected in the previous video frame image of the current frame image and any goods are detected in other historical video frame images before the previous video frame image. Based on this, if any first associated goods are in the tracking lost status and there are no second associated goods associated with them, it indicates that the first associated goods are only detected in the first video but lost in the latest video frame, and the second video has never detected the goods. In this regard, the goods information of any first associated goods can be deleted from the first moving goods list to reduce the calculation amount in the subsequent fusion process and save the local storage space of the intelligent cabinet.

[0074] In addition, the intelligent cabinet can update the quantity of the goods corresponding to the target goods type and the target moving mode in the goods count list in the following manner: first, when there are second associated goods associated with any first associated goods, determine the quantity, target goods type, and target moving mode of the any first associated goods; then update the quantity of the goods corresponding to the target goods type and the target moving mode in the goods count list according to the quantity. The goods in the goods count list are counted according to two dimensions of goods type and moving mode. Therefore, when updating the quantity of any goods, the corresponding goods quantity can be found according to the target goods type and the target moving mode and then updated to ensure the comprehensiveness and accuracy of the quantity update.

[0075] Among them, the intelligent storage cabinet can first query the current quantity of goods corresponding to the target goods type and the target moving method in the goods counting list; then use the sum of the quantity of the target goods and the current quantity of goods as the updated quantity of goods corresponding to the target goods type and the target moving method, that is, add the quantity of the identified target goods to the current quantity of goods corresponding to the target goods type and the target moving method.

[0076] In an embodiment, when a blocking device for controlling the entry and exit of goods is assembled at the edge of the goods placement area, the intelligent storage cabinet can also respond to detecting that the blocking device is closed, count the quantity of goods corresponding to different moving methods and different goods types according to the goods counting list, and then send the quantity of goods to a preset statistics party. Among them, the above-mentioned counted quantity of goods can be used to represent that "s1 pieces of goods of type 1 are put in and s2 pieces are taken out", "s3 pieces of goods of type 2 are put in and s4 pieces are taken out", etc., which will not be elaborated here. It can be understood that the above-mentioned statistics party can be the server connected to the intelligent storage cabinet or other statistics parties. This statistics party can receive the statistical results uploaded by multiple different intelligent storage cabinets respectively, and then perform statistics or intelligent analysis on the above-mentioned statistical results according to at least one statistical dimension. The analysis results can be further used to guide the goods placement strategy to promote the development of business such as goods sales.

[0077] According to an embodiment of the present disclosure, the intelligent storage cabinet acquires the first current frame image and the second current frame image respectively collected by the first camera and the second camera for the goods placement area. The first camera and the second camera are respectively assembled at different positions outside the goods placement area; then respectively identify the first goods in the first current frame image and the second goods in the second current frame image, and determine the target goods moved at the current moment according to the identification results of the first goods and the second goods; when the target goods type and the target moving method of the target goods are determined, update the quantity of goods corresponding to the target goods type and the target moving method in the goods counting list according to the quantity of the target goods. The target moving method includes taking out from or putting into the goods placement area.

[0078] It can be seen that this solution uses two cameras (i.e., the first camera and the second camera) to collect videos for the goods placement area respectively, comprehensively identify the first goods and the second goods based on the currently captured images respectively, and then determine the target goods that have been moved based on the recognition results of the goods in the two images. It can be understood that, on the one hand, using two cameras with lower costs can effectively reduce the hardware cost of the intelligent cabinet compared with gravity sensors and disposable RFID wireless radio frequency tags; on the other hand, since the first camera and the second camera are respectively assembled at different positions outside the goods placement area, the video shooting angles of the two cameras are different, so the loss of goods caused by occlusion by hands or other items can be avoided as much as possible during the goods tracking process (the possibility of simultaneous loss of the two cameras is extremely low); moreover, the target goods that have been moved are comprehensively determined based on the recognition results of the first goods and the second goods by the two cameras, which helps to improve the recognition accuracy of the target goods.

[0079] Figure 4 It is a flowchart of another goods recognition method shown in an exemplary embodiment of the present disclosure. The following combines Figure 4-11 In the embodiments of the present disclosure, taking the intelligent cabinet equipped with a cabinet door as an example, the goods recognition, tracking and counting processes of the method of the present disclosure will be described in detail. As Figure 4 shown, this method is applied to an intelligent cabinet and includes steps 401-409.

[0080] Step 401, obtain the current state of the cabinet door.

[0081] Step 402, determine whether the cabinet door is open.

[0082] First, the intelligent cabinet first senses the current state of the cabinet door through a pre-installed sensor, and then determines whether the cabinet door is opened at the current moment. If it is opened, it proceeds to step 405; otherwise, it proceeds to step 403.

[0083] Step 403, determine whether a counting message has been sent after this door closing.

[0084] Step 404, send a counting message to the statistics party and reset the relevant flag bits.

[0085] If the cabinet door is closed, it indicates that the user has completed the process of moving the goods at the current moment. At this time, it can be determined whether a corresponding counting message has been sent to the statistics party for this closing behavior of the cabinet door. It can be understood that since the door must be opened before closing, the counting message sent in the above process can be used to represent "the number of goods taken out or put into the intelligent cabinet during the period from the most recent opening to the most recent closing".

[0086] If the counting message has been sent, the current processing procedure can be directly ended. Otherwise, a counting message for the current opening behavior of the cargo door can be sent to a preset statistics party, and the counting message includes the quantities of various types of goods taken out or put in as recorded in the goods counting list. Additionally, the determination of whether the corresponding counting message has been sent to the statistics party can be made based on the value of a preset flag bit. Therefore, the flag bit should be reset after sending the counting message, which will not be elaborated here. Of course, the current processing procedure can also be ended after sending the counting message.

[0087] Step 405: Obtain the current frame images respectively captured by two cameras.

[0088] At this time, the cargo door is opened, so the first camera and the second camera can respectively capture the first video and the second video for the goods placement area, and thus the intelligent cargo cabinet can obtain the first current frame image in the first video and the second current frame image in the second video.

[0089] Step 406: Process the two current frame images respectively to obtain the moving goods lists corresponding to the two videos.

[0090] Based on the first current frame image and the second current frame image, the intelligent cargo cabinet can respectively process them to obtain the corresponding first moving goods list and second moving goods list. The specific processing procedure can be referred to in the following description for Figure 5 detailed description and will not be elaborated here for the time being.

[0091] Step 407: Obtain the current frame lists corresponding to the two current frame images respectively.

[0092] Step 408: Integrate the contents of the two current frame lists.

[0093] Step 409: Update the moving goods list and the goods counting list according to the integration result.

[0094] After obtaining the first moving goods list and the second moving goods list, the intelligent cargo cabinet can also process the first current frame image and the second current frame image respectively to obtain the first current frame list corresponding to the first video and the second current frame list corresponding to the second video. Then, integrate the goods information recorded in the first current frame list and the second current frame list, and update the first moving goods list, the second moving goods list, and the goods counting list for counting according to the integration result. The process can be referred to in the following description for the appendix Figure 7 for detailed description and will not be elaborated here for the time being.

[0095] Figure 5 is a flowchart showing the identification and tracking of goods for any current frame image in an exemplary embodiment of the present disclosure. As Figure 5 shown, this process includes steps 501 - 509.

[0096] Step 501: Detect the goods that have been moved in the current frame image.

[0097] The goods in any current frame image can be detected by a pre-trained goods detection model, as well as the current coordinates of each good in the current frame image. And a corresponding current frame list can be generated according to the relevant information of the detected goods. The goods information of any good recorded in this list can include: the goods identifier (i.e., goods ID) assigned to the good, the current coordinates of the good, the corresponding moment of the good (i.e., the current moment), the goods label of the good (if the goods label is recognized, the recognized goods label is recorded; if the goods label is not recognized, a default label such as "foreign object" or "unknow" can be recorded), etc., which will not be elaborated here.

[0098] The goods corresponding to the goods information recorded in the moving goods list of the video to which any current frame image belongs can generally be divided into two categories. One category is the (temporarily) lost goods that were not detected in the previous (even consecutive multiple) video frame images of the any current frame image, and the other category is the non-lost goods that were detected in the previous video frame image of the any current frame image. After detecting the goods in any current frame image, the tracking information of each good recorded in the moving goods list can be updated according to its goods information. Among them, steps 502 - 504 are the processes of updating the goods of the non-lost goods, and steps 505 - 509 are the processes of updating the goods of the lost goods. The intelligent cargo cabinet can start traversing each lost good in step 505 after traversing all the non-lost goods. Of course, in the case where there are no lost goods in the moving goods list, only steps 502 - 504 can be executed; and in the case where there are no non-lost goods in the moving goods list, only steps 505 - 509 can be executed, which will not be elaborated here.

[0099] Step 502: Traverse each non-lost good in the moving goods list that has not been lost.

[0100] Step 503: Determine the movement type of the current non-lost good.

[0101] Step 504: Identify the goods label of the current non-lost good.

[0102] Overall, first traverse each non-lost good corresponding to the moving goods list. The traversal process of any non-lost good (hereinafter referred to as the current good) includes step 503 and step 504. Among them, step 503 is used to determine the movement type of the current good, and the specific determination process can be referred to the description below for Figure 6 ; step 504 is used to identify the goods label of the current good, and the specific identification process can be referred to the description below for Figure 7Details are not described here for the moment as per the record.

[0103] Step 505: Traverse each missing item temporarily missing in the current frame list.

[0104] Overall, first traverse each missing item corresponding to the moving item list. The traversal process of any one missing item includes steps 506 to 509.

[0105] Step 506: Determine whether to delete the item information of the current missing item.

[0106] First, the time interval between the current moment and the last appearance moment corresponding to the latest coordinates of the current missing item recorded in the moving item list can be determined. If this time interval exceeds a preset time threshold, it indicates that the item has been missing for too long (it may have been put back into the item placement area or moved out of the video frame range). At this time, the tracking information of this item recorded in the moving item list can be deleted, and step 505 can be entered to start traversing the next missing item. It can be understood that after the tracking information of the current missing item is deleted, the tracking process for this item ends. On the contrary, if the time interval does not exceed the time threshold, it indicates that the current missing item is only temporarily missing and may reappear in the video frame image after the current moment. At this time, step 507 is entered for further judgment.

[0107] Step 507: Determine whether the current missing item has been taken out.

[0108] Read the current value of the movement type identifier behavior_status of the current missing item in the moving item list. If the current value is -1, it indicates that the current missing item has been taken out of the item placement area, and step 508 can be entered at this time; on the contrary, if the current value is 1, it indicates that the current missing item has been placed in the item placement area, and step 509 can be entered at this time.

[0109] Step 508: If the temporary deletion flag bDelete of the current missing item is true, delete the tracking information of the current missing item.

[0110] At this time, the current value of the temporary deletion flag bDelete of the current missing item can be checked. If the current value is true, the tracking information of the current missing item recorded in the moving item list can be directly deleted at this time to end the tracking process for this item. Among them, since the tracking information of the current missing item is recorded in the moving item list with the item identifier of the current missing item as the index, when deleting the tracking information of the current missing item, its item identifier should also be deleted - that is, all data of the current missing item are deleted from the moving item list. The deletion of the tracking information of other items from the moving item list in other embodiments of the present disclosure is similar and will not be elaborated.

[0111] Conversely, if the current value is false, the tracking information of the current missing item recorded in the moving item list may not be deleted, and the process may directly proceed to step 505 to traverse the next missing item or to step 509.

[0112] Step 509, perform missing re-matching on the current missing item.

[0113] The current missing item may be regarded as an old item, and the item detected in any current frame image that is not recorded in the moving item list may be regarded as a new item. For any new item, the intelligent vending cabinet can determine the appearance time of the item (i.e., the time corresponding to any current frame image) and the disappearance time of each old item. If the any new item and an old item are actually the same item (i.e., they match), the appearance time of the any new item must be later than the disappearance time of the old item. Based on this, further judgment can be made among the old items whose disappearance time is earlier than the appearance time of the any new item. At this time, the item labels of each old item can be checked in sequence. If the any new item and an old item match, their item labels must be the same. Based on this, further judgment can be made among the old items whose item labels are the same as the item label of the any new item. At this time, the old items that are not at the edge of the screen can also be screened out from each old item. At this time, if there are multiple screened old items, the old item whose disappearance position is closest to the appearance position of the any new item can be determined as the old item that matches the any new item. At this time, the tracking information of the old item recorded in the moving item list can be updated according to the item information of the any new item recorded in the current frame list, so as to achieve re-matching of the item.

[0114] Figure 6 It is a flowchart showing a process for determining the movement type of a currently non-missing item (hereinafter referred to as the current item) according to an exemplary embodiment of the present disclosure. As Figure 6 shown, the process includes steps 601-617.

[0115] Step 601, determine whether the current item identifier of the current item is greater than the maximum item identifier of the current frame list.

[0116] The intelligent vending cabinet can sequentially increment the item identifiers assigned to each detected item. At this time, the maximum item identifier (i.e., max_id) of the moving item list is the maximum item identifier at the current moment.

[0117] If the current item identifier assigned to the newly detected current item is greater than the maximum item identifier, it indicates that the current item is an item newly added in the current frame image relative to the previous video frame image. At this time, step 602 can be entered; otherwise, if the current item identifier is not greater than (less than or equal to) the maximum item identifier, it indicates that the current item is not a newly added item, and step 603 can be entered at this time.

[0118] Step 602: Update the maximum item identifier in the moving item list to the current item identifier, and initialize both the inner consecutive frame number and the outer consecutive frame number corresponding to the current item identifier to 0.

[0119] Step 603: Determine whether the current item is within the inner marking frame.

[0120] Specifically, it can be determined according to the current coordinates of the current item recorded in the current frame list, that is, by the relative position of the current coordinates and the edge marking frame. If the current item is within the inner marking frame, step 604 is entered; otherwise, if the current item is outside the inner marking frame, step 610 is entered.

[0121] Step 604: Increment the inner consecutive frame number corresponding to the current item identifier by 1 and reset the outer consecutive frame number to 0.

[0122] The inner consecutive frame number of any item in the moving item list is used to represent the number of times the item continuously appears within the inner marking frame, and the outer consecutive frame number is used to represent the number of times the item continuously appears outside the outer marking frame. Since it has been determined that the current item is within the inner marking frame at this time, the inner consecutive frame number can be incremented by 1, and the outer consecutive frame number is set to zero (of course, if it is already zero, there is no need to reset).

[0123] Step 605: Determine whether the current item identifier is a new identifier.

[0124] Determining whether the current item identifier is a new identifier is to determine whether the current item is a new item. This step can be determined according to the judgment result of step 601: If the current item identifier is greater than the maximum item identifier, it can be determined that the current item identifier is a new identifier, and step 617 can be entered at this time; otherwise, if the current item identifier is not greater than the maximum item identifier, it can be determined that the current item identifier is not a new identifier - the item identifier corresponding to the current item must already exist in the moving item list, and step 606 can be entered at this time.

[0125] Step 606: Determine whether the inner consecutive frame number is greater than the first threshold N1.

[0126] If the number of consecutive inner frames is greater than the first threshold N1, it can be determined that the current item has appeared in the item placement area for multiple consecutive frames. Therefore, step 607 can be entered at this time; otherwise, the current judgment process can be directly ended, and the next item detected in the current frame image can be traversed.

[0127] Step 607: Update the current position identifier of the current item recorded in the current frame list to 1, and update the previous position identifier to the current position identifier.

[0128] This step essentially updates the previous position identifier last_status of the current item recorded in the moving item list to 1 to indicate that the current item is inside the item placement area at the current moment.

[0129] Step 608: Determine whether the current item is out of bounds.

[0130] Specifically, it can be judged by the position of the current item relative to the edge marking frame and the position identifier of the item identifier corresponding to it in the moving item list: if the current item is on the first side of the edge marking frame, and the position identifier of the item identifier corresponding to it in the moving item list indicates that the corresponding item is on the second side, it can be determined that the current item has moved from the second side to the first side, that is, it has crossed the boundary (crossed the edge marking frame of the item placement area during the movement). At this time, step 609 can be entered; conversely, if the two are on the same side, it can be determined that the current item has not crossed the boundary. At this time, the current judgment process can be directly ended, and the next item detected in the current frame image can be traversed.

[0131] Step 609: Update the current movement type identifier to 1, update the previous movement type identifier to the current movement type identifier, and update the value of the current count flag to false.

[0132] This step essentially updates the previous movement type identifier last_behavior_status of the current item recorded in the moving item list to false to indicate that the current item has been placed inside the item placement area. And updating the value of the current count flag to false is used to represent that the above-mentioned out-of-bounds process has not been counted.

[0133] Step 610: Determine whether the current item is outside the outer marking frame.

[0134] As can be understood in combination with step 603, if the current item is within the outer marking frame, it can be determined that the current item is currently in the buffer area between the outer marking frame and the inner marking frame. At this time, to avoid jumps in the judgment result and the counting result, the current judgment process can be directly ended, and the traversal of the next item detected in the current frame image (not shown in the figure) can be started. Conversely, if the current item is outside the outer marking frame, it can be determined that the current item is currently outside the item placement area, and at this time, step 611 can be entered.

[0135] The specific processes of the following steps 611-616 are similar to those of the foregoing steps 604-609, and the specific processes will not be elaborated herein.

[0136] Step 611, increment the outer consecutive frame count corresponding to the current item identifier by 1 and reset the inner consecutive frame count to zero.

[0137] Step 612, determine whether the current item identifier is a newly added identifier.

[0138] Step 613, determine whether the inner consecutive frame count is greater than the second threshold N2.

[0139] Among them, the above first threshold N1 and the second threshold N2 can be the same or different, and can be preset according to the specific usage scenario.

[0140] Step 614, update the current position identifier of the current item recorded in the current frame list to -1, and update the previous position identifier to the current position identifier.

[0141] Step 615, determine whether the current item goes out of bounds.

[0142] Step 616, update the current movement type identifier to -1, update the previous movement type identifier to the current movement type identifier, and update the value of the current counting flag to false.

[0143] Step 617, add tracking information of the current item to the moving item list.

[0144] In the case where it is determined that the current item identifier is a newly added identifier (i.e., the current item is a newly added item), the intelligent storage cabinet can add a piece of tracking information corresponding to the current item to the moving item list. That is, the indexing method and the information content can be similar to other tracking information and will not be elaborated herein.

[0145] Figure 7 It is a flowchart for identifying the item label of the current non-lost item (hereinafter referred to as the current item) shown in an exemplary embodiment of the present disclosure. As Figure 7 shown, the process includes steps 701-708.

[0146] Step 701: Determine whether the current item is outside the container (i.e., the item placement area).

[0147] The determination method can refer to the description in the foregoing embodiments and will not be elaborated here. If it is determined that the current item is outside the container, proceed to step 702; otherwise, start identifying the item identifier of the next non-lost item.

[0148] Step 702: Determine whether the current item already has an item label and whether its item label is "foreign object".

[0149] For items whose item types have not been identified yet, their item labels can be uniformly set to default "foreign object" or "unknow", etc. Here, "foreign object" is taken as an example.

[0150] The item label of the current item may already be recorded in the moving item list. If the label is not "foreign object", it indicates that the item identifier of the current item has been identified before the current moment, such as it can be "x cola", "y sprite", "p ballpoint pen", "q biscuit", etc. So at this time, there is no need to identify again, and directly end the current identification process and start identifying the item identifier of the next non-lost item. On the contrary, if the item label of the current item is "foreign object", then the true category of this item needs to be identified at this time, and at this time, it can proceed to step 703.

[0151] Step 703: Identify the item label of the current item through the item identification model.

[0152] Any of the current frame images can be input into the item identification model to identify the item identifier of the current item by the model, and obtain at least one type label representing the item type of the item and the similarity of each type label output by the item identification model. Or, in order to improve the identification accuracy of the item identification model, a cropped image of a preset size containing the current item can also be cropped from any of the current frame images, and then the cropped image is input into the item identification model to obtain at least one type label representing the item type of the item and the similarity of each type label output by the model.

[0153] Step 704: Determine whether the similarity between the current item output by the item identification model and each label meets the similarity threshold corresponding to the label.

[0154] Step 705: Use the item label that meets the similarity threshold as the alternative label of the current item.

[0155] According to the training situation of the item identification model, the same or different similarity thresholds can be preset for different item types. Based on this, the item label that meets the similarity threshold can be used as the alternative label of the current item.

[0156] Step 706, further determine that the item label of the current item is "foreign object".

[0157] Of course, if the similarity of each candidate label output by the item recognition model is less than the corresponding similarity threshold, it can be determined that the recognition fails, that is, the accurate category of the current item is not recognized. At this time, it can be determined that the item label of the current item is still "foreign object".

[0158] Step 707, determine whether the number of alternative labels meets the threshold.

[0159] Step 708, use the alternative label with the most occurrences as the item label of the current item.

[0160] In the case where the number of alternative labels meets the quantity threshold thresh, the alternative label with the most occurrences among them can be used as the item label of the current item. For example, in the case of determining two types of alternative labels, "Coke" and "Milk", from the output result of the item recognition model, if the number of "Coke" labels is 2 and the number of "Milk" labels is 5, then "Milk" can be determined as the label of the current item, that is, it is determined that the current item is milk.

[0161] Figure 8 It is a flowchart showing a process of fusing the content of two current frame lists shown in an exemplary embodiment of the present disclosure. As Figure 8 shown, this process includes steps 801 - 814.

[0162] Step 801, respectively obtain the item information recorded in the two current frame lists.

[0163] The intelligent vending cabinet can first determine the first current frame list and the second current frame list generated and updated by the foregoing method to obtain the item information respectively recorded in the two lists.

[0164] Step 802, count the number of items num1 and num2 detected in the two current frame images.

[0165] Step 803, determine whether both num1 and num2 are greater than 0.

[0166] Count the number of the first items num1 recorded in the first current frame list and the number of the second items num2 recorded in the second current frame list, and determine whether both of them are greater than 0. If both num1 and num2 are greater than 0, it indicates that at least one item is detected in the first current frame image and the second current frame image respectively. At this time, step 805 can be entered to determine the size relationship between num1 and num2.

[0167] Step 804, update the two moving item lists according to the single - path recognition result greater than 0.

[0168] If one of num1 and num2 is 0 and the other is greater than 0, the two-way moving goods list can be updated according to the recognition result of the single path greater than 0, that is, the first moving goods list and the second moving goods list are updated only according to the recognition result of the goods on this recognized path.

[0169] Step 805, determine whether num1 is greater than num2.

[0170] Step 806, traverse mainly based on num1.

[0171] Step 807, traverse mainly based on num2.

[0172] When both num1 and num2 are greater than 0, the intelligent storage cabinet can traverse mainly based on the path with more goods. That is, when num1 > num2, traverse mainly based on the recognition result of the first goods; when num1 < num2, traverse mainly based on the recognition result of the second goods.

[0173] Step 808, fuse the recognition results of the two paths of num1 and num2.

[0174] For the specific fusion process, reference can be made to the detailed description below for Figure 9 and will not be elaborated here for the time being.

[0175] Step 809, update the two-way goods list respectively according to the fusion result.

[0176] Step 810, traverse each good in the current frame list.

[0177] Taking the update of any one of the moving goods lists as an example, the specific update process can be referred to the records in steps 810 - 814 below. During the traversal process, the traversal steps for any good (hereinafter referred to as the current good) are as follows:

[0178] Step 811, determine whether the current good identifier exists in the moving goods list.

[0179] First, determine whether the current good identifier of the current good has been recorded in the said any one of the moving goods lists, that is, query whether the current good identifier exists in the moving goods list. If it exists, go to step 812; otherwise, end the traversal of the current good and directly start the traversal of the next good.

[0180] Step 812, determine whether the current value of the temporary deletion flag bDelete of the current good is true.

[0181] When bDelete of the current good = true, go to step 813; otherwise, go to step 814.

[0182] Step 813: Update the tracking information recorded in the moving goods list according to the current goods.

[0183] At this time, the current goods have not been temporarily deleted. Therefore, the tracking information recorded in the moving goods list can be updated according to the current goods to achieve the tracking effect of the current goods.

[0184] Step 814: Delete the current ID and its tracking information from the moving goods list.

[0185] At this time, the current goods have not been temporarily deleted. At this time, the current goods identifier and its corresponding tracking information can be directly deleted from the moving goods list, and the current goods will no longer be tracked hereafter.

[0186] Figure 9 It is a flowchart showing the fusion of two recognition results shown in an exemplary embodiment of the present disclosure. As Figure 9 shown, the process includes steps 901-911.

[0187] Step 901: Divide all the second goods corresponding to num2 into two parts: the second associated goods that have been associated and the second non-associated goods that have not been associated.

[0188] As described above, Figure 9 the prerequisite condition of the shown flowchart is that num1>num2 and both are greater than 0. At this time, the num1 first goods can be divided into first associated goods and first non-associated goods; and the num2 second goods can be divided into second associated goods and second non-associated goods. Among them, the above division is realized based on the association relationship between the first goods and the second goods. Any first associated good after division has a second associated good associated with it. Similarly, any second associated good also has a first associated good associated with it. In other words, the first associated goods and the second associated goods are in one-to-one correspondence. And the first non-associated goods do not have second goods associated with them, and the second non-associated goods do not have first goods associated with them, which will not be elaborated here. Suppose the corresponding relationship after division is as shown in Table 1 below:

[0189] First Goods Second Goods Whether Associated A1 B1 Associated A2 B2 Associated A3 B3 Not Associated A4 Not Associated

[0190] Table 1

[0191] It can be seen that num1 = 4, num2 = 3. The first associated goods include A1-2, the first non-associated goods include A3-4; the second associated goods include B1-2, and the second non-associated goods include B3.

[0192] Step 902: Traverse mainly with num1 paths, associate the first associated goods with the second associated goods, and associate the first non-associated goods with the second non-associated goods.

[0193] The specific association process can be referred to the detailed description in steps 905 - 911, which will not be elaborated here for the time being.

[0194] Step 903, determine whether the quantity of the second unassociated goods is equal to 0.

[0195] After traversing the first associated goods, if the quantity of the second unassociated goods is equal to 0, the traversal process of the first unassociated goods can be directly ended (not shown in the figure); on the contrary, when the quantity of the second unassociated goods is greater than 0, it can further proceed to step 904.

[0196] Step 904, traverse the first unassociated goods and update the final count result of the goods.

[0197] As shown in Table 1, A3 and A4 can be traversed in sequence to further determine whether there is an association relationship between them and B3, and update the quantity of the corresponding goods recorded in the goods count list according to the above relationship and their movement types (take out or put in), which will not be elaborated here.

[0198] Step 905, obtain the association relationship between the first goods and the second goods.

[0199] That is, determine the above information such as the first associated goods, the second unassociated goods, the second associated goods, and the second unassociated goods.

[0200] Step 906, determine whether the current goods have not been associated and the value of its temporary deletion flag bDelete is true.

[0201] Step 907, set bRealDel = true for the current goods and delete the tracking information of the current goods in the goods movement list.

[0202] Whether the current goods have not been associated (the second goods), that is, the current goods are the first unassociated goods. As shown in Table 1, if the value of the temporary deletion flag bDelete of A3 (there is no associated second goods) is true, it indicates that the goods have been temporarily deleted (it may not be recognized in the previous one or more historical frame images of the current frame image). At this time, since the second camera also does not detect the goods, it can proceed to step 907 to delete the goods to avoid subsequent continuous tracking and save the computing resources of the intelligent storage cabinet.

[0203] Step 908, determine whether the current goods are the first associated goods.

[0204] Conversely, if the current item is the first associated item or its bRealDel = false, then deleting the item should be avoided. At this time, step 908 can be entered to further determine whether the item is the first associated item.

[0205] Step 909: Update the current count result according to the current status of both associated parties.

[0206] If the current item is the first associated item, step 909 can be entered to update the current count result of the corresponding item in the item count list according to the current status of the first associated item and its corresponding second associated item. For the specific process, reference can be made to the detailed description below for the appendix Figure 10 which will not be elaborated here for the time being.

[0207] Step 910: Update the count result of the current item.

[0208] For the specific process, reference can be made to the detailed description below for the appendix Figure 11 which will not be elaborated here for the time being.

[0209] Step 911: Check if the current item has an associated relationship with the list of items that are not associated with the other party.

[0210] Similar to step 904, when the current item is the first non-associated item and its bRealDel = false, the current item can be verified for an association with each second non-associated item to further determine whether there is an associated relationship between the two, and the quantity of the corresponding item recorded in the item count list can be updated accordingly based on the above relationship and their movement types (take out or put in), which will not be elaborated further

[0211] Figure 10 is a flowchart showing a process of updating the current count result according to the current status of both associated parties in an exemplary embodiment of the present disclosure. As Figure 10 shown, the process includes steps 1001 - 1013.

[0212] Step 1001: Determine the item identifier of the second associated item associated with the current item (hereinafter referred to as the associated ID).

[0213] As mentioned above, Figure 10 the two parties are any first associated item and its corresponding second associated item. Here, A1 and B1 shown in Table 1 will be used as an example for illustration.

[0214] Step 1002: Check if the associated ID is recorded in the moving item list of the other party.

[0215] Determine whether the goods identifier of B1 is recorded in the second moving goods list. It can be understood that if the goods identifier is recorded in the second moving goods list, the tracking information of B1 will naturally also be recorded in the second moving goods list.

[0216] If the goods identifier is recorded in the second moving goods list, step 1004 can be entered; otherwise, step 1003 can be entered.

[0217] Step 1003, update the counting result of the current goods.

[0218] For the specific update process, reference can be made to the detailed description below for Figure 11 and will not be elaborated here for the time being.

[0219] Step 1004, determine whether both bDelete of the associated ID and the current ID (i.e., the goods identifier of the current goods) are true.

[0220] Further determine the current value of bDelete of A1 and the current value of bDelete of B1. It can be understood that if both of the above bDelete are true, it indicates that both goods have been temporarily deleted, so step 1005 can be entered at this time; otherwise, step 1006 can be entered.

[0221] Step 1005, delete the tracking information of the corresponding ID in the corresponding list respectively; and set bRealDel of the corresponding ID to true.

[0222] Delete the goods identifier of A1 and its tracking information in the first moving goods list, and delete the goods identifier of B1 and its tracking information in the second moving goods list.

[0223] Step 1006, determine whether the goods labels of the current goods and the associated goods are the same.

[0224] Determine whether the goods labels of A1 and B1 are the same: If the two labels are the same, it is possible that they are the same goods, and step 1012 can be entered at this time; conversely, if the two labels are different, they must not be the same goods, and step 1007 can be entered at this time.

[0225] Step 1007, determine whether either the current goods or the associated goods is a foreign object and its goods are also "foreign objects".

[0226] The counting label of any product is the product label corresponding to which product label the product is counted. In the case where the product labels of A1 and B1 are different, if either one is a foreign object and its counting label is "foreign object", it means that the identification result of the other product that is not a foreign object is wrong. At this time, it can be corrected by going to step 1008. Otherwise, it can be further judged by going to step 1009.

[0227] Step 1008, correct the counting result under the current counting label; obtain the counting label corresponding to the label that is not a foreign object in the two labels, and update the new non-foreign object label as the counting result of the current label.

[0228] For example, when the product labels of A1 and B1 are "Coke" and "Foreign Objects" respectively, and the counting label of B1 is "Foreign Objects", the current quantity under the counting label of "Foreign Objects" can be reduced by one; and the product label of B1 can be updated to "Coke", and the current quantity under the counting label of "Coke" can be increased by one.

[0229] In addition, when the values ​​of the movement type identifier behavior_status of A1 and B1 are both -1, it indicates that A1 and B1 are both taken out of the smart container. At this time, the current quantities under the two counting labels "Coke" and "Foreign Objects" can be reduced by one respectively.

[0230] Step 1009, determining whether the current product and the associated product are not foreign objects, and whether the counting tags of the two are not "foreign objects".

[0231] Step 1010, respectively update the product counting results of both and dissolve the association; and simultaneously update the associated part and the unassociated part of the smaller party.

[0232] If A1 and B1 are not foreign objects, and their counting labels are "Coke" and "Milk" respectively - obviously, they are not the same product. At this time, the association between A1 and B1 is wrong. Therefore, the association between A1 and B1 can be removed first, and the second associated product and the second non-associated product in the second product can be updated, such as updating B1 from the second associated product to the second non-associated product.

[0233] Step 1011, update the current product counting result.

[0234] For the specific update process, please refer to the following Figure 11 The detailed description is not repeated here.

[0235] Step 1012, determining whether the current movement type of the current item is the same as the current movement type of the associated item.

[0236] Step 1013, update the counting result of the associated goods on the lesser party; dissolve the association relationship and update the associated part and the unassociated part on the lesser party at the same time.

[0237] If the movement types of A1 and B1 are different, the counting results corresponding to each product corresponding to B1 can be updated; and the association between A1 and B1 can be released, and the second associated product and the second non-associated product in the second product can be updated, such as updating B1 from the second associated product to the second non-associated product.

[0238] Figure 11 FIG. 1 is a flow chart showing an exemplary embodiment of the present disclosure for updating the current product counting result. Figure 11 As shown, the process includes steps 1101-1105.

[0239] Step 1101, determining whether the previous motion type identifier (corresponding to the historical frame image) and the current motion type identifier (corresponding to the current frame image) are the same, that is, whether current_behavior_status is equal to last_behavior_status.

[0240] If the two are not equal, it indicates that the current product has crossed the boundary during the movement, and the products corresponding to the behavior need to be counted, and the process can proceed to step 1102; otherwise, it indicates that the product has not crossed the boundary, and the next product can be traversed.

[0241] Step 1102, determine whether the counting tag of the current product is the same as the product tag, whether the counting tag is a "foreign object", and whether the product corresponding to the above-mentioned taking out or putting in behavior has been counted in the product counting list.

[0242] If the current item's counting tag is different from the item's label, it indicates that the current item's counting tag is wrong, that is, the counting result is wrong; if the counting tag is "foreign matter", it indicates that the current item is actually a product, but it is counted under the "foreign matter" label, and the counting result can be updated in step 1103. On the contrary, if the current item's counting tag is the same as the item's label or it has not been counted, there is no need to update the counting result, and the counting result can be updated in step 1104.

[0243] Step 1103, update the value corresponding to the current count tag, such as subtract 1 from "foreign matter" and add 1 to "cola", or add 1 to "foreign matter" and subtract 1 from "cola", etc. In addition, bCount needs to be reset to false.

[0244] Step 1104, determine whether the goods corresponding to the taking out or putting in behavior have been counted in the goods counting list; and whether the counting label and the goods label of the current goods are the same and whether both labels are not "foreign objects".

[0245] If the above conditions are met, it means that the same current product is counted under different counting tags. At this time, it is necessary to proceed to step 1105 to update the counting result.

[0246] Step 1105, update the counting result of the current label; and set bCount to true, and unify the counting label and product label of the current product.

[0247] At this point, the statistics and updates of the number of goods corresponding to different types of goods and different moving methods in the goods counting list are completed, and the goods counting list now records accurate counting results. After that, when the smart container detects that the container door is closed, it can send a counting message containing the above counting results to the preset statistics party so that the statistics party can perform intelligent analysis based on this. Of course, the smart container can also perform analysis based on the above counting results locally, and send supply or placement suggestions for the goods to the management party to improve the sales effect of the goods. In addition, the smart container described in the present disclosure can also have functions such as automatic payment, which will not be repeated here.

[0248] As can be seen from the above embodiments, this solution uses two cameras (i.e., the first camera and the second camera) to collect videos for the goods placement area respectively, and comprehensively identify the first goods and the second goods based on the current frame images collected respectively, and then determine the target goods to be moved based on the identification results of the goods in the two images. It can be understood that, on the one hand, the use of two cameras with lower costs can effectively reduce the hardware cost of the smart cabinet compared to gravity sensors and disposable RFID wireless radio frequency tags; on the other hand, because the first camera and the second camera are respectively installed at different positions outside the goods placement area, the video shooting angles of the two cameras are not the same, so in the process of goods tracking, it is possible to avoid the loss of goods caused by the occlusion of hands or other objects (the possibility of simultaneous loss of two cameras is extremely low); and the target goods to be moved are comprehensively determined by the identification results of the first goods and the second goods by the two cameras, which helps to improve the recognition accuracy of the target goods.

[0249] Corresponding to the above-mentioned embodiment of the product identification method, the present disclosure also provides an embodiment of a product identification device.

[0250] The present disclosure provides a product identification device, the device comprising one or more processors, the processors being configured to:

[0251] Acquire a first current frame image and a second current frame image respectively captured by a first camera and a second camera for a product placement area, wherein the first camera and the second camera are respectively installed at different positions outside the product placement area;

[0252] Identify the first item in the first current frame image and the second item in the second current frame image respectively, and determine the target item being moved at the current moment according to the recognition results of the first item and the second item;

[0253] When the target item type and the target movement mode of the target item are determined, update the quantity of the item corresponding to the target item type and the target movement mode in the item count list according to the quantity of the target item, where the target movement mode includes taking out from or putting into the item placement area.

[0254] In one embodiment, for any one of the first current frame image and the second current frame image, identify the item being moved in the any one of the current frame images, and the processor is further configured to:

[0255] Detect the items included in the any one of the current frame images and their current coordinates;

[0256] When the current coordinates of the item are different from the historical coordinates of the item, determine that the item is the item being moved, where the historical coordinates are the position coordinates of the item in the historical frame image, and the historical frame image is a video frame image in the video to which the any one of the current frame images belongs and is before the any one of the current frame images;

[0257] Identify the item type of the item being moved.

[0258] In one embodiment, edge marking frames located at the edge of the item placement area are set in each video frame image included in the video to which the any one of the current frame images belongs, and the processor is further configured to:

[0259] When it is determined that the item being moved is outside the edge marking frame at the current moment, identify the item type of the item being moved; or,

[0260] When it is determined that the item being moved is moved from the inside of the edge marking frame to the outside, identify the item type of the item being moved.

[0261] In one embodiment, the processor is further configured to:

[0262] If the item being moved detected from a continuous preset number of video frame images including the any one of the current frame images is on the same side of the edge marking frame, determine that the item is on this side of the edge marking frame at the current moment.

[0263] In one embodiment, the edge marking frame includes an inner marking frame and an outer marking frame with a preset interval, and the processor is further configured to:

[0264] If the moved goods are within the inner marking frame at any moment, it is determined that the moved goods are inside the edge marking frame at the any moment; and,

[0265] If the moved goods are outside the outer marking frame at any moment, it is determined that the moved goods are outside the edge marking frame at the any moment.

[0266] In one embodiment,

[0267] Detecting the goods and their current coordinates included in the any current frame image includes: inputting the any current frame image into a goods detection model and obtaining the position information of the detected goods in the any current frame image output by the goods detection model; and / or,

[0268] Identifying the goods type of the moved goods includes: inputting the any current frame image or a cropped image obtained by cropping the detected goods in the any current frame image into a goods identification model and obtaining a type label output by the goods identification model for characterizing the goods type of the goods.

[0269] In one embodiment, the goods placement area is inside the intelligent storage cabinet, and the goods detection model and / or the goods identification model are deployed locally in the intelligent storage cabinet.

[0270] In one embodiment, for any current frame image among the first current frame image and the second current frame image, the processor is further configured to:

[0271] When the any current frame image indicates that the moved goods cross the edge of the goods placement area, identify the moving manner of the moved goods, where the moving manner includes taking out from or putting into the goods placement area.

[0272] The processor is further configured to:

[0273] When the first goods are detected in the first current frame image, record the goods information of the first goods in the first current frame list; and when the second goods are detected in the second current frame image, record the goods information of the second goods in the second current frame list;

[0274] Determine the target goods being moved at the current moment according to the goods information respectively recorded in the first current frame list and the second current frame list.

[0275] In one embodiment, there is a corresponding moving goods list in either the first video to which the first current frame image belongs or the second video to which the second current frame image belongs. The moving goods list is used to record the tracking information of the goods identified from the either video with the goods identifier as the index. For any goods detected from the current frame image of the either video, the processor is further configured to:

[0276] When it is determined that the any goods is the same as the goods represented by any goods identifier in the moving goods list, update the tracking information recorded with the any goods identifier as the index in the moving goods list according to the any goods;

[0277] When it is determined that the any goods is different from the goods represented by each goods identifier in the moving goods list, assign a new goods identifier to the any goods, and record the tracking information of the any goods with the new goods identifier as the index in the moving goods list.

[0278] In one embodiment, the tracking information recorded in the moving goods list includes a tracking status. Among them, the tracking continuous status is used to represent that the any goods is detected from the previous video frame image of the current frame image, and the tracking lost status is used to represent that the any goods is not detected from the previous video frame image of the current frame image and the any goods is detected from other historical video frame images before the previous video frame image; the processor is further configured to:

[0279] If the tracking status corresponding to the any goods identifier is the tracking continuous status, update the other tracking information recorded with the any goods identifier as the index except the first value;

[0280] If the tracking status corresponding to the any goods identifier is the tracking lost status, update all the tracking information recorded with the any goods identifier as the index, wherein the tracking lost status is updated to the tracking continuous status.

[0281] In one embodiment, determining that the any goods is the same as the goods with the tracking lost status corresponding to any goods identifier in the moving goods list includes at least one of the following:

[0282] The appearance time of the any goods is later than the appearance time of the any goods identifier;

[0283] The goods type of the any goods is the same as the goods type of the goods represented by the any goods identifier;

[0284] The distance between the any goods and the goods represented by the any goods identifier is not greater than the distance between the any goods and other goods with the tracking lost status.

[0285] In one embodiment, the processor is further configured to:

[0286] When the quantities of the first item and the second item are both greater than zero, perform a fusion process on the recognition results of the first item and the second item to determine the target item being moved at the current moment;

[0287] When the quantity of any one of the first item and the second item is greater than zero and the quantity of the other item is zero, determine the said any one item as the target item being moved at the current moment.

[0288] In one embodiment, there is a first moving item list in the first video to which the first current frame image belongs, and a second moving item list in the second video to which the second current frame image belongs. The moving item list of any video is used to record the tracking information of the items recognized from the said any video with the item identifier as the index; when the quantity of the first item is greater than the quantity of the second item, the processor is further configured to:

[0289] Determine the association relationship between each first item and each second item according to the item information, and divide the first items into first associated items and / or first non-associated items, and divide the second items into second associated items and / or second non-associated items;

[0290] Traverse each first item. Among them, if any first associated item has a second associated item associated with it, determine the said any first associated item as the target item being moved at the current moment, and update the corresponding tracking information recorded in the first moving item list according to the said any first associated item.

[0291] In one embodiment, the tracking information includes a tracking status. Among them, the tracking continuous status is used to represent that the said any item is detected from the previous video frame image of the current frame image, and the tracking lost status is used to represent that the said any item is not detected from the previous video frame image of the current frame image and the said any item is detected from other historical video frame images before the previous video frame image; the processor is further configured to:

[0292] If the said any first associated item is in the tracking lost status and there is no second associated item associated with it, delete the item information of the said any first associated item in the first moving item list.

[0293] In one embodiment, the processor is further configured to:

[0294] When any first associated item has a second associated item associated with it, determine the quantity, target item type, and target moving method of the said any first associated item;

[0295] Update the quantity of goods corresponding to the target goods type and the target movement mode in the goods count list according to the quantity.

[0296] In one embodiment, the processor is further configured to:

[0297] Query the current quantity of goods corresponding to the target goods type and the target movement mode in the goods count list;

[0298] Use the sum of the quantity of the target goods and the current quantity of goods as the updated quantity of goods corresponding to the target goods type and the target movement mode.

[0299] In one embodiment, the processor is further configured to:

[0300] In response to detecting that the blocking device is opened, send an opening message to the first camera and the second camera respectively, where the opening message is used to trigger the first camera and the second camera to start recording a first video and a second video respectively, where a first current frame image belongs to the first video and a second current frame image belongs to the second video;

[0301] In response to detecting that the blocking device is closed, send a closing message to the first camera and the second camera respectively, where the closing message is used to trigger the first camera and the second camera to stop recording the first video and the second video respectively.

[0302] In one embodiment, the processor is further configured to:

[0303] In response to detecting that the blocking device is closed, count the quantity of goods corresponding to different movement modes and different goods types according to the goods count list;

[0304] Send the quantity of goods to a preset statistical party.

[0305] An embodiment of the present disclosure also provides an intelligent container, including: a goods placement component corresponding to a goods placement area for placing goods; a first camera for collecting a first current frame image of the goods placement area; a second camera for collecting a second current frame image of the goods placement area, a processor; and a memory for storing processor-executable instructions; where the processor is configured to implement the goods recognition method described in any of the above embodiments.

[0306] An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps in the goods recognition method described in any of the above embodiments are implemented.

[0307] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the related methods, and will not be elaborated herein.

[0308] Figure 12 FIG. 4 is a schematic block diagram of a device 1200 for item identification according to an embodiment of the present disclosure. For example, the device 1200 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, a smart container, etc.

[0309] Referring to Figure 12 , the device 1200 may include one or more of the following components: a processing component 1202, a memory 1204, a power component 1206, a multimedia component 1208, an audio component 1210, an input / output (I / O) interface 1212, a sensor component 1214, and a communication component 1216, a first camera 1218, and a second camera 1220.

[0310] The processing component 1202 generally controls the overall operation of the device 1200, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 1202 may include one or more processors 1222 to execute instructions to complete all or part of the steps of the above item identification method. In addition, the processing component 1202 may include one or more modules to facilitate the interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate the interaction between the multimedia component 1208 and the processing component 1202.

[0311] The memory 1204 is configured to store various types of data to support the operation of the device 1200. Examples of such data include instructions for any application or method operating on the device 1200, contact data, phone book data, messages, pictures, videos, etc. The memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0312] The power component 1206 provides power to various components of the device 1200. The power component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1200.

[0313] The multimedia component 1208 includes a screen that provides an output interface between the device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the device 1200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0314] The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC) that is configured to receive external audio signals when the device 1200 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1204 or transmitted via the communication component 1216. In some embodiments, the audio component 1210 further includes a speaker for outputting audio signals.

[0315] The I / O interface 1212 provides an interface between the processing component 1202 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0316] The sensor component 1214 includes one or more sensors for providing an assessment of various aspects of the state of the device 1200. For example, the sensor component 1214 can detect the on / off state of the device 1200, the relative positioning of components, such as the display and keypad of the device 1200. The sensor component 1214 can also detect a change in the position of the device 1200 or a component of the device 1200, the presence or absence of user contact with the device 1200, the orientation or acceleration / deceleration of the device 1200, and a change in the temperature of the device 1200. The sensor component 1214 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 1214 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 1214 can further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0317] The communication component 1216 is configured to facilitate communication between the device 1200 and other devices in a wired or wireless manner. The device 1200 can access a communication standard-based wireless network, such as WiFi, 2G or 3G, 4G LTE, 6G NR, or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0318] In an exemplary embodiment, the device 1200 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-mentioned goods identification method.

[0319] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 1204 including instructions, and the above instructions can be executed by the processor 1222 of the device 1200 to complete the above-mentioned goods identification method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0320] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the embodiments disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0321] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

[0322] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0323] The methods and apparatuses provided by the embodiments of the present disclosure have been described in detail above. Specific examples are used herein to illustrate the principles and implementation manners of the present disclosure. The description of the above embodiments is only used to help understand the method and its core idea of the present disclosure; at the same time, for those of ordinary skill in the art, according to the idea of the present disclosure, there will be changes in the specific implementation manners and application scopes. In summary, the content of the present disclosure should not be construed as a limitation to the present disclosure.

Claims

1. A goods identification method, comprising: Obtaining a first current frame image and a second current frame image respectively collected by a first camera and a second camera for a goods placement area, wherein the first camera and the second camera are respectively assembled at different positions outside the goods placement area; Identifying a first good in the first current frame image and a second good in the second current frame image respectively. When the first good is detected in the first current frame image, recording the goods information of the first good in a first current frame list; and when the second good is detected in the second current frame image, recording the goods information of the second good in a second current frame list; And determining a target good moved at the current moment according to the recognition results of the first good and the second good, including: determining the target good moved at the current moment according to the goods information respectively recorded in the first current frame list and the second current frame list; There is a corresponding moving goods list in any one of the first video to which the first current frame image belongs and the second video to which the second current frame image belongs. The moving goods list is used to record the tracking information of the goods identified from any one of the videos with the goods identifier as the index. For any good detected from the current frame image of any one of the videos, when it is determined that the any good is the same good as the good represented by any goods identifier in the moving goods list, updating the tracking information recorded with the any goods identifier as the index in the moving goods list according to the any good; when it is determined that the any good is different from the goods represented by each goods identifier in the moving goods list, allocating a new goods identifier for the any good, and recording the tracking information of the any good with the new goods identifier as the index in the moving goods list; When the target goods type and the target moving mode of the target good are determined, updating the goods quantity corresponding to the target goods type and the target moving mode in the goods count list according to the quantity of the target good, where the target moving mode includes taking out from or putting into the goods placement area.

2. The method according to claim 1, for any one of the first current frame image and the second current frame image, identifying the goods moved in the any one of the current frame images, including: Detecting the goods included in the any one of the current frame images and their current coordinates; When the current coordinates of the goods are different from the historical coordinates of the goods, determining that the goods is a moved good, where the historical coordinates are the position coordinates of the goods in a historical frame image, and the historical frame image is a video frame image in the video to which the any one of the current frame images belongs and before the any one of the current frame images; Identifying the goods type of the moved good.

3. The method according to claim 2, edge marking frames located at the edge of the goods placement area are set in each video frame image included in the video to which the any one of the current frame images belongs. The identifying the goods type of the moved good includes: When it is determined that the moved item is located outside the edge marking frame at the current moment, identify the item type of the moved item; Or, When it is determined that the moved item is moved from the inside of the edge marking frame to the outside, identify the item type of the moved item.

4. The method according to claim 3, determining that the moved item is located on a certain side of the edge marking frame at the current moment includes: If the moved item detected from a continuous preset number of video frame images including the current frame image is located on the same side of the edge marking frame, it is determined that the item is on this side of the edge marking frame at the current moment.

5. The method according to claim 3, wherein the edge marking frame includes an inner marking frame and an outer marking frame with a preset interval, and determining the position of the moved item relative to the edge marking frame at any moment includes: If the moved item is located within the inner marking frame at any moment, it is determined that the moved item is located inside the edge marking frame at the any moment; And, If the moved item is located outside the outer marking frame at any moment, it is determined that the moved item is located outside the edge marking frame at the any moment.

6. The method according to claim 2, Detecting the goods included in any of the current frame images and their current coordinates includes: Input the current frame image into the item detection model, and obtain the position information of the detected item in the current frame image output by the item detection model; And / or, The identifying the item type of the moved item includes: inputting the current frame image or a cropped image obtained by cropping the detected item in the current frame image into the item identification model, and obtaining the type label for characterizing the item type of the item output by the item identification model.

7. The method according to claim 6, wherein the item placement area is inside the intelligent storage cabinet, and the item detection model and / or the item identification model are deployed locally in the intelligent storage cabinet.

8. The method according to claim 1, for any current frame image among the first current frame image and the second current frame image, further includes: When the current frame image indicates that the moved item crosses the edge of the item placement area, identify the moving mode of the moved item, and the moving mode includes taking out from or putting into the item placement area.

9. The method according to claim 1, wherein the tracking information recorded in the mobile goods list includes a tracking status, where The tracking continuous state is used to represent that any item is detected from the previous video frame image of the current frame image, and the tracking lost state is used to represent that any item is not detected from the previous video frame image of the current frame image and any item is detected from other historical video frame images before the previous video frame image; the updating the tracking information recorded with the any item identifier in the moving item list according to the any item includes: If the tracking state corresponding to the any item identifier is the tracking continuous state, update other tracking information recorded with the any item identifier except the first value; If the tracking status corresponding to any of the item identifiers is the tracking lost status, then update all the tracking information recorded with the any item identifier as the index, wherein the tracking lost status is updated to the tracking continued status.

10. The method according to claim 9, determining that the item corresponding to the tracking status of any item identifier in the moving item list being the tracking lost status is the same item, includes at least one of the following: The appearance time of the any item is later than the appearance time of the any item identifier; The item type of the any item is the same as the item type of the item characterized by the any item identifier; The distance between the any item and the item characterized by the any item identifier is not greater than the distance between the any item and other items with the tracking status of tracking lost.

11. The method according to claim 1, the determining the target item moved at the current moment according to the recognition results of the first item and the second item, includes: When the quantities of both the first item and the second item are greater than zero, perform a fusion process on the recognition results of the first item and the second item to determine the target item moved at the current moment; When the quantity of any one of the first item and the second item is greater than zero and the quantity of the other item is zero, determine the any item as the target item moved at the current moment.

12. The method according to claim 11, there is a first moving item list in the first video to which the first current frame image belongs, and a second moving item list in the second video to which the second current frame image belongs. The moving item list of any video is used to record the tracking information of the items identified from the any video with the item identifier as the index; when the quantity of the first item is greater than the quantity of the second item, the performing a fusion process on the recognition results of the first item and the second item to determine the target item moved at the current moment, includes: Determine the association relationship between each first item and each second item according to the item information, and divide the first items into first associated items and / or first non-associated items, and divide the second items into second associated items and / or second non-associated items; Traverse each first item, wherein if any first associated item has a second associated item associated with it, then determine the any first associated item as the target item moved at the current moment, and update the corresponding tracking information recorded in the first moving item list according to the any first associated item.

13. The method according to claim 12, wherein the tracking information includes a tracking status, where The tracking continued status is used to represent that the any item is detected from the previous video frame image of the current frame image, and the tracking lost status is used to represent that the any item is not detected from the previous video frame image of the current frame image and the any item is detected from other historical video frame images before the previous video frame image; the method further includes: If any first associated item is in the tracking lost status and there is no second associated item associated with it, then delete the item information of the any first associated item in the first moving item list.

14. The method according to claim 12, wherein updating the quantity of the target goods type and the target movement mode in the goods count list according to the quantity of the target goods comprises: When there is a second associated good associated with any first associated good, determining the quantity, the target goods type and the target movement mode of the any first associated good; Updating the quantity of the target goods type and the target movement mode corresponding thereto in the goods count list according to the quantity.

15. The method according to claim 1, wherein updating the quantity of the target goods type and the target movement mode in the goods count list according to the quantity of the target goods comprises: Querying the current quantity of goods corresponding to the target goods type and the target movement mode in the goods count list; Taking the sum of the quantity of the target goods and the current quantity as the updated quantity of goods corresponding to the target goods type and the target movement mode.

16. The method according to claim 1, wherein a blocking device for controlling the entry and exit of goods is assembled at the edge of the goods placement area, and the method further comprises: In response to detecting that the blocking device is turned on, sending an on message to the first camera and the second camera respectively, the on message being used to trigger the first camera and the second camera to start recording a first video and a second video respectively, wherein a first current frame image belongs to the first video and a second current frame image belongs to the second video; In response to detecting that the blocking device is turned off, sending an off message to the first camera and the second camera respectively, the off message being used to trigger the first camera and the second camera to stop recording the first video and the second video respectively.

17. The method according to claim 1, wherein a blocking device for controlling the entry and exit of goods is assembled at the edge of the goods placement area, and the method further comprises: In response to detecting that the blocking device is turned off, counting the quantity of goods corresponding to different movement modes and different goods types according to the goods count list; Sending the quantity of goods to a preset statistical party.

18. A goods identification device, the device comprising one or more processors configured to: Obtain a first current frame image and a second current frame image respectively collected by a first camera and a second camera for a goods placement area, the first camera and the second camera being respectively assembled at different positions outside the goods placement area; Identifying a first good in the first current frame image and a second good in the second current frame image respectively, and recording the goods information of the first good in a first current frame list when the first good is detected in the first current frame image; and recording the goods information of the second good in a second current frame list when the second good is detected in the second current frame image; And determining a target good being moved at the current moment according to the recognition results of the first good and the second good, including: determining the target good being moved at the current moment according to the goods information respectively recorded in the first current frame list and the second current frame list; There is a corresponding moving goods list for either the first video to which the first current frame image belongs or the second video to which the second current frame image belongs. The moving goods list is used to record the tracking information of the goods identified from either video with the goods identifier as the index. For any goods detected from the current frame image of either video, when it is determined that the any goods is the same as the goods represented by any goods identifier in the moving goods list, the tracking information recorded with the any goods identifier as the index in the moving goods list is updated according to the any goods; when it is determined that the any goods is different from the goods represented by each goods identifier in the moving goods list, a new goods identifier is assigned to the any goods, and the tracking information of the any goods is recorded with the new goods identifier as the index in the moving goods list; When the target goods type and the target moving mode of the target goods are determined, the quantity of the goods corresponding to the target goods type and the target moving mode is updated in the goods counting list, and the target moving mode includes taking out from or putting into the goods placement area.

19. An intelligent storage cabinet, characterized in that, Comprising: A goods placement component corresponding to the goods placement area for placing goods; A first camera for collecting a first current frame image of the goods placement area; A second camera for collecting a second current frame image of the goods placement area; A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 17.

20. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method according to any one of claims 1 to 17.

Images