Methods, systems, and storage media for checking the number of items in a shopping cart.

Abstract

The present disclosure provides a method, system, and storage medium for checking the number of products in a shopping cart. The method for checking the number of products in a shopping cart includes the steps of: acquiring an image in a shopping cart in real time during a shopping process, pre-processing the image, and obtaining a processed image; performing target detection and tracking on the processed image based on a deep learning model to obtain a first trajectory, and performing target detection and tracking on the processed image based on digital image processing to obtain a second trajectory; judging the shopping behavior based on the first trajectory when a product with a product code scan operation is put into or taken out of the shopping cart, judging the shopping behavior based on the second trajectory when a product with a product code scan operation is put into or taken out of the shopping cart, and obtaining a judgment result; and, when the number of products in the shopping cart does not match the number of products scanned with the code, alerting the customer on the shopping cart side and / or feeding back discrepancy information to the supermarket side.

Description

Technical Field

[0001] This application claims the priority of a Chinese patent application with the application number 202210392643.X, which was filed with the Chinese Patent Office on April 14, 2022, and all the contents of the application are incorporated herein by reference.

[0002] This disclosure relates to the technical field of computer vision, for example, a method, a system, and a storage medium for checking the number of items in a shopping cart.

Background Art

[0003] Supermarket shopping is a lifestyle that cannot be replaced by online shopping. With the needs of the market and the development of technology, smart shopping carts with self-checkout functions have already appeared in large supermarkets. Customers can scan the codes of the items they need to purchase during the shopping process and quickly check out after the shopping is completed, significantly reducing the time spent queuing for checkout in traditional shopping.

[0004] In related technologies, the smart shopping cart primarily prevents scanning errors by determining whether the number of items in the shopping cart matches the number of items in the customer's shopping list. This technology can be implemented using hardware, such as a gravity-sensitive scale, where the weight of each item is pre-stored in a database. During the purchasing process, the weight of the scanned items, retrieved from the database, is compared with the change in weight measured by the gravity-sensitive scale. This technology can also be implemented using software, such as by image difference recognition, where foreground and background images of the shopping cart are calculated using difference and background modeling, and matching and recognition are performed on the foreground image. Alternatively, it can be implemented using skin tone modeling, where a motion target is acquired using difference, and the skin tone model determines whether the motion target is holding an item. If it is determined that the motion target is holding an item, a neural network model recognizes the item. A camera head acquires images of items being placed in and removed from the shopping cart, and the number of purchased items is recognized through image preprocessing, feature extraction, and a neural network model.

[0005] Related technologies have at least the following problems: hardware-dependent solutions require communication between multiple devices, leading to complex maintenance later on and high initial investment costs; conventional image processing solutions are highly dependent on the environment and background, resulting in significantly reduced recognition performance and poor generalization when the environment and other elements interfere; and methods for recognizing products based on neural network models require the creation of large product datasets, leading to misrecognition and detection failures when products overlap if new products are not updated in a timely manner. [Overview of the project]

[0006] This disclosure provides a method, system, and storage medium for checking the number of items in a shopping cart.

[0007] In the first mode, A method for checking the number of items in a shopping cart, and this method is The process involves acquiring images of the shopping cart in real time during the shopping process, pre-processing the images in the shopping cart, and obtaining the processed images. A step comprising: obtaining a first trajectory by performing target detection and tracking based on a deep learning model on the processed image; and obtaining a second trajectory by performing target detection and tracking based on digital image processing on the processed image, wherein the acquisition of the first trajectory and the acquisition of the second trajectory are performed simultaneously. The steps include: determining whether a shopping activity has occurred based on the first trajectory when a product whose product code has been scanned is added to or removed from the shopping cart; determining whether a shopping activity has occurred based on the second trajectory when a product whose product code has not been scanned is added to or removed from the shopping cart; and obtaining the determination result. The process includes, if, based on the aforementioned determination, it is determined that the number of items in the shopping cart does not match the number of items scanned by code, at least one of the following steps is taken: the shopping cart system issues a warning to the customer, and / or the mismatch information is fed back to the supermarket. This provides a way to check the number of items in your shopping cart.

[0008] In the second embodiment, A system for checking the number of items in a shopping cart, and this system is A shopping cart configured to hold products, An image acquisition terminal is provided above one of the sides of the shopping cart and is configured to acquire images within the storage area of ​​the shopping cart in real time. A code scanning terminal is provided in the shopping cart and is configured to perform a code scanning operation on the products. The shopping cart includes a payment terminal configured to verify the number of items in the shopping cart based on a method that links images within the shopping cart's storage area acquired with code scan information for the products, and verifies the number of items in the shopping cart. We provide a system to check the number of items in a shopping cart.

[0009] In the third aspect, The present invention provides a storage medium that stores computer instructions that, when executed by a processor, enable a method for determining the number of items in a shopping cart. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of the flow chart for a method of checking the number of items in a shopping cart according to an embodiment of the present disclosure. [Figure 2] This is a schematic diagram of a flowchart illustrating a method for checking the number of items in another shopping cart according to an embodiment of the present disclosure. [Figure 3] This is a schematic diagram of a system for checking the number of items in a shopping cart according to an embodiment of the present disclosure. [Figure 4] This is a schematic diagram of a processed image according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0011] The present application will be described and explained below with reference to the drawings and embodiments. The specific embodiments described herein are merely for the purpose of interpreting the present application.

[0012] The drawings in the following description are merely some examples or embodiments of the present invention.

[0013] The term "examples" as used in this application means that certain features, structures, or characteristics described in conjunction with an example may be included in at least one example of this application. Even if the phrase appears in multiple locations in the specification, they do not necessarily refer to the same example, nor do they represent mutually exclusive, independent, or alternative examples. Furthermore, the terms "first," "second," and "third" are merely descriptive and should not be understood as indicating or implying relative importance.

[0014] Unless otherwise defined, technical or scientific terms relating to this application should have the ordinary meaning understood by a person skilled in the art. Similar phrases relating to this application, such as “1,” “one,” “one kind,” and “the,” are not intended to indicate a limitation of number and may refer to singular or plural. The terms “include,” “incorporate,” “have,” and any variations thereof relating to this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to the listed steps or units and may include steps or units not listed or other steps or units specific to those processes, methods, products, or apparatus.

[0015] In the embodiments of this disclosure, "and / or" describes the relationship between related objects and indicates that three types of relationships are possible. For example, "A and / or B" can represent three situations: A existing alone, A and B existing simultaneously, and B existing alone. The letter " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0016] The following describes the concepts related to the embodiments of this disclosure. Optical flow is the instantaneous velocity of a pixel moving in the observation plane of an object moving in space. The optical flow method is a method for calculating motion information of objects in adjacent frames by finding the corresponding relationship between the previous frame and the current frame based on the time domain changes of pixels in an image sequence and the relationships between adjacent frames.

[0017] The target detection algorithms based on deep learning models include one-stage methods such as the YOLO (You Only Look Once) system and the Single Shot MultiBox Detector (SSD) system. The main idea is to uniformly perform dense sampling that can adopt different scales and aspect ratios at different positions of the picture or feature map, and directly predict the type and position information of the target using the features extracted by the convolutional neural network. And two-stage methods such as the Regions-Convolutional Neural Network (R-CNN) system. The main idea is to first generate a series of sparse region methods by heuristic methods or convolutional neural networks, then classify and regress these bounding boxes, and finally summarize the results.

[0018] The optical flow algorithm of the Dense Inverse Search-based method (DIS) is an algorithm that balances the optical flow quality and the calculation time.

[0019] Regarding the self-checkout shopping cart, if a customer takes an unpaid item due to an intentional or unintentional scan omission of the item, it will cause losses to the supermarket. If the customer no longer wants the item that has been code-scanned but forgets to code-scan and return it, it will cause losses to the customer and reduce the satisfaction of the customer's shopping experience.

[0020] Figure 1 is a schematic diagram of the flow of a method for checking the number of items in a shopping cart according to an embodiment of the present disclosure. As shown in Figure 1, an embodiment of the present disclosure provides a method for checking the number of items in a shopping cart, the method including: step S1 of acquiring an image of the shopping cart in real time during the shopping process, preprocessing the image of the shopping cart to obtain a processed image; step S2 of obtaining a first trajectory by performing target detection and tracking based on a deep learning model on the processed image, and obtaining a second trajectory by performing target detection and tracking based on digital image processing on the processed image, with the acquisition of the first and second trajectories occurring simultaneously; step S3 of making a judgment on a shopping activity based on the first trajectory when an item for which a product code scan operation has been performed is added to or removed from the shopping cart, and making a judgment on a shopping activity based on the second trajectory when an item for which a product code scan operation has been performed is added to or removed from the shopping cart, and obtaining a judgment result; and step S4 of issuing a warning to the customer and / or feeding back the mismatch information to the supermarket if the number of items in the shopping cart and the number of items for which a code scan has been performed do not match among the judgment results.

[0021] The method for checking the number of items in a shopping cart according to an embodiment of the present disclosure can achieve the following technical effects. During the shopping process, according to the shopping state, different trajectory detection and tracking methods are used to determine the shopping behavior, and further, by determining whether the number of items in the shopping cart matches the number of items scanned by the code, it can accurately arouse the customer's attention as to whether the number of items in the shopping cart matches the number of items in the shopping list, friendly arouse the customer's attention, effectively improve the customer's shopping experience, avoid unnecessary losses in the supermarket, be adaptable to various complex application scenarios, be able to check the number of items in the shopping cart in an application scenario where the change of light irradiation is frequent, have good generalization, have low requirements for equipment, and compared with a solution that depends on hardware equipment, can reduce the early investment of the equipment and is also beneficial for later maintenance.

[0022] By feeding back the abnormal information to the terminal device of the supermarket, when the customer is about to leave the store, it is possible to conduct an inspection by human intervention on the shopping cart with abnormal shopping information. For the supermarket, it can effectively avoid losses.

[0023] The method for checking the number of items in a shopping cart according to an embodiment of the present disclosure is applicable to a smart shopping cart with a self-checkout function in which an image acquisition terminal, a code scanning terminal, and a settlement terminal are integrated. The smart shopping cart with the self-checkout function requires that each item placed in the shopping cart is an item for which a code scanning operation has been performed. If an item for which a code scanning operation has not been performed is placed in the smart shopping cart, the smart shopping cart will give corresponding reminders.

[0024] In some embodiments, the image preprocessing includes grayscale conversion, geometric transformation, mask processing, and image enhancement. The deep learning models used include SSD-based models, YOLO-based models, and R-CNN-based models.

[0025] Target detection and tracking based on digital image processing can employ methods such as background subtraction or optical flow.

[0026] In some embodiments, obtaining a first trajectory by performing target detection and tracking based on a deep learning model on a processed image includes: step S21, which detects whether or not there is a hand in the processed image of the current frame; if there is a hand in the processed image of the current frame, proceeding to step S22; and if there is no hand in the processed image of the current frame, proceeding to step S23; step S22, which calculates whether or not the hand detection frame in the processed image of the current frame matches the current trajectory; if there is a match, adding the processed image of the current frame to the image set corresponding to the current trajectory and obtaining a first trajectory based on the processed current trajectory; and step S23, which puts the state of the current trajectory into a pending state, and after a predetermined time interval, puts the state of the current trajectory into a completed state.

[0027] The first trajectory obtained in step S22 is the trajectory obtained for one instance of the hand taking an item into and out of the shopping cart; in other words, multiple first trajectories corresponding to multiple instances of the user's hand taking an item are obtained.

[0028] Before calculating whether the hand detection frame matches the current trajectory, the process further includes determining whether the hand is holding an item. By marking the image corresponding to the hand detection frame where the hand is holding an item and the image corresponding to the hand detection frame where the hand is not holding an item with different marks, the type of first trajectory can be determined subsequently based on these marks, and different determination results can be determined based on the different types of first trajectories.

[0029] If the hand detection frame and the current trajectory do not match, it indicates that the currently detected hand detection frame may be a detection frame where another hand entered the image and was detected. For example, when different users are shopping using the same shopping cart, different hands may be picking up items from the shopping cart. If it is determined that the hand detection frame and the current trajectory do not match, a new trajectory can be created, that is, a new trajectory can be created for the currently detected hand detection frame, meaning that in this embodiment, multiple first trajectories may exist simultaneously. The above embodiment ensures that multiple users can use one shopping cart simultaneously, improving the customer experience and ensuring accuracy of recognition.

[0030] For the processed image of the current frame, a matching operation is performed between the position of the hand obtained by target detection and the position of the hand in the current trajectory. For example, the area of ​​the overlapping region between the position of the hand obtained by target detection and the most recently determined position of the hand in the current trajectory is calculated. If the area of ​​this overlapping region is greater than a preset threshold, it is determined that the position of the hand obtained by target detection matches the current trajectory. In this case, the position of the hand obtained by target detection is located in the current trajectory, and the processed image of the current frame is added to the image set corresponding to the current trajectory. If the position of the hand obtained by target detection does not match the position of the hand in the current trajectory, the trajectory in which the position of the hand obtained by target detection is located may be another new trajectory. The current trajectory that is put into a pending state without matching with a new image frame is called a pending trajectory, and the pending state of the trajectory is represented by the pending trajectory.

[0031] If there is no hand in the processed image of the current frame, the current trajectory state is put into a pending state, and after a predetermined time interval, the current trajectory state is changed to a completed state, and it is also possible to determine the predicted trajectory. The predicted trajectory is a trajectory obtained by calculating and predicting the direction and distance of hand movement when no hand movement is detected. In this embodiment, if there is no hand in the processed image of the current frame, for example, if there is no hand in the image due to a software or hardware operation error, the direction and distance of hand movement in future time periods can be predicted based on the already determined direction and speed of hand movement in the current trajectory, and the trajectory determined based on the predicted direction and distance of hand movement is the predicted trajectory. If a processed image containing a hand is detected again within a predetermined time interval while the current trajectory is in a pending state, the hand detection frame in the processed image can be matched with the most recently predicted hand position in the predicted trajectory. This matching method is similar to the method used to match the hand detection frame with the current trajectory, thereby determining whether the position of the hand detection frame in the processed image is located on the predicted trajectory. If the position of the hand detection frame in the processed image is located on the predicted trajectory, the trajectory of the hand detection frame's position is then determined, starting from the processed image. This subsequently determined trajectory is called the matching trajectory, and the matching trajectory is considered to be a trajectory that matches the predicted trajectory or the current trajectory in a pending state.

[0032] In the above embodiment, the current trajectory, predicted trajectory, and matching trajectory may all be the first trajectory that is finally determined. By setting the predicted trajectory and the matching trajectory, the trajectory accuracy can be improved and the fault tolerance of this embodiment can be improved.

[0033] In some embodiments, step S2, which involves performing target detection and tracking based on digital image processing on the processed image to obtain a second trajectory, includes step S24, which detects whether or not there is a moving object in the processed image of the current frame, and proceeds to step S25 if there is a moving object in the processed image of the current frame, and proceeds to step S26 if there is no moving object in the processed image of the current frame; step S25, which adds the processed image of the current frame to the image set corresponding to the current trajectory and obtains a second trajectory based on the processed current trajectory; and step S26, which puts the state of the current trajectory into a pending state.

[0034] Step S25 involves matching the position of the moving object with the current trajectory. If the position of the moving object matches the current trajectory, the processed image of the current frame is added to the image set corresponding to the current trajectory. If the position of the moving object does not match the current trajectory, i.e., if the current trajectory (optical flow) does not match the position of the moving object, a new trajectory can be created. This operation is similar to the operation when the hand detection frame and the current trajectory do not match in the above embodiment and will not be described again here. The optical flow method includes the DIS optical flow algorithm, optical flow direction, and contour area filtering. The presence or absence of a moving object in the processed image of the current frame can be detected using the optical flow method or background subtraction.

[0035] In step S26, after the current trajectory state is put into a pending state, the direction and distance of movement of the moving object in a future time period can be predicted based on the direction and velocity of movement of the moving object that have already been determined in the current trajectory. The trajectory determined based on the predicted direction and distance of movement of the moving object is the predicted trajectory. If a processed image of the moving object is detected again in a predetermined time interval while the current trajectory is in a pending state, the position of the moving object in the processed image can be matched with the most recently predicted position of the moving object in the predicted trajectory. The matching method is similar to the method of matching the hand detection frame with the current trajectory, and this makes it possible to determine whether the position of the moving object in the processed image is located on the predicted trajectory. If the position of the moving object in the processed image is located on the predicted trajectory, the trajectory of the position of the moving object is determined starting from the processed image, and this subsequently determined trajectory is called the matching trajectory. The matching trajectory is considered to be a trajectory that matches the predicted trajectory or the current trajectory that is in a pending state. In the above embodiment, the current trajectory, predicted trajectory, and matching trajectory may all be the final determined second trajectory.

[0036] In some embodiments, step S3 further includes determining whether the first and second trajectories have ended, determining whether the ended trajectories are valid, recording trajectory information for trajectories that have not ended, proceeding to step S1, and making a shopping action determination for the valid first or second trajectory. Specifically, if the number of pending frames for the first trajectory is greater than the first preset threshold, it is determined that the first trajectory has ended; if the number of pending frames for the first trajectory is less than or equal to the first preset threshold, it is determined that the first trajectory has not ended; if the number of pending frames for the second trajectory is greater than the first preset threshold, it is determined that the second trajectory has ended; and if the number of pending frames for the second trajectory is less than or equal to the first preset threshold, it is determined that the second trajectory has not ended. If the hand cannot be detected, the direction of hand movement is calculated and predicted, a correction calculation is performed to obtain a trajectory, and if the number of trajectory frames is greater than the number of pending frames, it is considered that the trajectory has already ended.

[0037] In some embodiments, determining whether a completed trajectory is valid or not involves obtaining the trajectory length of the completed trajectory, and determining that the completed trajectory is valid if the trajectory length is greater than a second preset threshold, and invalid if the trajectory length is less than or equal to the second preset threshold.

[0038] In some embodiments, when a product code scan operation is performed in step S3 and the product is added to or removed from the shopping cart, the determination of the shopping activity based on a valid first trajectory is made in step S31, which classifies the valid first trajectory into first-class trajectories, second-class trajectories, and third-class trajectories based on the order of time when images of hands holding a product and images of hands not holding a product were added to the image set corresponding to the valid first trajectory, and if the valid first trajectory is a first-class trajectory, the first ratio of the number of frames of images of hands holding a product to the number of frames of images not holding a product is obtained, and the first ratio Step S32 includes determining whether the product has been placed in the shopping cart if the first ratio is greater than the third preset threshold, and determining whether the product has not been placed in the shopping cart if the first ratio is less than or equal to the third preset threshold; and step S33 includes determining whether the valid first trajectory is the third type trajectory, obtaining the second ratio of the number of frames of images in which the hand is holding the product to the number of frames in which the hand is not holding the product, determining whether the product has been removed from the shopping cart if the second ratio is greater than the fourth preset threshold, and determining whether the product is still in the shopping cart if the second ratio is less than or equal to the fourth preset threshold. In this way, by recognizing the process of shopping based on the trajectory of the product, it is possible to accurately recognize the act of adding or removing a product from the shopping cart after a code scan operation has been performed, and by classifying valid trajectories, it is possible to effectively resolve situations in which the operation stagnates within the monitoring range of the camera head and misrecognition of the operation occurs.

[0039] In step S31, a first trajectory in which the time of adding an image of a hand holding a product in the image set is earlier and the time of adding an image of a hand not holding a product is later can be designated as a first-class trajectory, a first trajectory in which the time of adding an image of a hand holding a product is later and the time of adding an image of a hand not holding a product is earlier can be designated as a third-class trajectory, and trajectories other than the first and third-class trajectories can be designated as second-class trajectories. The second-class trajectory can correspond to a trajectory in which items are taken in and out of a shopping cart without hands.

[0040] In some embodiments, if a product is added to or removed from the shopping cart in step S3 for which a product code scan operation has not been performed, determining the shopping activity based on a valid second trajectory includes step S34, which determines whether a product has been added to or removed from the shopping cart based on the optical flow direction of the valid second trajectory. In this way, based on the optical flow trajectory of the product, if a code scan operation has not been performed, the product added to or removed from the shopping cart is confirmed.

[0041] In some embodiments, obtaining a determination result in step S3 includes determining that an item has been brought in if the item has been placed in the shopping cart and remains in the shopping cart; determining that an item has been removed if the item has not been placed in the shopping cart and remains in the shopping cart; determining that an item has been exchanged if the item has been placed in the shopping cart and remains in the shopping cart; determining that an item was taken in and out without a cart if the item has not been placed in the shopping cart and remains in the shopping cart; determining that an item has escaped if the optical flow direction is away from the shopping cart; and determining that an item has been thrown in if the optical flow direction is toward the shopping cart. In this way, the number of items in the shopping cart can be accurately statistically determined, and the consistency between the number of items in the shopping cart and the number of items scanned by code can be determined.

[0042] In this embodiment, the determination result for determining a shopping activity based on the first trajectory includes a determination that the item was brought in, obtained when the item was placed in the shopping cart and the item remained in the shopping cart; a determination that the item was taken out, obtained when the item was not placed in the shopping cart and the item was taken out of the shopping cart; a determination that the item was exchanged, obtained when the item was placed in the shopping cart and the item was taken out of the shopping cart; and a determination that the item was taken out and put in without hands, obtained when the item was not placed in the shopping cart and the item remained in the shopping cart. The determination result for determining a shopping activity based on the second trajectory includes a determination that the item escaped, obtained when the optical flow direction is away from the shopping cart; and a determination that the item was thrown in, obtained when the optical flow direction is towards the shopping cart.

[0043] If both a first and second trajectory exist simultaneously for a single act of putting items into or taking them out of a shopping cart, this indicates that the act involved either putting items into or taking them out of the shopping cart while holding them by hand, or simply putting items into or taking them out of the shopping cart empty-handed. By classifying the first trajectory, it is possible to determine whether or not the act involved putting items into or taking them out of the shopping cart empty-handed. In other words, if the item is not placed in the shopping cart, and the item is still inside the shopping cart, the result is that the item was put in or taken out empty-handed. For actions involving adding or removing items from a shopping cart using a handheld method, the system first determines whether a corresponding code scan operation was performed before the item was placed in the shopping cart and / or after it was removed from the shopping cart. If a code scan operation was performed, the system uses the first trajectory to determine whether a shopping action occurred. The determination results include a determination that the item was brought in (when the item was placed in the shopping cart and is still in the shopping cart), a determination that the item was removed (when the item was not placed in the shopping cart and was removed from the shopping cart), and a determination that the item was exchanged (when the item was placed in the shopping cart and was removed from the shopping cart). If no code scan operation occurred, the system uses the second trajectory to determine whether a shopping action occurred, links it to the optical flow direction, and alerts the user that the item was added to or removed from the shopping cart without a code scan.

[0044] For a single act of putting an item in or taking it out of a shopping cart, if there is only a second trajectory and no hand is visible, it indicates that an item has been put in or taken out of the shopping cart. This is linked to whether or not the corresponding code scan operation was performed before the item was placed in the shopping cart, and a shopping action is determined. For example, if the optical flow direction is directed towards the shopping cart and it is determined that the corresponding code scan operation was performed before the item was placed in the shopping cart, the result is that the item was thrown in and the item has already been code scanned, and there is no need to issue a warning. If the optical flow direction is directed towards the shopping cart and it is determined that the corresponding code scan operation was not performed before the item was placed in the shopping cart, the result is that the item was thrown in and the item has not been code scanned, and it is necessary to warn the user to scan the code. If the optical flow direction is away from the shopping cart, the result is that the item has escaped.

[0045] In this embodiment, the first preset threshold, the second preset threshold, the third preset threshold, and the fourth preset threshold can be set by a person skilled in the art according to their actual needs.

[0046] Figure 2 is a schematic diagram of the flow of a method for checking the number of items in a shopping cart according to an embodiment of the present disclosure. As shown in Figure 2, during the shopping process, video frames of the contents of the shopping cart are read in real time, image preprocessing is performed, and target detection and tracking are performed on the processed image to obtain the trajectory of the items. Of these, target detection and tracking include target detection based on a deep learning model and target detection based on background subtraction, which are performed simultaneously. For target detection based on a deep learning model, first, target detection is performed on the image of the current frame to determine whether or not there is a hand. If there is a hand in the image of the current frame, the hand detection frame is matched with the last frame of each trajectory. If the hand detection frame and the last frame of the trajectory match, trajectory tracking is performed and the image of the current frame is added to the image set corresponding to the trajectory. If the hand detection frame and the last frame of the trajectory do not match, it is necessary to create a new trajectory for the hand detection frame. If there is no hand in the image of the current frame, the state of the current trajectory is put into a pending state. For target detection based on background subtraction, first, it is determined whether or not there is a moving object in the image of the current frame. The system determines whether a moving object exists in the current frame's image and if the object's position matches the current trajectory, it performs trajectory tracking and adds the current frame's image to the image set corresponding to the current trajectory. If the object's position does not match the current trajectory, it creates a new trajectory. If there is no moving object in the current frame's image, it puts the current trajectory into a pending state. If the number of pending frames of the current trajectory is greater than the first preset threshold, the current trajectory is considered finished. If the number of pending frames of the current trajectory is less than or equal to the first preset threshold, the current trajectory is not considered finished, and trajectory information is recorded. The system then reads video frames of the shopping cart's contents and performs the steps described above. After the current trajectory is finished, the system determines whether the current trajectory is valid. If the trajectory length of the current trajectory is less than or equal to the second preset threshold, the trajectory is invalid and cleared. If the trajectory length of the current trajectory is greater than the second preset threshold, the trajectory is considered valid, and the shopping state and actions are determined. The determination result is obtained, and in the determination result,If the number of items in the shopping cart does not match the number of items scanned by the code, the shopping cart will issue a warning to the customer, and the supermarket staff will also issue a warning on their handheld terminal.

[0047] Figure 3 is a schematic diagram of a system for checking the number of items in a shopping cart according to an embodiment of the present disclosure. Figure 4 is a schematic diagram of a processed image according to an embodiment of the present disclosure. As shown in Figure 3, an embodiment of the present disclosure further provides a system 10 for checking the number of items in a shopping cart, comprising: a shopping cart 110 configured for placing items; an image acquisition terminal 120 provided above one of the sides of the shopping cart 110 and configured to acquire an image within the shopping cart's storage area; a code scan terminal 130 provided on the shopping cart 110 and configured to perform a code scan operation on items; and a payment terminal 140 provided on the shopping cart 110 and configured to link the acquired image within the shopping cart's storage area and product code scan information, and to check the number of items in the shopping cart based on the method for checking the number of items in the shopping cart described above. The system is adaptable to various supermarket environments, has strong generalizability, reduces system costs, is easy to maintain later, and can accurately draw the customer's attention to whether the number of items in the shopping cart matches the list of purchased items.

[0048] Embodiments of the present disclosure further provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement a method for determining the number of items in a shopping cart in any of the embodiments described above.

[0049] This disclosure will be described with reference to flow diagrams and / or block diagrams of methods, systems, and computer program products according to embodiments of this disclosure. It should be understood that each flow and / or block in the flow diagrams and / or block diagrams, and the combination of flows and / or blocks in the flow diagrams and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a dedicated computer, an embedded processor, or other programmable data processing device to generate a machine in which instructions executed by the processor of the computer or other programmable data processing device generate a device for realizing the functions specified in one flow of a flow diagram or one or more blocks of multiple flows and / or block diagrams.

[0050] These computer program instructions may be stored in computer-readable memory capable of guiding a computer or other programmable data processing device to operate in a particular manner, thereby including instructions stored in the computer-readable memory as a product of an instruction unit, which implements the functions specified in one or more flows of a flow diagram and / or one or more blocks of a block diagram.

[0051] These computer program instructions may be implemented in a computer or other programmable data processing device, thereby enabling the computer or other programmable device to perform a series of operational steps to generate processing implemented by the computer, and providing instructions executed in the computer or other programmable device to implement the functionality of a specified step in one or more flows of a flow diagram and / or one or more blocks of a block diagram.

Claims

1. A method for checking the number of items in a shopping cart, and this method is The process involves acquiring images of the shopping cart in real time, pre-processing the images in the shopping cart, and obtaining the processed images. A step comprising: obtaining a first trajectory by performing target detection and tracking based on a deep learning model on the processed image; and obtaining a second trajectory by performing target detection and tracking based on digital image processing on the processed image, wherein the acquisition of the first trajectory and the acquisition of the second trajectory are performed simultaneously. The steps include: determining whether a shopping activity has occurred based on the first trajectory when a product whose product code has been scanned is added to or removed from the shopping cart; determining whether a shopping activity has occurred based on the second trajectory when a product whose product code has not been scanned is added to or removed from the shopping cart; and obtaining the determination result. The process includes, if, based on the aforementioned determination, it is determined that the number of items in the shopping cart does not match the number of items scanned by code, at least one of the following steps is taken: the shopping cart system issues a warning to the customer, or the mismatch information is fed back to the supermarket. method.

2. The step of obtaining a first trajectory by performing target detection and tracking based on a deep learning model on the processed image is as follows: A step to detect whether or not there is a hand in the image after processing of the current frame, The steps include: in response to the presence of a hand in the processed image of the current frame, calculating whether the hand detection frame in the processed image of the current frame matches the current trajectory; in response to the hand detection frame matching the current trajectory, adding the processed image of the current frame to the image set corresponding to the current trajectory, and obtaining the first trajectory based on the processed current trajectory; The process includes the steps of: in response to the absence of a hand in the processed image of the current frame, setting the current trajectory state to a pending state; and, after a predetermined time interval, setting the current trajectory state to a completed state. The method according to claim 1.

3. The step of obtaining a second trajectory by performing target detection and tracking based on digital image processing on the image after the above processing is as follows: A step to detect whether or not there is a moving object in the processed image of the current frame, In response to the presence of a moving object in the processed image of the current frame, the processed image of the current frame is added to the image set corresponding to the current trajectory, and the second trajectory is obtained based on the processed current trajectory. The process includes the step of setting the current trajectory state to a pending state in response to the absence of a moving object in the processed image of the current frame. The method according to claim 1.

4. Before the step of making a judgment about shopping behavior based on the first and second trajectories, A step of determining whether the first trajectory and the second trajectory have ended, A step of determining whether the completed trajectory is valid or not, The steps include recording trajectory information for unfinished trajectories, acquiring images of the shopping cart in real time during the shopping process, preprocessing the images of the shopping cart, obtaining processed images, performing target detection and tracking based on a deep learning model on the processed images to obtain a first trajectory, and returning to the execution of the operation to perform target detection and tracking based on digital image processing on the processed images to obtain a second trajectory, The process further includes the step of determining whether a shopping activity has occurred based on a valid first trajectory or a valid second trajectory, If the number of pending frames for the first trajectory is greater than the first preset threshold, it is determined that the first trajectory has ended; if the number of pending frames for the first trajectory is less than or equal to the first preset threshold, it is determined that the first trajectory has not ended; if the number of pending frames for the second trajectory is greater than the first preset threshold, it is determined that the second trajectory has ended; and if the number of pending frames for the second trajectory is less than or equal to the first preset threshold, it is determined that the second trajectory has not ended. The method according to claim 1.

5. The step of determining whether the completed trajectory is valid or not is: A step of obtaining the trajectory length of the completed trajectory, wherein the completed trajectory is valid if the trajectory length is greater than a second preset threshold, and the completed trajectory is invalid if the trajectory length is less than or equal to the second preset threshold. The method according to claim 4.

6. When a product code scan operation is performed and the product is added to or removed from the shopping cart, the step of determining the shopping activity based on a valid first trajectory is: The steps include classifying the valid first trajectory into a first-class trajectory, a second-class trajectory, or a third-class trajectory based on the order of time at which images of a hand holding a product and images of a hand not holding a product were added to the image set corresponding to the valid first trajectory, If the valid first trajectory is the first type trajectory, the first ratio is obtained between the number of frames of images in which the hand is holding the product and the number of frames in which the hand is not holding the product. If the first ratio is greater than a third preset threshold, it is determined that the product is in the shopping cart. If the first ratio is less than or equal to the third preset threshold, it is determined that the product is not in the shopping cart. The process includes the step of obtaining a second ratio between the number of frames in images where the hand is holding the product and the number of frames in images where the hand is not holding the product, if the valid first trajectory is the third type trajectory, determining that the product has been removed from the shopping cart if the second ratio is greater than or equal to the fourth preset threshold, and determining that the product is still in the shopping cart if the second ratio is less than or equal to the fourth preset threshold. The method according to claim 4.

7. When items that have not undergone product code scanning are added to or removed from the shopping cart, the step of determining the shopping activity based on a valid second trajectory is: The step includes determining whether to add or remove an item from a shopping cart based on the optical flow direction of the effective second trajectory, The method according to claim 6.

8. The steps to obtain the judgment result are: If an item is placed in a shopping cart and the item remains in the shopping cart, the process involves determining that the item has been brought in. The steps include obtaining a determination that the product has been removed if the product is not in the shopping cart and has been removed from the shopping cart, The steps include: obtaining a determination that the product has been replaced when the product has been placed in the shopping cart and removed from the shopping cart; The steps include: obtaining a determination that the item was taken out and put back into the shopping cart empty-handed, when the item is not in the shopping cart and the item is still in the shopping cart; The process includes the steps of: obtaining a determination that the product has escaped if the optical flow direction is away from the shopping cart, and obtaining a determination that the product has been thrown in if the optical flow direction is toward the shopping cart; The method according to claim 7.

9. A system for checking the number of items in a shopping cart, and this system is A shopping cart configured to hold products, An image acquisition terminal is provided above one side of the shopping cart and is configured to acquire images within the storage area of ​​the shopping cart in real time. A code scanning terminal is provided in the shopping cart and is configured to perform a code scanning operation on the products. The shopping cart includes a payment terminal provided in the shopping cart, which is configured to link images within the shopping cart's storage area acquired with code scan information for the products, and to verify the number of products in the shopping cart according to the method described in any one of claims 1 to 8. system.

10. A computer instruction is stored which, when executed by a processor, implements the method described in any one of claims 1 to 8. Computer-readable storage medium.

Images