A vision-based anti-loss detection method and device for a modified intelligent shopping cart

Through the modified visual-based intelligent shopping cart loss prevention detection method, the video image difference and deep learning model are used to identify items in the shopping cart, solving the problems of missing inspection and high cost of existing intelligent shopping cart detection, achieving efficient and accurate prevention Destructive testing is suitable for widespread promotion and store expansion.

CN118628967BActive Publication Date: 2024-11-12XIAN CHAOHI NET TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411110352.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-12
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

The existing anti-destructive inspection methods of smart shopping carts have missed inspection problems, especially difficult to detect backlogs or hidden goods, and the high cost and high computing power demand limit its promotion and deployment.

Method used

Adopt the modified intelligent shopping cart loss prevention detection method based on vision. By obtaining video images in the shopping cart, differential processing and item identification are performed, and the item information in the shopping list is compared with the deep learning model, reminding the unchecked items, reducing manual labor Judge costs and improve loss prevention efficiency.

Benefits of technology

It effectively improves the accuracy and loss prevention effect of items in the shopping cart, reduces the operating costs of supermarkets, avoids missed inspection and losses of goods, and is suitable for widespread promotion and store expansion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118628967B_ABST
    Figure CN118628967B_ABST
Patent Text Reader

Abstract

The present invention relates to a vision-based modified intelligent shopping cart loss prevention detection method and device, comprising acquiring a video image during a shopping process, and using the acquired video image to determine whether items in the shopping cart have changed; when it is determined that the items in the shopping cart have changed, acquiring a differential image; identifying the differential image to obtain item information in the differential image; comparing the identified item information with item information in a shopping list, and issuing a reminder if the identified item information does not belong to the items in the shopping list; effectively improving the loss prevention capability and reducing the cost of manual judgment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of smart shopping technology in supermarkets, and in particular to a vision-based loss prevention detection method and device for a modified smart shopping cart. Background Art

[0002] When shoppers are shopping in supermarkets, they often put personal belongings such as clothes, cups and bags into the shopping carts. At the same time, when they put the purchased goods into the shopping carts during the shopping process, some of the purchased goods will be mixed in with the personal belongings. During the checkout process, the mixed goods are often not checked out. When passing through the security inspection equipment, if the unchecked goods are detected, it will cause great discomfort to the customers. If they are not detected by the security inspection equipment, the unchecked goods will be taken out and cause certain losses to the supermarket.

[0003] In order to solve the above problems, some supermarkets on the market have installed AI visual inspection gates at the exits. The cameras on the inspection gates are used to detect whether the items in the user's shopping cart match the items in the checkout system. In this way, the user can push the shopping cart and leave directly after checkout. However, the limitations of this method are also obvious. When the user checks out and leaves, a large number of items have been piled up in the shopping cart. The AI ​​visual inspection gate can only detect the items visible from the outside of the cart basket, while the items that are piled up or invisible at the bottom of the cart basket cannot be detected, resulting in more missed inspections.

[0004] There are also anti-loss inspections for smart shopping carts on the market, which are carried out through quality + visual inspection. This inspection method requires the smart shopping cart device itself to have a large AI computing power, so the cost of each shopping cart device is relatively high. At the same time, due to the continuous operation of AI computing power, the power supply and heat dissipation of the shopping cart are relatively difficult to solve. Even if the heat dissipation problem can be solved, the cost is very high. Otherwise, a set of edge computing equipment must be deployed in the store to continuously send the photos taken by the shopping cart smart device to the computing device. This not only increases the operating cost and maintenance cost, but also is not conducive to the promotion of smart shopping carts. When a new store needs to open a smart shopping cart, a set of edge computing equipment must be deployed in the store again, which greatly increases the opening cost and deployment time.

[0005] Whether it is the AI ​​computing power on the smart shopping cart or the edge computing power of the store, the current method on the market is to directly identify each shopping behavior and shopping items. Considering the customer's shopping experience, it is basically required that the customer's purchased items be identified within 2 seconds after they are placed in the cart basket from the shelf. This requires very high edge computing power, and the accuracy of both action and product recognition is also very high. The high requirements for equipment and algorithm technology have greatly affected the promotion speed of smart shopping carts.

[0006] However, in order to reuse old items, many supermarkets will use the smart equipment of smart shopping carts + the body of old shopping carts to modify them into smart shopping carts. This modified smart shopping cart has the advantages of being reasonably priced and being able to reuse the original shopping carts. However, during the use of this modified smart shopping cart, it is unable to accurately detect and determine the attributes of the items in the shopping cart, which brings great inconvenience to shopping. Summary of the invention

[0007] In order to solve the above technical problems, the present application provides a method and device for loss prevention detection of a modified intelligent shopping cart based on vision, which can simply and effectively intelligently identify the attributes of items in a shopping cart through image vision, effectively reduce supermarket labor costs, and improve loss prevention efficiency and loss prevention effects.

[0008] The present application is implemented through the following technical solutions, a vision-based anti-loss detection method for a modified smart shopping cart, including:

[0009] Acquire video images during the shopping process, and use the acquired video images to determine whether items in the shopping cart have changed;

[0010] When it is determined that the items in the shopping cart have changed, the image before the change and the image after the change in the video image are differentiated to obtain a differential image;

[0011] Recognize the difference image and obtain the object information in the difference image;

[0012] The item information recognized in the differential image is compared with the item information in the shopping list to determine whether the recognized item is an item in the shopping list. If the recognized item does not belong to the items in the shopping list, a reminder is issued.

[0013] Preferably, the method further includes acquiring images in the shopping cart before shopping, and identifying the images in the shopping cart before shopping to obtain the shopper's personal belongings information.

[0014] Preferably, before using the acquired video image to determine whether the items in the shopping cart have changed, the ROI area of ​​each frame of the video image is extracted, and the image of the ROI area in each frame of the video image is used to determine whether the items in the shopping cart have changed.

[0015] Preferably, the ROI area of ​​each frame image is the area within the shopping box of the shopping cart.

[0016] Preferably, the method for determining whether the items in the shopping cart have changed includes:

[0017] The contour image of the moving target is obtained by using the difference between the current frame image and the previous frame image and the next frame image in the video image, and the area of ​​the target contour image is used to determine whether the items in the shopping cart have changed.

[0018] Preferably, the trained deep learning model is used to identify objects in the differential image.

[0019] Preferably, when the identified item does not belong to the items in the shopping list, the identified item information is compared with the shopper's personal item information. When the identified item is a personal item, no reminder is given, while when the identified item is not a personal item, a reminder is given.

[0020] A vision-based modified intelligent shopping cart loss prevention detection device, comprising:

[0021] An image acquisition device, arranged on the shopping cart, for acquiring image information in the shopping basket of the shopping cart;

[0022] A judgment module, which uses the video images collected during the shopping process to judge whether the items in the shopping cart have changed;

[0023] The difference module, when the judgment module judges that the items in the shopping cart have changed, performs a difference between the image before the change and the image after the change in the collected video image to obtain a difference image;

[0024] A recognition module, used for recognizing object information in the differential image;

[0025] A comparison module compares the item information identified by the recognition module with the item information in the shopping list to determine whether the identified item is in the shopping list;

[0026] The reminder module sends a reminder when the comparison module determines that the item identified by the recognition module is not in the shopping list.

[0027] The beneficial effects of this application are:

[0028] The video image of the shopping cart is collected during the shopping process. If the image in the shopping cart changes, it means that the items in the shopping cart have changed. The present application uses the image before each change in the shopping cart to differentiate with the image after the change to obtain each differential image, and then recognizes the differential image to obtain the item information in the differential image. The item information is compared with the information in the shopping list to determine whether the item that caused the image in the shopping cart to change is an item in the shopping list. If it is an item on the list, it means that the item has not been entered into the shopping list and there is a risk of loss. In this case, a reminder will be issued to avoid the loss of items in the supermarket, effectively improve the loss prevention capability, and reduce the cost of manual judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 It is a flow chart of a loss prevention detection method for a modified intelligent shopping cart based on vision in an embodiment of the present invention;

[0031] Figure 2 is a block diagram of a vision-based modified intelligent shopping cart loss prevention detection device in an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the structure of a smart shopping cart provided in an embodiment of the present invention;

[0033] Figure 4 Schematic diagram of the structure of the computer system of the electronic device in this embodiment. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0035] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, systems, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0038] In this embodiment, the shopping cart is a smart shopping cart. Figure 3 The shopping cart includes a body 1 and a smart terminal 3 mounted on the body 1 via a terminal mounting seat 2. The smart terminal 3 includes a scanning camera 4 for scanning and identifying purchased items. The scanning camera 4 can also collect image information in a basket in the shopping cart in real time.

[0039] Specifically, when in use, after the shopper obtains the items he needs to buy from the shelf, he scans the goods through the scanning camera 4, loads the items to be purchased into the shopping list in the smart terminal 3, and at the same time, the scanning camera 4 collects and records the image of the shopping cart basket in real time, that is, obtains all image information in the shopping cart basket during the shopping process.

[0040] In this embodiment, Figure 1 A vision-based anti-loss detection method for a modified smart shopping cart is shown, comprising:

[0041] Acquire video images during the shopping process, and use the acquired video images to determine whether items in the shopping cart have changed;

[0042] In one embodiment, before obtaining the video image of the shopping process, the shopper logs in on the smart terminal 3. After completing the login, the scanning camera 4 on the smart terminal 3 takes a picture of the basket of the shopping cart to obtain the image in the basket before shopping, and sends the image in the basket before shopping to the cloud server. The cloud server uses the trained deep learning network model to recognize the received image and obtain the items in the shopping basket of the shopping cart before shopping. Since the item is in the shopping basket before shopping, the item is recorded as the shopper's personal item on the smart terminal 3, and the photo of the personal item is also saved;

[0043] After completing the login and registration of personal belongings, the shopping process begins. During the shopping process, the scanning camera 4 on the smart terminal 3 collects the image of the shopping cart basket in real time to obtain the collected video image; since this application needs to run continuously during the shopping process, and requires the feedback to be relatively real-time and the perception to be accurate, while ensuring that the amount of calculation is not too large, the scanning camera 4 can be selected according to this requirement. The method of selecting the scanning camera 4 is a routine selection made by the staff in this technical field after considering the above factors. Therefore, this embodiment does not describe the selection of the scanning camera 4 in detail.

[0044] When collecting video images, the scanning camera 4 is used to collect images of the basket of the shopping cart during the entire shopping process, and each frame of the image and the difference between the previous frame of the image and the next frame of the image are used to determine whether the items in the basket of the shopping cart have changed. If it is determined that there is a change, the video images collected this time are used as a time sequence image set of a shopping cart image change, and the time sequence image set is a video image between the time when the items in the basket of the previous shopping cart change and the time when the items in the basket of the current shopping cart change;

[0045] Similarly, a video image is obtained each time the items in the shopping cart change, and each video image obtained is used as a time-series image set of the shopping cart image change. In this embodiment, a time-series image set of each shopping cart image change is obtained, and the time-series image set of each shopping cart image change is used to obtain the items in the shopping cart each time the items in the shopping cart change, instead of analyzing all the images in the shopping cart during calculation, which effectively reduces the computing power.

[0046] When acquiring each frame of the video image, the directly acquired image usually contains more interference, and these interference factors need to be eliminated and filtered first. Since this embodiment mainly processes and calculates the basket image of the shopping cart, only the basket image needs to be left, which can effectively reduce computing power.

[0047] In one embodiment, in order to obtain only the image inside the basket as the image for subsequent processing, this embodiment adopts the following method to obtain the image inside the basket, that is, the method for obtaining the ROI area image is that the scanning camera 4 continuously shoots the basket of the shopping cart at a speed of 5 frames per second. The photo taken during shooting is a rectangle, but the shopping basket is imaged as a trapezoid or a trapezoid-like shape with an arc on the upper edge in the image. This causes the captured photo to include images outside the shopping basket. The images outside the shopping basket may contain goods on the shelf or changes in the ground during movement. In this case, they need to be filtered out, otherwise it will interfere with the perception of customer shopping and will also affect the subsequent product recognition.

[0048] The image outside the basket of the shopping cart can be eliminated by the ROI method. The ROI is the area inside the basket of the shopping cart. The area outside the ROI is eliminated. Specifically, the image is converted into a single-channel grayscale value, which can be used for the subsequent threshold processing. In order to detect the contour, binarization + Hough transform is used for processing. The grayscale image needs to be binarized first to obtain the binarization result, and the image is converted into a black and white image, which is highlighted to facilitate the contour detection algorithm; the threshold processing converts the object area in the image into white, and all object pixels have the same brightness value. The Hough method is used to calculate the outline of the shopping basket from these white pixels; after obtaining the outline of the basket of the shopping cart, the area outside the outline is eliminated to form a shopping box image, and the image acquisition of the area of ​​interest is completed.

[0049] After obtaining the video image of each change of items in the shopping cart (time-series image set), the differential image is obtained by differentiating the image before the change and the image after the change in the video image; that is, the differential image is obtained by differentiating the start image and the end image in the time-series image set. When performing image difference, since the acquired images are all acquired through the same scanning camera, the image difference algorithm commonly used in the technical field can be directly used to acquire the differential image during the difference. Since the image difference algorithm adopts the conventional algorithm in the technical field, it will not be described in detail here.

[0050] After obtaining the differential image every time the items in the shopping cart change, the differential image is recognized, and the items that caused the change in the items in the shopping cart are obtained based on the recognition result, and then it is determined whether the items put into the shopping cart are items on the purchase list. The method for recognizing the differential image in this embodiment is the same as the method for recognizing the pictures collected in the shopping cart box before shopping, and both use a deep learning network for recognition.

[0051] In one embodiment, the deep learning network is trained as follows:

[0052] Obtain images of all merchandise and personal belongings and annotate them;

[0053] Use images of goods and personal belongings as training sets for deep learning networks;

[0054] During the training process, the pictures of goods and personal belongings are used as the input of the deep learning network, and the annotations corresponding to the pictures are used as the output of the deep learning network. The deep learning network is trained. During the training process, the training is completed when the deep learning network converges.

[0055] When using the completed deep learning network to recognize the image or differential image in the cart basket before shopping, the image or differential image in the cart basket before shopping is input into the trained deep learning network to obtain the item information in the image or differential image in the cart basket before shopping.

[0056] The above is a brief description of the training of the deep learning network. Since this application adopts a conventional deep learning network, which belongs to the conventional means in this technical field, a detailed description is not given here.

[0057] When the item information in the differential image is identified, the information of the item in the identified differential image is compared with the information of the item in the shopping list to determine whether the item in the shopping cart is the item in the shopping list;

[0058] In this embodiment, the shopping list information is obtained in the following manner: when a shopper is shopping and needs to put the selected items into the basket of the shopping cart, the shopper first needs to scan the items at the scanning camera 4 on the smart terminal 3. During the scanning process, the information of the items will be recorded on the smart terminal 3 to form a shopping list. If the shopper needs to take out the items in the shopping cart, the corresponding items on the shopping list need to be deleted on the smart terminal 3, and the items deleted from the list during shopping need to be put back into the supermarket, thus forming the shopping list information during the shopping process.

[0059] When making a comparison, the item information on the shopping list is all the items that need to be paid for. If the identified item is not on the shopping list, it means that the item placed in the shopping cart basket has not been scanned, and the item may not be paid for in the subsequent payment process, resulting in the possibility of some products being smuggled. At this time, it is necessary to remind shoppers to scan the goods in the cart basket, which can effectively improve the loss prevention effect.

[0060] In one embodiment, since there are personal items of the shopper in the basket of the shopping cart, if the identified item is not in the shopping list, the identified item is uploaded to the cloud and compared with all items in the supermarket. If the identified item is not in the commodity library of the supermarket, it means that the item is the personal item of the shopper and no reminder is given. When giving a reminder, the shopper can be reminded through the smart terminal, or the identification result can be sent to the terminal to remind the security personnel of the supermarket.

[0061] When making a comparison, you can make the comparison according to the following scheme:

[0062] The loss prevention result of this order is calculated based on the user's shopping list and the result of each product recognition. If there is no product A in the shopping list at this time, and product A is not detected in the personal belongings, but product A is detected by recognition at this time, it is considered that this user may have smuggled goods, which is a high-risk order for loss prevention. Especially for the loss prevention of valuable goods, this processing method is very efficient. If there is no product A in the shopping list at this time, and product A is identified in the personal belongings, but product A is detected by product recognition at this time, it is considered that product A is a personal belonging, and there is no loss prevention risk for this order. If there is product A in the shopping list at this time, and there is no product A in the personal belongings, but product A is detected by recognition at this time, it is considered that the customer has purchased product A normally, and there is no loss prevention risk for this order. Every time the visual sensing detects the change of goods in the basket of the shopping cart, a loss prevention detection is performed, so that each layer of goods in the basket of the shopping cart is tested for loss prevention, which can effectively prevent the detection of backlogged goods in the basket of the shopping cart or the situation where the smuggled goods are deliberately placed at the bottom of the basket of the shopping cart or mixed with other goods, resulting in loss prevention omissions.

[0063] In another embodiment, in order to accurately determine whether the items in the basket of the shopping cart have changed, the method used in this embodiment is to calculate the moving target by performing a median filter three-frame difference method on the image in the basket of the shopping cart, and determine whether the goods in the basket of the shopping cart have changed according to the size of the outline of the moving object, including:

[0064] The scanning camera on the shopping cart continuously takes pictures of the basket of the shopping cart at a rate of 5 pictures per second from the beginning of shopping to the completion of shopping, taking pictures f1, f2, f3...f t-1 、f t 、f t+1 、f t+2 ..., where f t The picture of the cart basket at the moment when the purchased goods are put into the cart basket, f1, f2, f3 ...f t-1 is the basket frame image before the purchased goods are put into the basket, f t The following is a frame image of the purchased product after it is put into the shopping cart. The steps of using the frame difference method to calculate the frame difference contour are as follows:

[0065] (1) Scale the image to speed up processing;

[0066] (2) Perform Gaussian smoothing on the image;

[0067] (3) Convert to grayscale value;

[0068] (4) Binarization processing;

[0069] (5) Collect video sequence images in a certain period of time, assuming represents the current frame at time t, and the two adjacent frames before and after it are respectively represented by and Indicates; a difference operation is performed between two adjacent frames at time t, and the judgment results after the difference operation are as follows:

[0070] | (x,y,t)-f(x,y,t-1)| <T ;

[0071] | (x,y,t)-f(x,y,t-1)|>=T;

[0072] in (x,y,t), (x, y, t-1) are the pixel values ​​of the corresponding pixels at time t and t-1 respectively, and T is the threshold.

[0073] Then, the three-frame difference method is used to obtain the two-frame difference result D t , D t-1 and D t , D t+1 The result of determining the moving target edge D'n in the current frame f(x, y, t), wherein the method for obtaining D'n includes:

[0074] D'n=|D t-1 -D t |∩|D t+1 -D t |;

[0075] (6) Median filtering: There are still some isolated noise points in D'n. If they are not processed, it will affect the effective detection of moving targets. In this embodiment, median filtering is used to filter the moving target circle, which can not only effectively filter out the isolated noise points in the image, but also effectively protect the boundary information.

[0076] (7) Get the contour of D'n and define the threshold Y. If the contour area is greater than or equal to Y, it is considered that there is a change in the basket. If the contour area is less than Y, it is considered that there is no change in the basket.

[0077] Through the above methods, it can be seen that visual perception can sense whether the basket has changed. If there is a change and a photo of the basket ROI of the shopping cart is obtained, the principle of the inter-frame difference method is simple, the algorithm is simple to implement, the program design complexity is low, it is not very sensitive to scene changes such as light, and it can adapt to various dynamic environments. The calculation amount is small and the stability is good. It can quickly detect moving targets in the scene. In this way, even if only the CPU of the smart shopping cart device is used, the changes in the goods in the basket can be perceived based on the picture calculation, and it will not cause a large amount of battery power consumption.

[0078] In one embodiment, if the smart terminal 3 detects that an item in the item list has been reduced during use, it is necessary to determine whether the items in the shopping cart have changed. If so, no reminder will be given; if not, a reminder will be given, which can effectively avoid the situation where items on the list are removed but the items in the shopping cart are not put back.

[0079] In one embodiment, when performing identification, it includes:

[0080] When it is determined that there is a change in the product in the basket, the basket ROI image is uploaded to the server for product identification. In this way, the frequency and amount of data processing do not need to be high. Product identification is deployed in the cloud server. The server contains a deep learning model trained based on supermarket product information, which can detect the products in the picture. The detection steps are divided into:

[0081] (1) Image classification: Classify product images into different categories, such as food, clothing, electronic products, etc.

[0082] (2) Object detection: Locate and identify the location and bounding box of products in the image;

[0083] (3) Object segmentation: Separate the product from the background to achieve accurate product identification.

[0084] Since the image of the shopping cart basket is uploaded to start product recognition only after visual perception detects a change in the product, the call volume and calculation volume are relatively low, and the cloud server only needs a little AI computing power. In this way, unified deployment of computing and recognition on the cloud server can fully meet the product recognition requirements during loss prevention. In this way, there is no need to configure additional AI computing power for smart devices, nor is there a need to deploy edge computing equipment in stores, which is very conducive to the promotion of modified smart shopping carts and store expansion.

[0085] A vision-based modified intelligent shopping cart loss prevention detection device, comprising:

[0086] An image acquisition device, arranged on the shopping cart, for acquiring image information in the shopping basket of the shopping cart;

[0087] A judgment module, which uses the video images collected during the shopping process to judge whether the items in the shopping cart have changed;

[0088] The difference module, when the judgment module judges that the items in the shopping cart have changed, performs a difference between the image before the change and the image after the change in the collected video image to obtain a difference image;

[0089] A recognition module, used for recognizing object information in the differential image;

[0090] A comparison module compares the item information identified by the recognition module with the item information in the shopping list to determine whether the identified item is in the shopping list;

[0091] The reminder module sends a reminder when the comparison module determines that the item identified by the recognition module is not in the shopping list.

[0092] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0093] It should be noted that the computer system 400 of the electronic device shown in the figure is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0094] The computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the vision-based modified smart shopping cart loss prevention detection method in the above embodiment. In the RAM 403, various programs and data required for system operation are also stored. The CPU 401, the ROM 402 and the RAM 403 are connected to each other through the bus 404. The input / output (I / O) interface 405 is also connected to the bus 404.

[0095] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, a touch input, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0096] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, which can be loaded onto an intelligent terminal, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 409, and / or installed from a removable medium 411. When the computer program is executed by a central processing unit (CPU) 401, various functions defined in the system of the present application are executed.

[0097] It should be noted that the computer-readable medium shown in the embodiment of the present application is used to implement the vision-based modified smart shopping cart loss prevention detection method; the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0098] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0100] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0101] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0102] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0103] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0104] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0105] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A vision-based anti-loss detection method for a modified smart shopping cart, characterized in that: include: Acquire video images during the shopping process, and use the acquired video images to determine whether items in the shopping cart have changed; When it is determined that the items in the shopping cart have changed, the image before the change and the image after the change in the video image are differentiated to obtain a differential image; Recognize the difference image and obtain the object information in the difference image; Compare the item information identified in the differential image with the item information in the shopping list to determine whether the identified item is an item in the shopping list, and issue a reminder if the identified item does not belong to the items in the shopping list; Methods for determining whether items in the shopping cart have changed include: The contour image of the moving target is obtained by using the difference between the current frame image and the previous frame image and the next frame image in the video image, and the area of ​​the target contour image is used to determine whether the items in the shopping cart have changed; The method for determining whether the items in the shopping cart have changed is to calculate the moving target by performing a three-frame difference method on the median filter of the image in the shopping cart, and to determine whether the goods in the shopping cart have changed according to the size of the outline of the moving object, including: The scanning camera on the shopping cart continuously takes pictures of the basket of the shopping cart at a rate of 5 pictures per second from the beginning of shopping to the completion of shopping, taking pictures f1, f2, f3...f t-1 、f t 、f t+1 、f t+2 ..., where f t The picture of the cart basket at the moment when the purchased goods are put into the cart basket, f1, f2, f3 ...f t-1 is the basket frame image before the purchased goods are put into the basket, f t+1 The following is a frame image of the purchased product after it is put into the shopping cart. The steps of using the frame difference method to calculate the frame difference contour are as follows: (1) Scale the image to speed up processing; (2) Perform Gaussian smoothing on the image; (3) Convert to grayscale value; (4) Binarization processing; (5) Collect video sequence images in a certain period of time, assuming represents the current frame at time t, and the two adjacent frames before and after it are respectively represented by and Indicates; a difference operation is performed between two adjacent frames at time t, and the judgment results after the difference operation are as follows: | (x,y,t)-f(x,y,t-1)|<T | (x,y,t)-f(x,y,t-1)|>=T in (x,y,t), (x, y, t-1) are the pixel values ​​of the corresponding pixels at time t and t-1 respectively, and T is the threshold; Then, the three-frame difference method is used to obtain the two-frame difference result D t , D t-1 and D t , D t+1 The result of determining the moving target edge D'n in the current frame f(x, y, t), wherein the method for obtaining D'n includes: D'n=|D t-1 -D t |∩|D t+1 -D t |; (6) Median filtering: Use median filtering to filter the moving target circle and process some isolated noise points in D'n; (7) Get the contour of D'n and define the threshold Y. If the contour area is greater than or equal to Y, it is considered that there is a change in the basket. If the contour area is less than Y, it is considered that there is no change in the basket.

2. According to claim 1, a vision-based anti-loss detection method for a modified smart shopping cart is characterized in that: The method also includes obtaining images in the shopping cart before shopping, and identifying the images in the shopping cart before shopping to obtain the shopper's personal belongings information.

3. According to claim 1, a vision-based anti-loss detection method for a modified smart shopping cart is characterized in that: Before using the acquired video image to determine whether the items in the shopping cart have changed, it is necessary to extract the ROI area of ​​each frame image in the video image, and use the image of the ROI area in each frame image to determine whether the items in the shopping cart have changed.

4. According to claim 3, a vision-based anti-loss detection method for a modified smart shopping cart is characterized in that: The ROI area of ​​each frame image is the area within the shopping box of the shopping cart.

5. According to claim 1, a vision-based anti-loss detection method for a modified smart shopping cart is characterized in that: Use the trained deep learning model to identify objects in the difference image.

6. A vision-based anti-loss detection method for a modified smart shopping cart according to claim 2, characterized in that: When the identified item does not belong to the items in the shopping list, the identified item is compared with the shopper's personal items. When the identified item belongs to the personal item, no reminder is given. When the identified item does not belong to the personal item, a reminder is given.

7. A vision-based modified intelligent shopping cart loss prevention detection device, characterized in that: include: An image acquisition device, arranged on the shopping cart, for acquiring image information in the shopping basket of the shopping cart; A judgment module, which uses the video images collected during the shopping process to judge whether the items in the shopping cart have changed; The difference module, when the judgment module judges that the items in the shopping cart have changed, performs a difference between the image before the change and the image after the change in the collected video image to obtain a difference image; The recognition module is used to recognize the item information in the differential image; the comparison module compares the item information recognized by the recognition module with the item information in the shopping list to determine whether the recognized item is in the shopping list; A reminder module, when the comparison module determines that the item identified by the recognition module is not in the shopping list, the reminder module issues a reminder; Methods for determining whether items in the shopping cart have changed include: The contour image of the moving target is obtained by using the difference between the current frame image and the previous frame image and the next frame image in the video image, and the area of ​​the target contour image is used to determine whether the items in the shopping cart have changed; The method for determining whether the items in the shopping cart have changed is to calculate the moving target by performing a three-frame difference method on the median filter of the image in the shopping cart, and to determine whether the goods in the shopping cart have changed according to the size of the outline of the moving object, including: The scanning camera on the shopping cart continuously takes pictures of the basket of the shopping cart at a rate of 5 pictures per second from the beginning of shopping to the completion of shopping, taking pictures f1, f2, f3...f t-1 、f t 、f t+1 、f t+2 ..., where f t The picture of the cart basket at the moment when the purchased goods are put into the cart basket, f1, f2, f3 ...f t-1 is the basket frame image before the purchased goods are put into the basket, f t+1 The following is a frame image of the purchased product after it is put into the shopping cart. The steps of using the frame difference method to calculate the frame difference contour are as follows: (1) Scale the image to speed up processing; (2) Perform Gaussian smoothing on the image; (3) Convert to grayscale value; (4) Binarization processing; (5) Collect video sequence images in a certain period of time, assuming represents the current frame at time t, and the two adjacent frames before and after it are respectively represented by and Indicates; a difference operation is performed between two adjacent frames at time t, and the judgment results after the difference operation are as follows: | (x,y,t)-f(x,y,t-1)|<T | (x,y,t)-f(x,y,t-1)|>=T in (x,y,t), (x, y, t-1) are the pixel values ​​of the corresponding pixels at time t and t-1 respectively, and T is the threshold; Then, the three-frame difference method is used to obtain the two-frame difference result D t , D t-1 and D t , D t+1 The result of determining the moving target edge D'n in the current frame f(x, y, t), wherein the method for obtaining D'n includes: D'n=|D t-1 -D t |∩|D t+1 -D t |; (6) Median filtering: Use median filtering to filter the moving target circle and process some isolated noise points in D'n; (7) Get the contour of D'n and define the threshold Y. If the contour area is greater than or equal to Y, it is considered that there is a change in the basket. If the contour area is less than Y, it is considered that there is no change in the basket.