Abnormal purchase identification and intelligent purchase return method, device and storage medium

Abstract

The present invention discloses a method, device and storage medium for identifying abnormal add-on purchases and intelligently returning purchases, and belongs to the technical field of intelligent shopping carts. The method comprises: inputting a collected shopping process video frame by frame into a pre-trained ten-category target detection network, and outputting detection frame coordinates and category information of all commodities in each frame image; inputting the detection frame coordinates and category information into a target tracking algorithm, and generating trajectory information of each shopping commodity; performing a trajectory image filtering operation on the trajectory information, and outputting a filtered single-frame trajectory image; inputting the single-frame trajectory image into a pre-trained multi-attribute classification network, and outputting classification features of the corresponding commodity; comparing the classification features of the commodity with classification features in a pre-built multi-attribute standard library, and determining that abnormal add-on purchase behavior exists if any feature is inconsistent; the present invention realizes the process of identifying abnormal add-on purchases and intelligently returning purchases, effectively improving the settlement rate, reducing supermarket losses, and enhancing the user shopping experience.

Description

Technical Field

[0001] The present invention relates to a method, device and storage medium for identifying abnormal added purchases and intelligent purchase returns, belonging to the technical field of intelligent shopping carts. Background Art

[0002] Traditional shopping carts primarily serve as a temporary storage system for items, while checkout still requires a manual checkout counter. High customer traffic often leads to queues, resulting in lengthy shopping processes and a poor shopping experience. With consumers' increasing demand for convenience and a better shopping experience, and offline retailers' in-depth exploration of digital transformation, smart shopping carts have emerged as a new type of shopping tool. They combine advanced technologies such as the Internet of Things, artificial intelligence, and mobile payments to provide consumers with a more convenient and intelligent shopping experience.

[0003] Currently, artificial intelligence algorithms are being used increasingly widely in the shopping cart field. Using artificial intelligence algorithms, intelligent shopping is achieved, shortening user shopping time and enhancing the user experience. While smart shopping carts aim to enhance the user shopping experience, supermarkets also have higher requirements for loss prevention. Therefore, smart shopping carts must balance user experience and supermarket loss prevention needs. Examples include a smart shopping cart product return identification method and system with publication number CN117952507B, a smart shopping cart shopping behavior analysis method and system based on product trajectory analysis with publication number CN118097519B, and a supermarket intelligent theft prevention system and method with publication number CN106981150A.

[0004] The most common method for intelligent purchase and return verification relies primarily on weight recognition. The electronic scale at the bottom of the shopping cart identifies the weight, and the user's purchase decision is based on weight increases and decreases. The product name is then retrieved by matching the weight against a standard weight library. This symmetry requires higher precision and sensitivity, and external factors can also affect the weight change of the scale, such as manually pressing the frame when adding or returning an item, or the shaking of the shopping cart when pushing it. These external factors often prevent users from completing normal purchases and checking out. Furthermore, supermarkets often have many products of the same weight, and relying on weight matching cannot cover all supermarket items. Consequently, a large number of products require manual verification, which is very unfriendly to the user shopping experience. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a method, device and storage medium for abnormal purchase identification and intelligent purchase return, so as to realize the process of abnormal purchase identification and intelligent purchase return, effectively improve the settlement rate, reduce supermarket losses and enhance the user shopping experience.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides a method for identifying abnormal add-on purchases and intelligently canceling purchases, comprising:

[0008] The collected shopping process video is input into the pre-trained ten-category object detection network frame by frame, and the detection box coordinates and category information of all products in each frame are output;

[0009] Input the detection box coordinates and category information into the target tracking algorithm to generate the trajectory information of each shopping item;

[0010] Perform trajectory image filtering on the trajectory information and output the filtered single-frame trajectory image;

[0011] Input a single-frame trajectory image into a pre-trained multi-attribute classification network and output the classification features of the corresponding product;

[0012] Compare the product's classification features with those in the pre-built multi-attribute standard library. If any feature is inconsistent, it is determined that there is abnormal add-to-cart behavior.

[0013] When the user takes out the product, the classification features of the taken out product are compared with the classification features in the pre-built multi-attribute temporary library to obtain the current returned product barcode and realize intelligent return.

[0014] Furthermore, the multi-attribute temporary library stores the barcode / add-on-purchase serial number and classification characteristics of the added-on products; when the user takes out the products, the classification characteristics of the taken-out products are matched one by one with the classification characteristics in the multi-attribute temporary library; if there are records in which all matches are successful, the barcode / add-on-purchase serial number corresponding to the record is used as the identification of the taken-out product.

[0015] Furthermore, the training method of the ten-category target detection network includes:

[0016] Collect supermarket shopping scene pictures, annotate the hands and nine categories of goods in the pictures, and obtain the annotated dataset;

[0017] The labeled data set is input into the target detection network training to obtain the trained ten-class target detection network; the target detection network is an RCNN two-stage target detection network or a YOLO single-stage target detection network.

[0018] Furthermore, the training method of the multi-attribute classification network includes:

[0019] Perform multi-attribute classification and labeling on supermarket products to obtain labeled multi-attribute classification datasets;

[0020] The labeled multi-attribute classification data set is sent to the multi-attribute classification network for training to obtain a multi-attribute classification network based on supermarket products. The multi-attribute classification network is selected from any one of VGG, ResNet or DenseNet.

[0021] Furthermore, performing a trajectory image filtering operation on the trajectory information includes:

[0022] Calculate the distance between the center points of adjacent frames of the trajectory. The calculation formula is as follows:

[0023] ;

[0024] in Indicates the jth frame in the trajectory information and Frame, tracking id is The distance from the center point coordinates; , Indicates that the tracking id in the trajectory information is When, The coordinates of the center point of the frame image; , Indicates that the tracking id in the trajectory information is When, The coordinates of the center point of the frame image;

[0025] Based on minimum distance threshold and maximum distance threshold Delete still or abnormal motion frames, and set the conditions as follows:

[0026] Condition 1: ;

[0027] Condition 2: ;

[0028] When one of condition 1 and condition 2 is met, the The trajectory information of the frame image is retained. Frame image trajectory information, otherwise 、 The trajectory information of the frame image is retained;

[0029] Sort the image frame IDs in the remaining trajectory information, and select the image with the middle frame ID as the final retained shopping product trajectory image.

[0030] Furthermore, the method for constructing the multi-attribute standard library includes:

[0031] Stores the mapping relationship between product barcodes and classification features. Classification features include: color classification, shape classification, bulk identification, packaging material classification, and combination identification.

[0032] Furthermore, the method for constructing the multi-attribute temporary library includes:

[0033] When scanning a code to add to a purchase, the barcode and corresponding classification features are stored;

[0034] If the code is not scanned to add to the purchase, multiple random purchase serial numbers and corresponding classification features will be generated.

[0035] In a second aspect, the present invention provides a device for identifying and intelligently cancelling abnormal add-on purchases, which adopts any of the aforementioned methods for identifying and intelligently cancelling abnormal add-on purchases, including:

[0036] The first processing module is used to input the collected shopping process video frame by frame into a pre-trained ten-category object detection network, and output the detection box coordinates and category information of all products in each frame image;

[0037] The second processing module is used to input the detection box coordinates and category information into the target tracking algorithm to generate the trajectory information of each shopping item;

[0038] The image filtering module is used to perform a trajectory image filtering operation on the trajectory information and output a filtered single-frame trajectory image;

[0039] The third processing module is used to input the single-frame trajectory image into a pre-trained multi-attribute classification network and output the classification features of the corresponding product;

[0040] The add-to-cart identification module compares the product's classification features with those in a pre-built multi-attribute standard library. If any feature is inconsistent, it is determined that an abnormal add-to-cart behavior has occurred.

[0041] The intelligent return module is used to compare the classification features of the taken-out goods with the classification features in the pre-built multi-attribute temporary library when the user takes out the goods, obtain the barcode of the current returned goods, and realize intelligent return.

[0042] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the aforementioned methods when executed by a processor.

[0043] In a fourth aspect, the present invention provides a computer device, comprising:

[0044] Memory, used to store computer programs / instructions;

[0045] A processor is configured to execute the computer program / instructions to implement the steps of any of the aforementioned methods.

[0046] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the aforementioned methods when executed by a processor.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention provides a method, device and storage medium for identifying abnormal purchases and intelligent returns, which can identify abnormal purchases; when it is detected that the user has made an abnormal purchase, the software will promptly remind the user, and the user can promptly correct the abnormal behavior. This prevents abnormal behaviors from accumulating until settlement and then tracing back. The accumulation of multiple abnormal behaviors often makes it impossible for users to correct them in sequence, and manual intervention is required for normal settlement, which prolongs the shopping time and affects the user's shopping experience. The present invention can realize intelligent returns, and users do not need to scan codes or operate the software. They only need to take out the products they want to return to achieve accurate returns. The method of the present invention is not affected by external pressure or promotion, greatly shortens the return time, and effectively improves the user's shopping experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a method for identifying abnormal added purchases and intelligently canceling purchases provided by an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of a multi-attribute classification network structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0052] Example 1: This example introduces a method for identifying abnormal add-on purchases and intelligently canceling purchases, including:

[0053] The collected shopping process video is input into the pre-trained ten-category object detection network frame by frame, and the detection box coordinates and category information of all products in each frame are output;

[0054] Input the detection box coordinates and category information into the target tracking algorithm to generate the trajectory information of each shopping item;

[0055] Perform trajectory image filtering on the trajectory information and output the filtered single-frame trajectory image;

[0056] Input a single-frame trajectory image into a pre-trained multi-attribute classification network and output the classification features of the corresponding product;

[0057] Compare the product's classification features with those in the pre-built multi-attribute standard library. If any feature is inconsistent, it is determined that there is abnormal add-to-cart behavior.

[0058] When the user takes out the product, the classification features of the taken out product are compared with the classification features in the pre-built multi-attribute temporary library to obtain the current returned product barcode and realize intelligent return.

[0059] The application process of the abnormal purchase identification and intelligent purchase cancellation method provided in this embodiment specifically involves the following steps:

[0060] Step 1: Train a ten-class target detection network;

[0061] The ten categories are hands and nine categories of supermarket goods, where supermarket goods are classified according to category, color, and shape, for a total of ten categories; the category name cls1 represents hands, and cls2, cls3, cls4, cls5, cls6, cls7, cls8, cls9, and cls10 represent supermarket goods;

[0062] The object detection network collects and annotates supermarket shopping scene images, then trains the annotated dataset to generate an object detection network based on ten categories of supermarket merchandise. Object detection network models include, but are not limited to, two-stage object detection networks (RCNN series) and single-stage object detection networks (YOLO series).

[0063] Step 2: Train the supermarket product multi-attribute classification network;

[0064] The multi-attribute of supermarket goods is to divide supermarket goods into multi-attribute categories. The multi-attribute categories are: color, shape, whether it is bulk, packaging material, and whether it is a combination, which is a total of five multi-attribute categories. Among them, the colors are divided into: red, white, black, yellow, brown, green, purple, and blue; the shapes are divided into: bottled cylinders, canned cylinders, uniform cylinders, uniform cuboids, round cakes, ellipses, spheres, and irregular shapes; whether it is bulk is divided into: bulk and non-bulk; the packaging materials are divided into: glass, plastic, paper, and metal; whether it is a combination is divided into: combination and non-combination. The multi-attribute categories include but are not limited to color, shape, whether it is bulk, packaging material, and whether it is a combination. The classification rules under each category are only used as preferred examples, including but not limited to the classification names and rules described above;

[0065] The multi-attribute classification network is used to label supermarket products using multi-attribute classification. The labeled multi-attribute classification dataset is fed into the multi-attribute classification network for training, resulting in a multi-attribute classification network based on supermarket products. The backbone of the multi-attribute classification network includes but is not limited to VGG (Visual Geometry Group Convolutional Network), ResNet (Residual Convolutional Network), and DenseNet (Densely Connected Convolutional Network).

[0066] Step 3: During the shopping process, use the camera on the shopping cart to capture the shopping process video;

[0067] The shopping cart is a shopping cart containing an RGB camera;

[0068] The shopping process video is a video of the operation process of the product in the shopping cart captured by an RGB camera.

[0069] Step 4: Perform ten-category object detection on the shopping video to obtain product object information;

[0070] The product target information is to feed the shopping video into the ten-category target detection network in the picture frame sequence, and the network outputs the detection results of all products in the picture frame sequence. The detection results are the detection box coordinates and categories .in is the coordinate of the center point of the detection frame, The width and height of the detection box.

[0071] Step 5: Send the product target information to the target tracking algorithm and trajectory filtering algorithm to obtain the trajectory information of the shopping product;

[0072] The trajectory information is the coordinates and categories of shopping items in the picture sequence, such as Tracking in frame images for The trajectory information is , ,in , indicating tracking for When, The center coordinates of the shopping item detection frame in the frame image; The width and height of the shopping product detection box, Detection box category for shopping items.

[0073] Step 6: Send the shopping item trajectory information to the trajectory image screening algorithm to filter out a frame of trajectory image;

[0074] The calculation steps of the trajectory image screening algorithm are as follows:

[0075] 1) Calculate the center point distance between adjacent frames in the trajectory information. The calculation formula is as follows:

[0076] ;

[0077] in Indicates the jth frame in the trajectory information and Frame, tracking id is The distance from the center point coordinates; , Indicates that the tracking id in the trajectory information is When, The coordinates of the center point of the frame image; , Indicates that the tracking id in the trajectory information is When, The coordinates of the center point of the frame image.

[0078] 2) Delete images with abnormal changes between static and moving images in the trajectory information.

[0079] Set two distance thresholds, the minimum distance threshold and maximum distance threshold , set the conditions as follows:

[0080] Condition 1: ;

[0081] Condition 2: ;

[0082] When one of condition 1 and condition 2 is met, the The trajectory information of the frame image is retained. Frame image trajectory information, otherwise 、 The trajectory information of the frame images is retained.

[0083] 3) Sort the image frame IDs in the remaining trajectory information and select the image with the middle frame ID as the final retained shopping product trajectory image. Assume that after the above screening, there are still m frames of images in the trajectory information, and the sorting is , frame_id subscript Indicates the specific frame number of the image in the trajectory information. The index of the filtered trajectory information image is ;

[0084] The retained trajectory image index is calculated as follows:

[0085] ;

[0086] in Indicates the index of the retained trajectory information image in the trajectory information. Indicates the length of the trajectory information after filtering, and the symbol / / indicates rounding the result.

[0087] Step 7: Send the retained trajectory image to the multi-attribute classification network and output the multi-attribute classification features of the shopping product image;

[0088] The multi-attribute classification features are color, shape, whether it is loose, packaging material, and whether it is a combination. Each category is further divided into multiple categories. The specific classification rules are described in step 2. The output of the multi-attribute classification network is in the following order: color, shape, whether it is loose, packaging material, and whether it is a combination. For example, if an image of a blue can of cola is input, the multi-attribute classification output is: blue, uniform cylinder, non-loose, metal, and non-combination.

[0089] Step 8: Build a multi-attribute standard library to identify abnormal add-on purchases;

[0090] The multi-attribute standard library is used to label supermarket products with multiple attributes. Each product barcode and multi-attribute classification features are stored together to build a multi-attribute standard feature library. The storage format of the multi-attribute standard library is shown in Table 1:

[0091] Table 1 Multi-attribute standard library storage table

[0092] barcode color shape Is it a random name? Packaging material Is it a combination 6922507827888 yellow Uniform cuboid Non-scattered plastic Non-combination 6924743915848 green Uniform cylinder Non-scattered Paper Non-combination

[0093] The abnormal add-on purchase identification means that when a user scans the code of product A and puts in product B, the multi-attribute features of product B can be obtained through steps 3 to 7, and the multi-attribute features of B are compared with the multi-attribute features of A in the multi-attribute standard library; assuming that the multi-attribute features of product A are [ ], the multi-attribute characteristics of product B are [ ], the judgment conditions are as follows:

[0094] Condition 1: ;

[0095] Condition 2: ;

[0096] Condition 3: ;

[0097] Condition 4: ;

[0098] Condition 5: ;

[0099] Where color represents color, shape represents shape, scatter indicates whether it is a scattered item, material represents the packaging material, and combine indicates whether it is a combined item. If any of the above conditions is not met, it indicates that the multi-attribute feature comparison is inconsistent, A and B are not the same item, and there is abnormal add-to-cart behavior.

[0100] Step 9: Build a temporary multi-attribute database to implement smart purchase returns;

[0101] The multi-attribute temporary library indicates that when a user adds to a purchase, the multi-attribute characteristics of the current added product are obtained through steps 3 to 7; if the user scans the code to add to a purchase, the barcode and the multi-attribute characteristics are saved to the multi-attribute temporary library; if the user does not scan the code to add to a purchase, a 13-digit random number is generated as the added purchase serial number, and the added purchase serial number and the multi-attribute characteristics are saved to the multi-attribute temporary library.

[0102] The storage format of the multi-attribute temporary library is shown in Table 2:

[0103] Table 2 Multi-attribute temporary library storage table

[0104] Barcode / order number color shape Is it a random name? Packaging material Is it a combination 6922507827888 yellow Uniform cuboid Non-scattered plastic Non-combination 1827309728318 green Uniform cylinder Non-scattered Paper Non-combination

[0105] The smart return process is when a user takes out an item. Steps 3 to 7 are used to obtain the multi-attribute characteristics of the currently taken out item, and the multi-attribute characteristics are compared with the multi-attribute temporary library. When all five multi-attribute characteristics match, the barcode or add-to-purchase serial number in the multi-attribute temporary library is the corresponding taken out item.

[0106] The comparison method of the extracted product with the product in the multi-attribute temporary library is as follows: If the multi-attribute feature corresponding to the extracted product A is [ ], the features in the multi-attribute temporary library are [[ [ ]],in Indicates the barcode or purchase order number of the kth data item in the multi-attribute temporary database, where k is the length of the multi-attribute temporary database. The setting conditions are as follows:

[0107] Condition 1: ;

[0108] Condition 2: ;

[0109] Condition 3: ;

[0110] Condition 4: ;

[0111] Condition 5: ;

[0112] in is the index of the multi-attribute temporary library, ; When all the above conditions are met, the barcode or purchase order number of the product is taken out from the multi-attribute temporary library .

[0113] Based on step 8 above, the method of the present invention can identify abnormal add-ons. When an abnormal add-on is detected, the software promptly alerts the user, allowing them to promptly correct the abnormal behavior. This prevents abnormal behavior from accumulating and then being tracked back at checkout. The accumulation of multiple abnormal behaviors often results in users being unable to correct them sequentially, requiring manual intervention to complete checkout, which prolongs shopping time and impacts the user's shopping experience.

[0114] Based on step 9 above, the method of the present invention enables intelligent returns. Users do not need to scan a code or perform any software operations; they simply need to remove the item they wish to return to achieve a precise return. This method of the present invention is unaffected by external pressure or promotion, significantly shortens the return process, and effectively enhances the user's shopping experience.

[0115] The ten-category target detection network in step 1 includes but is not limited to ten categories, and other categories can also be replaced; the target detection network includes but is not limited to the RCNN series, Yolo series or other models can also be replaced;

[0116] The multi-attribute classification network in step 2; the multi-attribute classification rules include but are not limited to color, shape, whether it is bulk, packaging material, and whether it is a combination. Each category and its subcategories can be replaced with other classification rules and subcategories. The backbone network of the multi-attribute classification network includes but is not limited to VGG, ResNet, and DenseNet, and other networks can also be replaced;

[0117] Target tracking and trajectory filtering algorithms in step 5; target tracking algorithms include but are not limited to DeepSORT and ByteTrack, and other tracking algorithms can also be substituted; trajectory filtering algorithms include but are not limited to using the maximum sum of the distances between the center points of adjacent frame detection boxes, calculating the maximum trajectory length, the relative maximum distance of the trajectory, or other algorithms can also be substituted;

[0118] Step 6: The trajectory image filtering algorithm includes but is not limited to using the trajectory center point distance to filter. Other filtering algorithms can also be implemented.

[0119] In steps 8 and 9, the barcode in the construction of the multi-attribute standard library and the multi-attribute temporary library can also be replaced with the product name or other unique identifier of the product; the storage format is consistent with the multi-attribute classification rules, including but not limited to the existing storage format, and replacing it with other formats can also achieve the same function.

[0120] The following describes the contents involved in the above embodiment in conjunction with a preferred embodiment.

[0121] (1) Training a 10-category object detection network. The real-world shopping scene data is labeled in 10 categories. CLS1 represents a hand, and CLS2, CLS3, CLS4, CLS5, CLS6, CLS7, CLS8, CLS9, and CLS10 represent supermarket items. The labeled detection dataset is fed into the object detection network YOLOv12 for training, resulting in a 10-category YOLOv12 model.

[0122] (2) Train a multi-attribute classification network. Perform multi-attribute classification and labeling on product images in shopping scenarios, using the main color of the product as the color label. The labeling examples are shown in Table 3:

[0123] Table 3 Multi-attribute classification labeling table

[0124]

[0125] The id is the name of the picture, and each picture is labeled with multiple attributes.

[0126] Modify the standard resnet18 (18-layer residual network) network structure and change the number of network outputs to be consistent with the number of multi-attribute classifications. The modified resnet18 network structure is as follows Figure 2 The labeled multi-attribute dataset is fed into the modified resnet18 network for multi-attribute network training to obtain the multi-attribute classification model multi_resnet18.

[0127] (3) Obtaining a shopping video. The video is captured by an RGB camera on the shopping cart. To implement the method of the present invention, the shopping cart must be equipped with at least one RGB camera for acquiring the shopping video. The shopping video can be a video captured in real time by the camera, or a video of the shopping process captured by gravity.

[0128] (4) The shopping video is fed into the ten-class target detection network to obtain the detection information of the goods in the video frame sequence. The above shopping video is fed into the ten-class yolov12 model according to the picture frame sequence to obtain the product detection information of each frame. For example, the detection information of the jth detection box in the i-th frame is ,in Represents the detection information of the jth detection box of the i-th frame image, Indicates the coordinates of the center point of the jth detection box in the i-th frame. Indicates the width and height of the jth detection box of the i-th frame image, Indicates the category of the jth detection box in the i-th frame.

[0129] When i=12 and j=1, the video frame size is 1280x1024, and the corresponding detection frame is the hand detection frame. The detection frame information .

[0130] (5) All product detection information is fed into the target tracking and trajectory filtering algorithm to filter out the trajectory information of shopping products.

[0131] The target tracking algorithm uses ByteTrack (ByteDance target tracking algorithm), and the tracking algorithm outputs the Tracking in frame images for The trajectory information is , The tracking algorithm results are input into the trajectory filtering algorithm. The trajectory filtering algorithm accumulates the Euclidean distance of the center points of the detection frames of adjacent frames. The calculation formula is as follows:

[0132] ;

[0133] in The sum of the Euclidean distances between the center points of the adjacent frame detection boxes of each track is calculated. The track with the largest distance, excluding the hand, is selected as the track information of the shopping item.

[0134] The cumulative distances of the five product trajectories other than the hand are shown in Table 4:

[0135] Table 4 Statistics of trajectory center point distance indicators

[0136] id dist_center id_1 67 id_2 281 id_3 46 id_5 29 id_6 34

[0137] In Table 4, the cumulative distance between the center points of the adjacent frame detection frames of id_2 is the largest, and id_2 is determined to be the trajectory information of the shopping item.

[0138] (6) The trajectory information of the shopping item is fed into the trajectory image filtering algorithm to filter out a frame of trajectory information. Step (5) has calculated the distance between the centers of the adjacent frames of the shopping item trajectory information and set the maximum threshold of the center point distance. and minimum threshold .

[0139] Condition 1: ;

[0140] Condition 2: ;

[0141] According to conditions 1 and 2, redundant static detection frames and false detection frames in the shopping cart are deleted.

[0142] Finally, the remaining trajectory information frames are sorted by frame number frame_id. The index of the sorted trajectory information pictures is The retained trajectory image index is calculated as follows:

[0143] ;

[0144] The shopping item trajectory information id_2 has a total of 15 frames of images. The distances between the center points of adjacent frames of the 15 frames are shown in Table 5:

[0145] Table 5 Statistics of adjacent frame distances

[0146] frame_id dist 2 20 3 23 4 25 5 32 6 56 7 5 8 48 9 22 10 15 11 8 12 5 13 2 14 2 15 3

[0147] set up =10, =50, according to the above conditions 1 and 2, the frame numbers of the retained images are 1, 2, 3, 4, 5, 8, 9, 10, the trajectory information length is 8, and the index calculation formula is used to calculate 4, the frame number in the corresponding trajectory information is 5, and the 5th frame image in the trajectory information is finally retained.

[0148] (7) The retained trajectory image is fed into the multi-attribute classification network to output the multi-attribute classification features. The fifth frame trajectory image in step (6) is fed into the multi_resnet18 network, and the output multi-attribute features are [green, irregular, scattered, plastic, non-combination].

[0149] (8) Construct a multi-attribute standard library to identify abnormal add-on purchases. Assume that there are five products: cucumber-flavored boxed potato chips, barcode: 6924743915848; tomato-flavored boxed potato chips, barcode: 6924743915817; dishwashing liquid, barcode: 6920174783490; beer, barcode: 6949352201403; spicy strips, barcode: 6935284412918. The storage format of the standard library for these five products is shown in Table 6:

[0150] Table 6 Multi-attribute standard library storage instance table

[0151] barcode color shape Is it a random name? Packaging material Is it a combination 6924743915848 green Uniform cylinder Non-scattered Paper Non-combination 6924743915817 Purple Uniform cylinder Non-scattered Paper Non-combination 6920174783490 yellow Bottled Cylinder Non-scattered plastic Non-combination 6949352201403 green Uniform cylinder Non-scattered Metal Non-combination 6935284412918 White Uniform cuboid Non-scattered plastic Non-combination

[0152] If the product the user scans is beer and the product they put in is cucumber-flavored potato chips, the multi-attribute features of cucumber-flavored potato chips obtained from steps (3) to (7) are [green, uniform cylinder, non-bulk, paper, non-combined]. When compared with the multi-attribute features of beer in the standard library, the packaging material does not match, indicating that the user has made an abnormal purchase. The software will promptly remind the user to adjust this purchase behavior to avoid affecting the final settlement.

[0153] (9) Construct a multi-attribute temporary database to realize intelligent return purchase. Assume that there are 5 kinds of products in step (8). Except for the cucumber-flavored boxed potato chips, which are not scanned for purchase, all other products are scanned for purchase. The storage format of the constructed temporary database is shown in Table 7:

[0154] Table 7 Multi-attribute temporary library storage instance table

[0155] Barcode / order number color shape Is it a random name? Packaging material Is it a combination 9289173892018 green Uniform cylinder Non-scattered Paper Non-combination 6924743915817 Purple Uniform cylinder Non-scattered Paper Non-combination 6920174783490 yellow Bottled Cylinder Non-scattered plastic Non-combination 6949352201403 green Uniform cylinder Non-scattered Metal Non-combination 6935284412918 White Uniform cuboid Non-scattered plastic Non-combination

[0156] When the user takes out the dishwashing liquid, according to steps (3) to (7), the multi-attribute characteristics of the product taken out are [yellow, bottled cylindrical, non-bulk, plastic, non-combined]. Comparing the multi-attribute characteristics with the multi-attribute standard library, the barcode of the product taken out is 6920174783490, which corresponds to the product dishwashing liquid. This allows the user to complete the return function by simply taking out the product to be returned without any other operations.

[0157] Example 2: This example provides a method for identifying abnormal added purchases and intelligently canceling purchases using any one of the methods described in Example 1, including:

[0158] The first processing module is used to input the collected shopping process video frame by frame into a pre-trained ten-category object detection network, and output the detection box coordinates and category information of all products in each frame image;

[0159] The second processing module is used to input the detection box coordinates and category information into the target tracking algorithm to generate the trajectory information of each shopping item;

[0160] The image filtering module is used to perform a trajectory image filtering operation on the trajectory information and output a filtered single-frame trajectory image;

[0161] The third processing module is used to input the single-frame trajectory image into a pre-trained multi-attribute classification network and output the classification features of the corresponding product;

[0162] The add-to-cart identification module compares the product's classification features with those in a pre-built multi-attribute standard library. If any feature is inconsistent, it is determined that an abnormal add-to-cart behavior has occurred.

[0163] The intelligent return module is used to compare the classification features of the taken-out goods with the classification features in the pre-built multi-attribute temporary library when the user takes out the goods, obtain the barcode of the current returned goods, and realize intelligent return.

[0164] The specific functional implementation of each of the above modules can be found in the relevant content of the method in Example 1 and will not be elaborated on here.

[0165] Example 3: This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods described in Example 1 are implemented.

[0166] Embodiment 4: This embodiment provides a computer device, including:

[0167] Memory, used to store computer programs / instructions;

[0168] A processor, configured to execute the computer program / instructions to implement the steps of any one of the methods described in Example 1.

[0169] Example 5: This embodiment provides a computer program product, including a computer program / instruction, which implements the steps of any method described in Example 1 when executed by a processor.

[0170] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

[0171] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0172] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure and are not intended to limit its scope of protection. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the disclosed claims to be approved.

Claims

1. A method for identifying abnormal purchases and intelligently returning purchases, characterized in that: include: The collected shopping process video is input into a pre-trained ten-category object detection network frame by frame, and the detection frame coordinates and category information of all products in each frame image are output; the ten categories are hands and the nine categories are supermarket products, where supermarket products are classified according to category, color, and shape; Input the detection box coordinates and category information into the target tracking algorithm to generate the trajectory information of each shopping item; Perform a trajectory image filtering operation on the trajectory information and output a filtered single-frame trajectory image; this includes: calculating the distance between the center points of adjacent trajectory frames, deleting static or abnormal motion frames according to set conditions based on the minimum and maximum distance thresholds; sorting the image frame IDs in the remaining trajectory information, and selecting the image with the middle frame ID as the final retained shopping item trajectory image; Input a single-frame trajectory image into a pre-trained multi-attribute classification network and output the classification features of the corresponding product; Compare the product's classification features with those in the pre-built multi-attribute standard library. If any feature is inconsistent, it is determined that there is abnormal add-to-cart behavior. When the user takes out a product, the classification features of the product are compared with the classification features in the pre-built multi-attribute temporary library to obtain the current returned product barcode / add-on purchase serial number, realizing intelligent return purchase; The multi-attribute temporary library stores the mapping relationship between the barcode / add-to-purchase serial number and the classification characteristics of the added-to-purchase items; if the user scans the code to add to the purchase, the barcode and the multi-attribute characteristics are saved in the multi-attribute temporary library; if the user does not scan the code to add to the purchase, a 13-digit random number is generated as the add-to-purchase serial number, and the add-to-purchase serial number and the multi-attribute characteristics are saved in the multi-attribute temporary library; when the user takes out the item, the classification characteristics of the taken-out item are matched one by one with the classification characteristics in the multi-attribute temporary library; if there is a record in which all matches are successful, the barcode / add-to-purchase serial number corresponding to the record is used as the identification of the taken-out item; the classification characteristics include: color classification, shape classification, bulk identification, packaging material classification and combination identification.

2. The abnormal purchase identification and intelligent purchase cancellation method according to claim 1 is characterized in that: The training method of the ten-category target detection network includes: Collect supermarket shopping scene pictures, annotate the hands and nine categories of goods in the pictures, and obtain the annotated dataset; The labeled data set is input into the target detection network training to obtain the trained ten-class target detection network; the target detection network is an RCNN two-stage target detection network or a YOLO single-stage target detection network.

3. The abnormal purchase identification and intelligent purchase cancellation method according to claim 1 is characterized in that: The training method of the multi-attribute classification network includes: Perform multi-attribute classification and labeling on supermarket products to obtain labeled multi-attribute classification datasets; The labeled multi-attribute classification data set is sent to the multi-attribute classification network for training to obtain a multi-attribute classification network based on supermarket products. The multi-attribute classification network is selected from any one of VGG, ResNet or DenseNet.

4. The abnormal purchase identification and intelligent purchase cancellation method according to claim 1 is characterized in that: The performing of the trajectory image filtering operation on the trajectory information includes: Calculate the distance between the center points of adjacent frames of the trajectory. The calculation formula is as follows: ； in Indicates the distance between the jth frame and the j+1th frame in the trajectory information, with the tracking id being i as the center point coordinate; , Indicates that the tracking id in the trajectory information is When, The coordinates of the center point of the frame image; , Indicates that the tracking id in the trajectory information is When, The coordinates of the center point of the frame image; Based on minimum distance threshold and the maximum distance threshold Delete still or abnormal motion frames, and set the conditions as follows: Condition 1: ; Condition 2: ; If one of the conditions 1 and 2 is met, the The trajectory information of the frame image is retained. Frame image trajectory information, otherwise Frame, The trajectory information of the frame image is retained; Sort the image frame IDs in the remaining trajectory information, and select the image with the middle frame ID as the final retained shopping product trajectory image.

5. The abnormal purchase identification and intelligent purchase cancellation method according to claim 1 is characterized in that: The method for constructing the multi-attribute temporary library includes: When scanning a code to add to a purchase, the barcode and corresponding classification features are stored; If the code is not scanned to add to the purchase, multiple random purchase serial numbers and corresponding classification features will be generated.

6. A device for identifying and intelligently cancelling abnormal purchases, using the method for identifying and intelligently cancelling abnormal purchases according to any one of claims 1 to 5, characterized in that: include: The first processing module is used to input the collected shopping process video frame by frame into a pre-trained ten-category object detection network, and output the detection box coordinates and category information of all products in each frame image; The second processing module is used to input the detection box coordinates and category information into the target tracking algorithm to generate the trajectory information of each shopping item; The image filtering module is used to perform a trajectory image filtering operation on the trajectory information and output a filtered single-frame trajectory image; The third processing module is used to input the single-frame trajectory image into a pre-trained multi-attribute classification network and output the classification features of the corresponding product; The add-to-cart identification module compares the product's classification features with those in a pre-built multi-attribute standard library. If any feature is inconsistent, it is determined that an abnormal add-to-cart behavior has occurred. The intelligent return module is used to compare the classification features of the taken-out goods with the classification features in the pre-built multi-attribute temporary library when the user takes out the goods, obtain the barcode of the current returned goods, and realize intelligent return.

7. An electronic device, characterized in that: include: Memory, used to store computer programs / instructions; A processor configured to execute the computer program / instructions to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images