Method, apparatus, device and medium for detecting shopping settlement fraud risk behavior

Abstract

This application relates to methods, apparatus, equipment, and media for detecting fraudulent shopping checkout behavior. The method includes acquiring multiple consecutive video frames, detecting the human body bounding box and pose key points of a person in the video frames to obtain a pose detection result; determining the upper body bounding box based on the human body bounding box, and tracking the upper body bounding box in multiple consecutive video frames to obtain a motion trajectory; determining whether the person has triggered a shopping behavior based on the pose detection result and motion trajectory through trajectory analysis and pose analysis; if the person has triggered a shopping behavior, outputting and saving a detection record for the shopping behavior trigger event to obtain a shopping behavior detection result; and, if the shopping behavior detection result indicates the existence of a shopping behavior trigger event, determining a wrist region image based on wrist key points, performing product recognition on the wrist region image through open set object detection, outputting and saving a product recognition event record to obtain a shopping checkout detection result.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for detecting fraudulent activities during online checkout. Background Technology

[0002] With the rapid development of artificial intelligence technology, computer vision technology is increasingly being applied in retail scenarios, covering areas such as automatic product recognition, self-service checkout, human behavior analysis, and security monitoring. However, in terms of shopping behavior detection, traditional methods often rely on single image features, making it difficult to accurately distinguish between shopping and non-shopping behaviors in complex retail scenarios, resulting in a high false detection rate. Secondly, in product recognition, widely adopted object detection models based on the closed-set assumption heavily rely on large-scale, high-quality labeled data. For dynamically changing retail products, data collection and training costs are high, making it difficult to quickly adapt to new products. This leads to low detection rates and high costs when detecting fraudulent activities at checkout. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, device, and medium for detecting fraudulent activities during shopping checkout, aiming to improve the accuracy of detection and reduce detection costs.

[0004] To achieve the above objectives, a first aspect of this disclosure provides a method for detecting fraudulent activities during shopping checkout, the method comprising:

[0005] Acquire multiple consecutive video frames of the cashier within a preset range, and for the video frames containing human figures, detect the human body bounding box and pose key points of the human figures in the video frames to obtain pose detection results, wherein the pose key points include wrist key points.

[0006] Based on the human body bounding box, the upper body bounding box of the human object is determined, and the upper body bounding box is tracked in the consecutive video frames to obtain the motion trajectory of the same human object.

[0007] Based on the posture detection results and the motion trajectory, through trajectory analysis and posture analysis, it is determined whether each of the human objects has triggered a shopping behavior. If any human object triggers the shopping behavior, the detection record corresponding to the shopping behavior trigger event is output and saved for the shopping behavior trigger event of the human object, and the shopping behavior detection result is obtained.

[0008] If the shopping behavior detection result indicates the presence of a shopping behavior triggering event, the wrist region image is determined based on the wrist key points corresponding to the video frame. Then, the product is identified in the wrist region image through open set object detection. The product identification event record is output and saved to obtain the shopping checkout detection result.

[0009] In one possible implementation, the step of performing product recognition on the wrist region image through open set object detection, outputting and saving the product recognition event record, and obtaining the shopping checkout detection result includes:

[0010] Extract features of the product to be identified from the wrist region image that belong to the candidate recognition region;

[0011] Obtain at least one candidate product category feature;

[0012] A classification logic value is obtained by calculating the similarity between the features of the product to be identified and the features of each candidate product category.

[0013] Normalize all the classification logic values ​​to obtain the probability values ​​of the product to be identified belonging to each candidate product category;

[0014] Based on the probability values, the candidate product category with the highest probability value is determined as the product category of the product to be identified;

[0015] Output and save product identification event records that include at least the product category to obtain the shopping checkout detection result.

[0016] In one possible implementation, extracting features of the item to be identified from the wrist region image that belongs to the candidate recognition region includes:

[0017] Candidate recognition areas are determined based on the wrist area and the preset cashier interaction area;

[0018] The candidate recognition region satisfies all of the following constraints:

[0019] The ratio of the area of ​​the overlapping region between the candidate recognition region and the wrist region to the area of ​​the wrist region satisfies the first preset ratio constraint.

[0020] The center point of the candidate recognition area is located within the preset cashier interaction area, and the area ratio of the candidate recognition area to the preset cashier interaction area satisfies the second preset ratio constraint.

[0021] The items in the candidate recognition area are the products to be identified, determined according to a preset set of target product categories.

[0022] In one possible implementation, determining whether each person object triggers a shopping behavior based on the posture detection result and the motion trajectory through trajectory analysis and posture analysis includes:

[0023] Based on the movement trajectory, the dwell time of the person in the preset cashier detection area is determined by determining whether the center point of the upper body bounding box in the movement trajectory is within the preset cashier detection area.

[0024] Based on the posture detection results, by determining the positional relationship between the wrist key points and the preset cashier interaction area, it is determined whether the human object has interactive behavior with the cashier.

[0025] If the dwell time reaches a preset time threshold and it is determined that the person interacts with the cashier, then it is determined that the person has triggered a shopping behavior.

[0026] In one possible implementation, determining whether the person object interacts with the cashier based on the posture detection results, by determining the positional relationship between the wrist key points and the preset cashier interaction area, includes:

[0027] Determine the boundary points based on the preset cashier interaction area;

[0028] Calculate a scalar value for determining the positional relationship based on the coordinates of the boundary point and the wrist key point;

[0029] If the scalar value satisfies the preset scalar threshold constraint, and the movement direction of the wrist key point relative to the boundary line of the preset cashier interaction area is consistent with the preset interaction direction, then it is determined that the wrist key point has crossed the boundary line, and the interaction behavior is confirmed.

[0030] In one possible implementation, if any of the character objects triggers the shopping behavior, then for the shopping behavior trigger event of the character object, the detection record corresponding to the shopping behavior trigger event is output and saved, including:

[0031] In response to determining that the shopping behavior is triggered by the person object, obtain the unique identifier of the person object;

[0032] Based on the unique identifier, obtain at least one preceding video frame, the event-triggered video frame, and at least one subsequent video frame, based on the time of the shopping behavior trigger event.

[0033] Output and save the detection record, which includes the acquired unique identifier, the video frame, and metadata related to the shopping behavior triggering event, wherein the metadata includes at least a timestamp and the coordinates of the wrist key point.

[0034] In one possible implementation, determining the upper body bounding box of a person based on the human bounding box, and tracking the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same person, includes:

[0035] Obtain the bounding boxes of all upper body figures in the current video frame;

[0036] Based on the trajectory status of all the upper body bounding boxes in the previous video frame, predict the trajectory position of all the upper body bounding boxes in the current video frame.

[0037] The upper body bounding box of the current video frame is matched with the predicted trajectory position to determine the successfully matched upper body bounding box and the predicted trajectory position.

[0038] Based on the successfully matched upper body bounding box and the predicted trajectory position, the motion trajectory is updated to obtain the trajectory tracking result of the current video frame.

[0039] A second aspect of this disclosure provides an apparatus for detecting fraudulent activities during shopping checkout, the apparatus comprising:

[0040] The posture detection module is configured to acquire multiple consecutive video frames of the cashier within a preset range, and for the video frames containing human figures, detect the human body bounding box and posture key points of the human figures in the video frames to obtain posture detection results, wherein the posture key points include wrist key points.

[0041] The trajectory tracking module is configured to determine the upper body bounding box of the human body object based on the human body bounding box, and track the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same human body object.

[0042] The shopping behavior detection module is configured to determine whether each of the human objects has triggered a shopping behavior based on the posture detection results and the motion trajectory through trajectory analysis and posture analysis. If any human object triggers the shopping behavior, the module outputs and saves the detection record corresponding to the shopping behavior trigger event for the human object, thereby obtaining the shopping behavior detection result.

[0043] The shopping checkout detection module is configured to, when the shopping behavior detection result indicates the existence of a shopping behavior triggering event, determine the wrist region image based on the wrist key points corresponding to the video frame, perform product recognition on the wrist region image through open set object detection, output and save the product recognition event record, and obtain the shopping checkout detection result.

[0044] A third aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0045] A fourth aspect of this disclosure provides an electronic device, comprising:

[0046] A memory on which computer programs are stored;

[0047] A processor for executing the computer program in the memory to implement the steps of the method of any one of the first aspects.

[0048] This invention provides a method, apparatus, device, and medium for detecting fraudulent activities during online checkout. Compared with existing technologies, it has the following advantages:

[0049] The system acquires multiple consecutive video frames of the checkout counter within a preset range. For video frames containing human figures, it detects the human body bounding box and pose key points of the human figures in the video frames to obtain pose detection results. The pose key points include wrist key points. Based on the human body bounding box, it determines the upper body bounding box of the human figure and tracks the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same human figure. Based on the pose detection results and the motion trajectory, it determines whether each human figure triggers a shopping behavior through trajectory analysis and pose analysis. If any human figure triggers a shopping behavior, it outputs and saves the detection record corresponding to the shopping behavior trigger event to obtain a shopping behavior detection result. If the shopping behavior detection result indicates the existence of a shopping behavior trigger event, it determines the wrist region image based on the wrist key points corresponding to the video frames and performs product recognition on the wrist region image through open set object detection. It outputs and saves the product recognition event record to obtain a shopping checkout detection result. The shopping behavior detection method that combines pose keypoints and region geometric constraints can accurately identify shopping behavior, improve the accuracy and reliability of detection, and open set object detection does not rely on large-scale labeled datasets, which can significantly reduce the cost and time of product recognition.

[0050] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0051] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0052] Figure 1 This is a flowchart illustrating a method for detecting fraudulent activities during shopping checkout, according to an embodiment of the specification.

[0053] Figure 2 A block diagram of a device for detecting fraudulent shopping checkout behavior is shown in the embodiment of the specification.

[0054] Figure 3 This is a block diagram illustrating another device for detecting fraudulent shopping checkout behavior, as shown in the embodiments of the specification. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] This application provides a method for detecting fraudulent activities during shopping checkout. Figure 1 This is a flowchart illustrating a method for detecting fraudulent activity during online checkout, according to one embodiment. The method includes:

[0057] In step S11, multiple consecutive video frames of the cashier within a preset range are acquired, and for the video frames containing human figures, the human body bounding box and pose key points of the human figures in the video frames are detected to obtain pose detection results, wherein the pose key points include wrist key points.

[0058] Wherein, the video frame is a single static image in the video stream and is the basic unit of video processing; the human object is a human target instance detected in the video frame; the human bounding box is the region used in computer vision to locate the position of the human object in the video frame; the pose key points are the body joint positions obtained in human pose detection; and the wrist key points are the wrist joint positions.

[0059] In this embodiment, an image acquisition device (such as an RGB camera) deployed above the cash register acquires multiple consecutive video frames of the cash register within a preset range. The image acquisition device can acquire video streams at a rate of 25-30 frames per second, and the video resolution can be set to common formats such as 1920×1080 or 1280×720 as needed. Then, human object detection processing is performed on each video frame. Specifically, a deep learning-based object detection algorithm (such as an improved version of YOLOv5 or YOLOv7) can be used to detect each human object in the image. For each detected human object, its human bounding box information is output. This bounding box can be a rectangle, represented by the coordinates of its upper left and lower right corners, such as ([x1, y1, x2, y2]).

[0060] Based on the obtained human bounding box, pose estimation is further performed on each detected human object. Human pose estimation algorithm based on convolutional neural network (such as HRNet or an improved version of OpenPose) is used to extract the pose key points of each human object. The pose key points include major human joints such as the top of the head, neck, shoulders, elbows, wrists, hips, knees and ankles, with particular attention paid to the localization of the wrist key point.

[0061] The above processing procedure is repeated for all video frames in a series of consecutive video frames to generate a series of human bounding boxes and pose key point data for each detected human object, thereby obtaining pose detection results. The pose detection results are temporarily stored in a memory buffer for subsequent shopping behavior analysis and product recognition processing.

[0062] In step S12, based on the human body bounding box, the upper body bounding box of the human object is determined, and the upper body bounding box is tracked in the consecutive video frames to obtain the motion trajectory of the same human object.

[0063] The upper half body frame is the upper body area cropped from the complete human body bounding box, usually from the head to the waist. The motion trajectory is the position sequence of the same human object in consecutive video frames, which can be used to analyze speed, direction and behavior patterns.

[0064] In this embodiment, each human body bounding box is represented in rectangular coordinates, including center point coordinates (x_center, y_center), width, and height. The corresponding upper body bounding box is calculated based on the detected entire human body bounding box. The calculation rule can be: based on the upper body occupying approximately 1 / 2 to 2 / 3 of the human body height, the height of the human body bounding box is adjusted to 60% of its height, while keeping the width unchanged, and the center point is horizontally aligned and vertically shifted by 10%. Then, a multi-target tracking algorithm based on Kalman filtering and multi-feature matching is used. Specifically, the ByteTrack algorithm is used to associate the upper body bounding boxes in multiple consecutive video frames, assigning a unique identifier (track_id) to each detected upper body bounding box, and maintaining consistency in identity in subsequent video frames. The tracking process comprehensively considers the target's appearance features (e.g., extracted through a convolutional neural network) and motion features (position and velocity predicted through Kalman filtering). The positional changes of the upper body bounding box in multiple consecutive video frames are recorded to obtain the motion trajectory for the same human object.

[0065] In step S13, based on the posture detection result and the motion trajectory, it is determined whether each of the human objects has triggered a shopping behavior through trajectory analysis and posture analysis. If any human object triggers the shopping behavior, the detection record corresponding to the shopping behavior trigger event is output and saved for the shopping behavior trigger event of the human object, and the shopping behavior detection result is obtained.

[0066] The motion trajectory analysis is a process of quantifying the motion trajectory, including the extraction of features such as dwell time, speed, and direction. The posture analysis is based on the spatial relationship calculation of key point coordinates to determine specific behavioral intentions, such as shopping behavior.

[0067] In this embodiment, the coordinates of the wrist key points and the movement trajectory are acquired in real time, and a dual condition analysis of trajectory analysis and posture analysis is performed on each person object: the dwell time of the person object in the preset detection area in front of the checkout counter is calculated. When the continuous dwell time exceeds a preset time threshold, such as 3 seconds, the first trigger condition is met; the relative positional relationship between the wrist key points of the person object and the preset checkout counter interaction area is detected in real time, and when the wrist key points are detected to cross the preset checkout counter boundary line in a specific direction, the second trigger condition is met; only when the same person object meets the above two conditions simultaneously is it determined that the person object has triggered a shopping behavior.

[0068] When it is determined that the shopping behavior is triggered by the person, a standardized detection record is generated for the shopping behavior trigger event of the person, the detection record is saved, and the shopping behavior detection result is obtained. The detection record includes at least the unique identity of the person, the behavior trigger timestamp, and the behavior type identifier. The associated multimodal data is automatically saved. The multimodal data includes multiple video frames before and after the behavior trigger time, wrist key point coordinate data, cashier interaction area coordinates, etc. The multimodal data is encapsulated in JSON format.

[0069] In step S14, if the shopping behavior detection result indicates the existence of a shopping behavior triggering event, the wrist region image is determined based on the wrist key points corresponding to the video frame, and the product is identified in the wrist region image through open set object detection. The product identification event record is output and saved to obtain the shopping checkout detection result.

[0070] The wrist region image is a fixed-size (e.g., 200×200 pixels) or dynamic-size (based on hand scale) region constructed based on wrist key points, and the image is obtained by cropping the original video frame according to the region. The open set object detection is a detection technology that only requires text description or example image, without retraining the model, and can dynamically identify newly added products.

[0071] In this embodiment, when the shopping behavior detection result indicates the existence of a shopping behavior triggering event, the shopping behavior detection result is output through a message queue, such as RabbitMQ, to trigger the start of the shopping checkout detection module. Based on the wrist key points corresponding to the video frame, a wrist region image is determined. For example, a 200×200 pixel rectangular region can be constructed with the wrist key points as the center, and the size can be adaptively adjusted according to the image resolution. The wrist region image is cropped from the original video frame and subjected to illumination normalization and image enhancement processing. An open set object detection architecture based on a vision-language model is adopted, and a backbone network such as ResNet-50 is used to extract the features of the product to be identified in the wrist region image. A cue learning module generates learnable category cue words, i.e., candidate product categories, and works in collaboration with CLIP (Contrastive Language-Image Pre-Training) text encoder to calculate the similarity matrix between the features of the product to be identified and the features of each candidate product category. The candidate product category with the highest similarity is taken as the product category of the product to be identified. Standardized product identification event records are generated and saved to obtain the shopping checkout detection result.

[0072] The method in the above technical solution acquires multiple consecutive video frames of the checkout counter within a preset range. For video frames containing human figures, it detects the human body bounding box and pose key points of the human figures in the video frames to obtain pose detection results. The pose key points include wrist key points. Based on the human body bounding box, it determines the upper body bounding box of the human figure and tracks the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same human figure. Based on the pose detection results and the motion trajectory, it determines whether each human figure triggers a shopping behavior through trajectory analysis and pose analysis. If any human figure triggers a shopping behavior, it outputs and saves the detection record corresponding to the shopping behavior trigger event to obtain a shopping behavior detection result. If the shopping behavior detection result indicates the existence of a shopping behavior trigger event, it determines the wrist region image based on the wrist key points corresponding to the video frames and performs product recognition on the wrist region image through open set object detection, outputs and saves the product recognition event record to obtain a shopping checkout detection result. By analyzing the movement trajectory and posture of individuals, dynamic analysis of user shopping behavior can effectively distinguish between genuine shopping activities and irrelevant background events, improving detection accuracy. Furthermore, the introduction of a behavior-triggered product detection mechanism activates product recognition only when shopping behavior is detected, avoiding blind inference across all video frames, reducing computational overhead, and enhancing the system's real-time responsiveness. Employing open-set object detection eliminates the need for large-scale labeled data and repeated training to identify diverse product categories, adapting to scenarios with frequent product updates and dynamic changes, significantly reducing deployment and maintenance costs. Automatically saving relevant data for key behavioral events improves data management and review efficiency. In summary, this method can improve detection and recognition accuracy while reducing costs and increasing efficiency.

[0073] In one possible implementation, step S14, which involves performing product recognition on the wrist region image using open set object detection, outputting and saving the product recognition event record, and obtaining the shopping checkout detection result, includes:

[0074] Extract features of the product to be identified from the wrist region image that belong to the candidate recognition region;

[0075] Obtain at least one candidate product category feature;

[0076] A classification logic value is obtained by calculating the similarity between the features of the product to be identified and the features of each candidate product category.

[0077] Normalize all the classification logic values ​​to obtain the probability values ​​of the product to be identified belonging to each candidate product category;

[0078] Based on the probability values, the candidate product category with the highest probability value is determined as the product category of the product to be identified;

[0079] Output and save product identification event records that include at least the product category to obtain the shopping checkout detection result.

[0080] The candidate recognition region is the region to be detected after being filtered by conditional constraints. The candidate product category feature is the embedding vector of the preset product category, which is generated through three mechanisms: text prompts, visual prompts, and prompt-free mode. The classification logic value is the original similarity score between the feature of the product to be identified and the candidate product category feature. The probability value is the confidence level of the product belonging to each category after normalization.

[0081] In this embodiment, a wrist region image is input into a YOLOE-v8L-seg model (yoloe-v8l-seg.pt) that incorporates a zero-shot learning mechanism during pre-training. Features of the product to be identified are extracted from the wrist region image through a backbone network and a feature pyramid (Neck), generating a deep feature vector ф(I)∈R for the product to be identified. d Simultaneously, candidate product category features ψ(c)∈R are obtained through at least one of three mechanisms: text prompts, visual prompts, and prompt-free mode. d Among them, text prompts can encode product category names (such as "Cola can" and "potato chip bag") into semantic vectors; visual prompts can generate visually guided embedded representations from example product images; and prompt-free mode can automatically retrieve candidate product categories by calling the built-in general product feature library.

[0082] Next, the system calculates the dot product similarity between the depth feature vector ф(I) of the product to be identified and the category feature ψ(c) of each candidate product, obtaining the classification logical value. All classification logical values ​​are then normalized using the mathematical function Softmax to convert them into probability values ​​P(c|I). The reasoning process can be expressed as follows:

[0083] P(c∣I)=Softmax(ф(I)T·ψ(c));

[0084] The candidate product category with the highest probability value is selected as the product category of the product to be identified. For example, the probability value of cherry-flavored cola of brand A is 0.08, the probability value of original cola of brand A is 0.9, and the probability value of original cola of brand B is 0.02. Original cola of brand A is determined as the product category of the product to be identified, and a standardized product identification event record is generated to obtain the shopping checkout detection result.

[0085] The above technical solution can identify product categories that do not appear in the training set, without the need to re-collect or label new product category data for new products. It is adapted to dynamic retail environments, can significantly reduce deployment costs, and has a simpler identification process, thus improving identification efficiency.

[0086] In one possible implementation, step S14, extracting features of the item to be identified from the wrist region image that belong to the candidate recognition region, includes:

[0087] Candidate recognition areas are determined based on the wrist area and the preset cashier interaction area;

[0088] The candidate recognition region satisfies all of the following constraints:

[0089] The ratio of the area of ​​the overlapping region between the candidate recognition region and the wrist region to the area of ​​the wrist region satisfies the first preset ratio constraint.

[0090] The center point of the candidate recognition area is located within the preset cashier interaction area, and the area ratio of the candidate recognition area to the preset cashier interaction area satisfies the second preset ratio constraint.

[0091] The items in the candidate recognition area are the products to be identified, determined according to a preset set of target product categories.

[0092] The wrist area is an image area determined by a preset size (e.g., 200×200 pixels) based on the wrist key points obtained from human posture detection. The preset cashier interaction area is the coordinate mapping of the physical space of the cashier in the image, which can be a polygon or a rectangle. The preset target product category set is the retail product classification system to be identified, such as beverages, snacks, etc., with more than 200 subcategories.

[0093] In this embodiment, the overlap area (IoU) between the candidate recognition area and the wrist area is not less than 60% of the wrist area, i.e., the overlap ratio is ≥0.6, to ensure that the product to be recognized is within the hand operation range; the center point of the candidate recognition area must be located within the cashier interaction area, and its area must not exceed 50% of the cashier interaction area, to avoid false detection of distant products or large interfering objects; according to a preset target product category set, the items within the candidate recognition area are classified, and only candidate recognition areas belonging to the preset target product category set (such as beverages, snacks, and 200+ retail subcategories) are retained.

[0094] The above technical solution effectively eliminates interfering targets such as goods in the background of shelves and non-commodity items held by people, such as people holding mobile phones, by using spatial and semantic constraints. It also identifies irrelevant candidate areas in advance, reduces the amount of detection, and lowers the false detection rate.

[0095] In one possible implementation, step S13, determining whether each person object triggers a shopping behavior based on the posture detection result and the motion trajectory through trajectory analysis and posture analysis, includes:

[0096] Based on the movement trajectory, the dwell time of the person in the preset cashier detection area is determined by determining whether the center point of the upper body bounding box in the movement trajectory is within the preset cashier detection area.

[0097] Based on the posture detection results, by determining the positional relationship between the wrist key points and the preset cashier interaction area, it is determined whether the human object has interactive behavior with the cashier.

[0098] If the dwell time reaches a preset time threshold and it is determined that the person interacts with the cashier, then it is determined that the person has triggered a shopping behavior.

[0099] The preset checkout detection area is a preset rectangular or polygonal area, such as a 1.5×2m area in front of the checkout counter. 2 Range, used to define the effective trajectory analysis space.

[0100] In this embodiment, based on the motion trajectory, it is calculated in real time whether the center point of the upper body bounding box of the person is located within a preset checkout detection area, such as a rectangular area of ​​1.5 meters × 2 meters in front of the checkout counter, and the continuous dwell time is accumulated through a timing module. The coordinates of the key points of the person's wrist are extracted and combined with the boundary line of the preset checkout interaction area, such as a virtual line segment based on the checkout counter scanner, to determine the scalar value by calculating the spatial geometric relationship, thereby determining the positional relationship between the wrist key points and the preset checkout interaction area. If the scalar value is greater than a preset scalar threshold and the movement direction of the wrist key points is consistent with the preset interaction direction vector, such as moving from the person's side to the cashier's side, then an interaction behavior is determined to have occurred. When the same person simultaneously meets the conditions of dwell time not less than a preset time threshold (such as 3 seconds) and the existence of interaction behavior, it is determined that the person has triggered a shopping behavior.

[0101] The above technical solution integrates dwell time and gesture interaction for dual determination, which can significantly reduce the false detection rate, improve the accuracy and reliability of shopping behavior detection, and thus improve the accuracy and reliability of shopping checkout fraud risk detection.

[0102] In one possible implementation, step S13, determining whether the person object interacts with the cashier based on the posture detection results by determining the positional relationship between the wrist key points and the preset cashier interaction area, includes:

[0103] Determine the boundary points based on the preset cashier interaction area;

[0104] Calculate a scalar value for determining the positional relationship based on the coordinates of the boundary point and the wrist key point;

[0105] If the scalar value satisfies the preset scalar threshold constraint, and the movement direction of the wrist key point relative to the boundary line of the preset cashier interaction area is consistent with the preset interaction direction, then it is determined that the wrist key point has crossed the boundary line, and the interaction behavior is confirmed.

[0106] Wherein, the boundary point is a point on the boundary line of the preset cashier interaction area, the scalar value is a numerical judgment quantity obtained through geometric operations (such as cross product), representing the orientation relationship between the wrist key point and the boundary line, the movement direction is the displacement vector of the wrist key point in continuous frames, and the preset interaction direction is the spatial motion vector that conforms to the checkout operation in shopping behavior.

[0107] In this embodiment of the disclosure, based on the boundary line of the preset cashier interaction area, two boundary points are determined as P1(x1,y1) and P2(x2,y2), and the coordinates of the wrist key point are P... w (x w ,y w The scalar value is Value, and the formula for calculating the scalar value can be:

[0108] Value=(y2-y1)(x w -x1)-(x2-x1)(y w -y1);

[0109] The above technical solution achieves interactive behavior recognition through spatial geometric calculation. It has a fast calculation speed and high accuracy, can adapt to different physical sizes of cash registers, and has strong generalization ability and deployment flexibility.

[0110] In one possible implementation, in step S13, if any of the character objects triggers the shopping behavior, then for the shopping behavior triggering event of the character object, outputting and saving the detection record corresponding to the shopping behavior triggering event includes:

[0111] In response to determining that the shopping behavior is triggered by the person object, obtain the unique identifier of the person object;

[0112] Based on the unique identifier, obtain at least one preceding video frame, the event-triggered video frame, and at least one subsequent video frame, based on the time of the shopping behavior trigger event.

[0113] Output and save the detection record, which includes the acquired unique identifier, the video frame, and metadata related to the shopping behavior triggering event, wherein the metadata includes at least a timestamp and the coordinates of the wrist key point.

[0114] The unique identifier is a number used to uniquely identify and distinguish different individuals, and the metadata is structured information describing behavioral events.

[0115] In this embodiment of the disclosure, when the shopping behavior of the person is determined, the unique identifier of the person is immediately obtained. Using the unique identifier as an index, multiple video frames within a preset time period based on the time of the shopping behavior trigger event are extracted from multiple consecutive video frames. These include several preceding video frames, an event-triggered video frame, and several subsequent video frames after the trigger. For example, 3 preceding video frames, 1 event-triggered video frame, and 3 subsequent video frames after the trigger. The extraction of multiple video frames within the preset time period based on the time of the shopping behavior trigger event can be dynamically adjusted according to the video frame rate, such as 30fps / 60fps. Simultaneously, a standardized detection record containing the unique identifier, preceding video frames, event-triggered video frames, subsequent video frames, and related metadata is generated. The metadata includes at least the wrist key point coordinates and a timestamp. Finally, the detection record can be output in a standardized format and persistently stored in a distributed database and object storage system.

[0116] The above technical solution enables complete digital recording of shopping behavior-triggered events, ensuring data integrity while supporting real-time processing in high-concurrency scenarios, and providing a standardized data foundation for subsequent behavior analysis and system optimization.

[0117] In one possible implementation, step S13, which involves determining the upper body bounding box of a person based on the human body bounding box, and tracking the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same person, includes:

[0118] Obtain the bounding boxes of all upper body figures in the current video frame;

[0119] In this embodiment of the disclosure, real-time inference is performed on the current video frame to detect all visible upper body bounding boxes. Each detection result includes the upper body bounding box coordinates (e.g., center point x, y, width w, height h), confidence score, and category label (fixed as upper_body).

[0120] Based on the trajectory status of all the upper body bounding boxes in the previous video frame, predict the trajectory position of all the upper body bounding boxes in the current video frame.

[0121] In this embodiment of the disclosure, based on the trajectory states of all the upper body bounding boxes in the previous video frame, a Kalman filter can be used to predict the trajectory position: An 8-dimensional state vector is established: [x,y,w,h,v] x ,v y ,v w ,v h A constant velocity model is used as the system dynamics model to predict the trajectory position and size of the upper body bounding box of each trajectory in the current video frame, and the Mahalanobis distance between the predicted upper body bounding box and the detected upper body bounding box is calculated as a motion similarity measure.

[0122] The upper body bounding box of the current video frame is matched with the predicted trajectory position to determine the successfully matched upper body bounding box and the predicted trajectory position.

[0123] Based on the successfully matched upper body bounding box and the predicted trajectory position, the motion trajectory is updated to obtain the trajectory tracking result of the current video frame.

[0124] In this embodiment, the Hungarian algorithm can be used for optimal matching. The Intersection over Union (IOU) between the upper body bounding box of the current video frame and the predicted trajectory position is used as the main association index. A minimum IOU threshold is set, for example, 0.3. Matching pairs below the minimum IOU threshold are discarded. Then, the matching is performed in two stages: First, the upper body bounding box of the current video frame is associated with the predicted trajectory position. This first round of association only uses high-confidence upper body bounding boxes with confidence levels higher than the first threshold, and matching is performed based on the motion similarity between the upper body bounding box and the predicted trajectory position, i.e., the predicted upper body bounding box. For the trajectory positions and high-confidence upper body bounding boxes that are not matched in the first round of association, they are associated with low-confidence upper body bounding boxes with confidence levels lower than the first threshold but higher than the second threshold, and matching is also performed based on motion similarity. The high-confidence detection boxes that are still not matched after the second round of association are retained as the initial values ​​for the new trajectory, and all unmatched low-confidence detection boxes are discarded.

[0125] Based on the successfully matched upper body bounding box and the predicted trajectory position, update the motion trajectory, update the state of all matched trajectories, and output the trajectory tracking result of the current frame.

[0126] The above technical solution can effectively and accurately associate the upper body bounding box by tracking the movement trajectory of the upper body bounding box, thereby improving the continuity and accuracy of shopping behavior detection results.

[0127] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0128] In one embodiment, such as Figure 2 As shown, a device for detecting fraudulent activities during shopping checkout is provided, comprising:

[0129] The posture detection module 210 is configured to acquire multiple consecutive video frames of the cash register within a preset range, and for the video frames containing human figures, detect the human body bounding box and posture key points of the human figures in the video frames to obtain posture detection results, wherein the posture key points include wrist key points.

[0130] The trajectory tracking module 220 is configured to determine the upper body bounding box of a person based on the human body bounding box, and track the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same person.

[0131] The shopping behavior detection module 230 is configured to determine whether each of the human objects has triggered a shopping behavior based on the posture detection result and the motion trajectory through trajectory analysis and posture analysis. If any human object triggers the shopping behavior, the module outputs and saves the detection record corresponding to the shopping behavior trigger event for the human object, thereby obtaining the shopping behavior detection result.

[0132] The shopping checkout detection module 240 is configured to, when the shopping behavior detection result indicates the existence of a shopping behavior triggering event, determine a wrist region image based on the wrist key points corresponding to the video frame, perform product recognition on the wrist region image through open set object detection, output and save the product recognition event record, and obtain the shopping checkout detection result.

[0133] In one possible implementation, the shopping checkout detection module 240 is configured as follows:

[0134] Extract features of the product to be identified from the wrist region image that belong to the candidate recognition region;

[0135] Obtain at least one candidate product category feature;

[0136] A classification logic value is obtained by calculating the similarity between the features of the product to be identified and the features of each candidate product category.

[0137] Normalize all the classification logic values ​​to obtain the probability values ​​of the product to be identified belonging to each candidate product category;

[0138] Based on the probability values, the candidate product category with the highest probability value is determined as the product category of the product to be identified;

[0139] Output and save product identification event records that include at least the product category to obtain the shopping checkout detection result.

[0140] In one possible implementation, the shopping checkout detection module 240 is configured as follows:

[0141] Candidate recognition areas are determined based on the wrist area and the preset cashier interaction area;

[0142] The candidate recognition region satisfies all of the following constraints:

[0143] The ratio of the area of ​​the overlapping region between the candidate recognition region and the wrist region to the area of ​​the wrist region satisfies the first preset ratio constraint.

[0144] The center point of the candidate recognition area is located within the preset cashier interaction area, and the area ratio of the candidate recognition area to the preset cashier interaction area satisfies the second preset ratio constraint.

[0145] The items in the candidate recognition area are the products to be identified, determined according to a preset set of target product categories.

[0146] In one possible implementation, the shopping behavior detection module 230 is configured as follows:

[0147] Based on the movement trajectory, the dwell time of the person in the preset cashier detection area is determined by determining whether the center point of the upper body bounding box in the movement trajectory is within the preset cashier detection area.

[0148] Based on the posture detection results, by determining the positional relationship between the wrist key points and the preset cashier interaction area, it is determined whether the human object has interactive behavior with the cashier.

[0149] If the dwell time reaches a preset time threshold and it is determined that the person interacts with the cashier, then it is determined that the person has triggered a shopping behavior.

[0150] In one possible implementation, the shopping behavior detection module 230 is configured as follows:

[0151] Determine the boundary points based on the preset cashier interaction area;

[0152] Calculate a scalar value for determining the positional relationship based on the coordinates of the boundary point and the wrist key point;

[0153] If the scalar value satisfies the preset scalar threshold constraint, and the movement direction of the wrist key point relative to the boundary line of the preset cashier interaction area is consistent with the preset interaction direction, then it is determined that the wrist key point has crossed the boundary line, and the interaction behavior is confirmed.

[0154] In one possible implementation, the shopping behavior detection module 230 is configured as follows:

[0155] In response to determining that the shopping behavior is triggered by the person object, obtain the unique identifier of the person object;

[0156] Based on the unique identifier, obtain at least one preceding video frame, the event-triggered video frame, and at least one subsequent video frame, based on the time of the shopping behavior trigger event.

[0157] Output and save the detection record, which includes the acquired unique identifier, the video frame, and metadata related to the shopping behavior triggering event, wherein the metadata includes at least a timestamp and the coordinates of the wrist key point.

[0158] In one possible implementation, the trajectory tracking module 220 is configured as follows:

[0159] Obtain the bounding boxes of all upper body figures in the current video frame;

[0160] Based on the trajectory status of all the upper body bounding boxes in the previous video frame, predict the trajectory position of all the upper body bounding boxes in the current video frame.

[0161] The upper body bounding box of the current video frame is matched with the predicted trajectory position to determine the successfully matched upper body bounding box and the predicted trajectory position.

[0162] Based on the successfully matched upper body bounding box and the predicted trajectory position, the motion trajectory is updated to obtain the trajectory tracking result of the current video frame.

[0163] In one possible implementation, the device further includes a recording tool module configured to:

[0164] Save and manage the detection records of shopping behavior trigger events and product recognition event records to provide a complete basis for subsequent manual review.

[0165] For specific limitations regarding a device for detecting fraudulent activities at checkout, please refer to the limitations of a method for detecting fraudulent activities at checkout described above, and will not be repeated here. Each module in the aforementioned device for detecting fraudulent activities at checkout can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0166] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described in the foregoing embodiments.

[0167] This disclosure also provides an electronic device, including:

[0168] A memory on which computer programs are stored;

[0169] A processor for executing the computer program in the memory to implement the steps of any of the methods described in the foregoing embodiments.

[0170] Figure 3 The illustrated device 100 for detecting fraudulent activities at checkout includes a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the device 100 may further include a communication component, which can be used for data interaction between the device 100 and other devices, such as sending or receiving data. It should be noted that in actual operation, the communication component is not limited to one, and the structure of this device 100 for detecting fraudulent activities at checkout does not constitute a limitation on the embodiments of this application.

[0171] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0172] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0173] The memory 1003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing program code and capable of being read by a computer, without limitation herein.

[0174] The memory 1003 is used to store program code for executing embodiments of the present disclosure, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the program code stored in the memory 1003 to implement the steps shown in the foregoing embodiment of a method for detecting shopping checkout fraud risk behavior.

[0175] This disclosure also provides a computer-readable storage medium storing program code. When the program code is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiment for detecting shopping settlement fraud risk behavior.

[0176] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present disclosure, various changes, modifications, substitutions and variations can be made to these embodiments, and all such changes, modifications, substitutions and variations fall within the protection scope of the present disclosure.

[0177] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction, and such combinations should also be considered as part of this disclosure. To avoid unnecessary repetition, this disclosure will not further describe the various possible combinations. The technical scope of this application is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for detecting fraudulent activities during shopping checkout, characterized in that, The method includes: Acquire multiple consecutive video frames of the cashier within a preset range, and for the video frames containing human figures, detect the human body bounding box and pose key points of the human figures in the video frames to obtain pose detection results, wherein the pose key points include wrist key points. Based on the human body bounding box, the upper body bounding box of the human object is determined, and the upper body bounding box is tracked in the consecutive video frames to obtain the motion trajectory of the same human object. Based on the posture detection results and the motion trajectory, through trajectory analysis and posture analysis, it is determined whether each of the human objects has triggered a shopping behavior. If any human object triggers the shopping behavior, the detection record corresponding to the shopping behavior trigger event is output and saved for the shopping behavior trigger event of the human object, and the shopping behavior detection result is obtained. If the shopping behavior detection result indicates the presence of a shopping behavior triggering event, the wrist region image is determined based on the wrist key points corresponding to the video frame. Then, the product is identified in the wrist region image through open set object detection. The product identification event record is output and saved to obtain the shopping checkout detection result.

2. The method according to claim 1, characterized in that, The process of performing product recognition on the wrist region image through open set object detection, outputting and saving product recognition event records, and obtaining shopping checkout detection results includes: Extract features of the product to be identified from the wrist region image that belong to the candidate recognition region; Obtain at least one candidate product category feature; A classification logic value is obtained by calculating the similarity between the features of the product to be identified and the features of each candidate product category. Normalize all the classification logic values ​​to obtain the probability values ​​of the product to be identified belonging to each candidate product category; Based on the probability values, the candidate product category with the highest probability value is determined as the product category of the product to be identified; Output and save product identification event records that include at least the product category to obtain the shopping checkout detection result.

3. The method according to claim 2, characterized in that, The step of extracting features of the product to be identified from the wrist region image that belong to the candidate recognition region includes: Candidate recognition areas are determined based on the wrist area and the preset cashier interaction area; The candidate recognition region satisfies all of the following constraints: The ratio of the area of ​​the overlapping region between the candidate recognition region and the wrist region to the area of ​​the wrist region satisfies the first preset ratio constraint. The center point of the candidate recognition area is located within the preset cashier interaction area, and the area ratio of the candidate recognition area to the preset cashier interaction area satisfies the second preset ratio constraint. The items in the candidate recognition area are the products to be identified, determined according to a preset set of target product categories.

4. The method according to claim 1, characterized in that, The step of determining whether each person object triggers a shopping behavior based on the posture detection result and the motion trajectory through trajectory analysis and posture analysis includes: Based on the movement trajectory, the dwell time of the person in the preset cashier detection area is determined by determining whether the center point of the upper body bounding box in the movement trajectory is within the preset cashier detection area. Based on the posture detection results, by determining the positional relationship between the wrist key points and the preset cashier interaction area, it is determined whether the human object has interactive behavior with the cashier. If the dwell time reaches a preset time threshold and it is determined that the person interacts with the cashier, then it is determined that the person has triggered a shopping behavior.

5. The method according to claim 4, characterized in that, The step of determining whether the person interacts with the cashier based on the posture detection results, by determining the positional relationship between the wrist key points and the preset cashier interaction area, includes: Determine the boundary points based on the preset cashier interaction area; Calculate a scalar value for determining the positional relationship based on the coordinates of the boundary point and the wrist key point; If the scalar value satisfies the preset scalar threshold constraint, and the movement direction of the wrist key point relative to the boundary line of the preset cashier interaction area is consistent with the preset interaction direction, then it is determined that the wrist key point has crossed the boundary line, and the interaction behavior is confirmed.

6. The method according to claim 1, characterized in that, If any of the aforementioned character objects triggers the shopping behavior, then for the shopping behavior trigger event of the character object, the detection record corresponding to the shopping behavior trigger event is output and saved, including: In response to determining that the shopping behavior is triggered by the person object, obtain the unique identifier of the person object; Based on the unique identifier, obtain at least one preceding video frame, the event-triggered video frame, and at least one subsequent video frame, based on the time of the shopping behavior trigger event. Output and save the detection record, which includes the acquired unique identifier, the video frame, and metadata related to the shopping behavior triggering event, wherein the metadata includes at least a timestamp and the coordinates of the wrist key point.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the upper body bounding box of a person based on the human body bounding box, and tracking the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same person includes: Obtain the bounding boxes of all upper body figures in the current video frame; Based on the trajectory status of all the upper body bounding boxes in the previous video frame, predict the trajectory position of all the upper body bounding boxes in the current video frame. The upper body bounding box of the current video frame is matched with the predicted trajectory position to determine the successfully matched upper body bounding box and the predicted trajectory position. Based on the successfully matched upper body bounding box and the predicted trajectory position, the motion trajectory is updated to obtain the trajectory tracking result of the current video frame.

8. A device for detecting fraudulent activities during shopping checkout, characterized in that, The device includes: The posture detection module is configured to acquire multiple consecutive video frames of the cashier within a preset range, and for the video frames containing human figures, detect the human body bounding box and posture key points of the human figures in the video frames to obtain posture detection results, wherein the posture key points include wrist key points. The trajectory tracking module is configured to determine the upper body bounding box of the human body object based on the human body bounding box, and track the upper body bounding box in the multiple consecutive video frames to obtain the motion trajectory of the same human body object. The shopping behavior detection module is configured to determine whether each of the human objects has triggered a shopping behavior based on the posture detection results and the motion trajectory through trajectory analysis and posture analysis. If any human object triggers the shopping behavior, the module outputs and saves the detection record corresponding to the shopping behavior trigger event for the human object, thereby obtaining the shopping behavior detection result. The shopping checkout detection module is configured to, when the shopping behavior detection result indicates the existence of a shopping behavior triggering event, determine the wrist region image based on the wrist key points corresponding to the video frame, perform product recognition on the wrist region image through open set object detection, output and save the product recognition event record, and obtain the shopping checkout detection result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images