Method and device for detecting shopping settlement fraud risk behavior, equipment and medium

Abstract

The invention relates to a method, device and equipment for detecting shopping settlement fraud risk behaviors and a medium, and the method comprises the steps: obtaining a plurality of continuous video frames, and detecting a human body bounding box and a posture key point of a character object in the video frames, thereby obtaining a posture detection result; determining an upper body bounding box according to the human body bounding box, and tracking the upper body bounding box in a plurality of continuous video frames to obtain a motion track; according to the posture detection result and the motion track, determining whether the character object triggers a shopping behavior through track analysis and posture analysis, and if the character object triggers the shopping behavior, outputting and storing a detection record for a shopping behavior triggering event of the character object to obtain a shopping behavior detection result; and under the condition that the shopping behavior detection result represents that a shopping behavior trigger event exists, determining a wrist area image according to the wrist key point, performing commodity identification on the wrist area image through open set target detection, outputting and storing a commodity identification event record, and obtaining a shopping settlement detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a method and device for detecting shopping settlement fraud risk behavior, equipment and medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, computer vision technology is gradually popularized in retail scenarios, covering automatic product recognition, self-service settlement, human object behavior analysis, and security monitoring. However, in the detection of shopping behavior, traditional methods rely on single image features, which are difficult to accurately distinguish between shopping behavior and non-shopping behavior in complex retail scenarios, resulting in high false detection rates. Secondly, in product recognition, widely used target detection models based on closed set assumptions rely heavily on large amounts of high-quality labeled data. For dynamically changing retail products, the cost of data collection and training is high, and it is difficult to quickly adapt to new products. This results in low detection rates and high costs when detecting shopping settlement fraud risk behavior. SUMMARY

[0003] The purpose of the present application is to provide a method, device, equipment and medium for detecting shopping settlement fraud risk behavior, which aims to improve the detection accuracy of shopping settlement fraud risk behavior and reduce the detection cost.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present disclosure provides a method for detecting shopping settlement fraud risk behavior, the method comprising: Obtaining a plurality of continuous video frames within a preset range of a checkout counter, and for the video frames in which a human object exists, detecting a human body bounding box and a posture key point of the human object in the video frames to obtain a posture detection result, wherein the posture key point includes a wrist key point; According to the human body bounding box, determining an upper body bounding box for the human object, and tracking the upper body bounding box in the plurality of continuous video frames to obtain a motion trajectory for the same human object; According to the posture detection result and the motion trajectory, determining whether each human object triggers a shopping behavior through trajectory analysis and posture analysis, and if any human object triggers the shopping behavior, outputting and saving a detection record corresponding to a shopping behavior triggering event of the human object to obtain a shopping behavior detection result; In the case where the shopping behavior detection result represents that there is a shopping behavior triggering event, determining a wrist region image according to the wrist key point corresponding to the video frame, and performing product recognition on the wrist region image through open set target detection to output and save a product recognition event record, and obtaining a shopping settlement detection result.

[0005] In a possible implementation, the shopping settlement detection result is obtained by performing product recognition on the wrist region image through open set target detection, outputting and saving a product recognition event record, and including: extracting features of a to-be-recognized product belonging to a candidate recognition region in the wrist region image; obtaining at least one candidate product category feature; obtaining a classification logic value by calculating a similarity between the features of the to-be-recognized product and each candidate product category feature; obtaining a probability value of the to-be-recognized product belonging to each candidate product category by performing normalization processing on all the classification logic values; determining, according to the probability value, the candidate product category with the highest probability value as the product category of the to-be-recognized product; outputting and saving a product recognition event record including at least the product category to obtain the shopping settlement detection result.

[0006] In a possible implementation, the extracting features of a to-be-recognized product belonging to a candidate recognition region in the wrist region image includes: determining a candidate recognition region according to the wrist region and a preset checkout counter interaction region; the candidate recognition region satisfies all the following constraints: a ratio of an overlapping area of the candidate recognition region and the wrist region to an area of the wrist region satisfies a first preset ratio constraint; a center point of the candidate recognition region is located in the preset checkout counter interaction region, and an area ratio of the candidate recognition region to the preset checkout counter interaction region satisfies a second preset ratio constraint; an object in the candidate recognition region is a to-be-recognized product determined according to a preset target product category set.

[0007] In a possible implementation, the determining, according to the posture detection result and the motion trajectory, whether each person object triggers a shopping behavior through trajectory analysis and posture analysis includes: determining, according to the motion trajectory, a stay time of the person object in a preset checkout counter detection region by determining whether a center point of the upper body bounding box in the motion trajectory of the person object is in the preset checkout counter detection region; determining, according to the posture detection result, whether the person object has an interaction behavior with the checkout counter by determining a positional relationship between the wrist key point and the preset checkout counter interaction region; if the stay time reaches a preset time threshold and it is determined that the person object has an interaction behavior with the checkout counter, it is determined that the person object triggers a shopping behavior.

[0008] In a possible implementation, the determining, according to the pose detection result, whether the character object has an interaction behavior with the checkout counter by determining a positional relationship between the wrist key point and a preset checkout counter interaction region, includes: determining a boundary point according to the preset checkout counter interaction region; calculating a scalar value for determining the positional relationship according to the coordinates of the boundary point and the wrist key point; if the scalar value satisfies a preset scalar threshold constraint, and a moving direction of the wrist key point relative to a boundary line of the preset checkout counter interaction region is consistent with a preset interaction direction, it is determined that the wrist key point crosses the boundary line, and it is confirmed that the interaction behavior occurs.

[0009] In a possible implementation, if any of the character objects triggers the shopping behavior, the detection record corresponding to the shopping behavior triggering event of the character object is output and saved, including: obtaining a unique identity of the character object in response to determining that the character object triggers the shopping behavior; obtaining, according to the unique identity, at least one preceding video frame, an event triggering video frame, and at least one subsequent video frame based on a time point of the shopping behavior triggering event; outputting and saving the detection record including the obtained unique identity, the video frames, and metadata related to the shopping behavior triggering event, wherein the metadata at least includes a timestamp and coordinates of the wrist key point.

[0010] In a possible implementation, the determining, according to the human body bounding box, an upper body bounding box for a character object, and tracking the upper body bounding box in the continuous multiple video frames to obtain a motion trajectory for the same character object, includes: obtaining all the upper body bounding boxes in a current video frame; predicting a trajectory position of all the upper body bounding boxes in the current video frame according to a trajectory state of all the upper body bounding boxes in a previous video frame; matching the upper body bounding boxes in the current video frame with the predicted trajectory positions to determine the upper body bounding boxes and the predicted trajectory positions that are successfully matched; updating the motion trajectory according to the upper body bounding boxes and the predicted trajectory positions that are successfully matched to obtain a trajectory tracking result of the current video frame.

[0011] A second aspect of the embodiments of the present disclosure provides a device for detecting a shopping settlement fraud risk behavior, the device comprising: The posture detection module is configured to acquire a plurality of continuous video frames of the checkout counter within a preset range, and detect, for the video frames in which a human object exists, a human body bounding box and posture key points of the human object in the video frames to obtain a posture detection result, wherein the posture key points include wrist key points; The trajectory tracking module is configured to determine, according to the human body bounding box, an upper body bounding box for the human object, and track the upper body bounding box in the plurality of continuous video frames to obtain a motion trajectory for the same human object. The shopping behavior detection module is configured to determine, according to the posture detection result and the motion trajectory, whether each human object triggers a shopping behavior through trajectory analysis and posture analysis, and if any human object triggers the shopping behavior, output and save a detection record corresponding to a shopping behavior triggering event of the human object to obtain a shopping behavior detection result. The shopping settlement detection module is configured to, in a case where the shopping behavior detection result represents that there is a shopping behavior triggering event, determine a wrist region image according to the wrist key points corresponding to the video frames, and perform product identification on the wrist region image through open set target detection to output and save a product identification event record, thereby obtaining a shopping settlement detection result.

[0012] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method of any one of the first aspect.

[0013] In a fourth aspect, the present disclosure provides an electronic device, comprising: a memory having a computer program stored thereon; a processor configured to execute the computer program in the memory to implement the steps of the method of any one of the first aspect.

[0014] The present disclosure provides a method, device, equipment and medium for detecting shopping settlement fraud risk behavior. Compared with the prior art, the present disclosure has the following advantages: The continuous multiple video frames of the checkout counter within a preset range are acquired, and for the video frames in which a character object exists, a human body bounding box and a posture key point of the character object in the video frame are detected to obtain a posture detection result, wherein the posture key point includes a wrist key point; an upper body bounding box of the character object is determined according to the human body bounding box, and the upper body bounding box is tracked in the continuous multiple video frames to obtain a motion trajectory of the same character object; whether each character object triggers a shopping behavior is determined through trajectory analysis and posture analysis according to the posture detection result and the motion trajectory, if any character object triggers the shopping behavior, a shopping behavior triggering event of the character object is output and saved, and a detection record corresponding to the shopping behavior triggering event is obtained to obtain a shopping behavior detection result; in the case that the shopping behavior detection result represents that there is a shopping behavior triggering event, a wrist region image is determined according to the wrist key point corresponding to the video frame, and open set target detection is performed on the wrist region image to perform product identification, and a product identification event record is output and saved to obtain a shopping settlement detection result. The shopping behavior detection method combining the posture key point and the region geometric constraint can accurately identify the shopping behavior, improve the accuracy and reliability of the detection, and the open set target detection does not need to rely on a large-scale labeled data set, which can significantly reduce the product identification cost and time.

[0015] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, and are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation on the present disclosure. In the drawings: Figure 1 FIG. 1 is a flow diagram illustrating a method for detecting a shopping settlement fraud risk behavior according to an embodiment of the present disclosure.

[0017] Figure 2 FIG. 2 is a block diagram of a device for detecting a shopping settlement fraud risk behavior according to an embodiment of the present disclosure.

[0018] Figure 3 FIG. 3 is another block diagram of a device for detecting a shopping settlement fraud risk behavior according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0020] The application provides a method for detecting shopping settlement fraud risk behaviors, Figure 1 is a flowchart of a method for detecting shopping settlement fraud risk behaviors according to an embodiment. The method comprises: In step S11, a plurality of continuous video frames of a checkout counter within a preset range are acquired, and for the video frames in which a human object exists, a human body bounding box and a posture key point of the human object in the video frame are detected to obtain a posture detection result, wherein the posture key point includes a wrist key point; Wherein, the video frame is a single static image in a video stream, which is the basic unit of video processing, the human object is a human target instance detected in the video frame, the human body bounding box is a region used to locate the position of the human object in the video frame in computer vision, the posture key point is the position of the body joint obtained in human posture detection, and the wrist key point is the position of the wrist joint.

[0021] In the embodiments of the present disclosure, an image acquisition device (such as an RGB camera) deployed above the checkout counter is used to acquire a plurality of continuous video frames of the checkout counter within a preset range. The image acquisition device can acquire a video stream at a rate of 25-30 frames per second, and the video resolution can be set to 1920x1080 or 1280x720 or other common formats according to actual needs; then, a target detection algorithm based on deep learning (such as an improved version of YOLOv5 or YOLOv7) is used to detect each human object in the image. For each detected human object, its human body bounding box information is output, which can be a rectangular box represented by the coordinates of the upper left corner and the lower right corner, such as ([x1, y1, x2, y2]); On the basis of obtaining the human body bounding box, posture estimation is further performed on each detected human object, and a human posture estimation algorithm based on a convolutional neural network (such as an improved version of HRNet or OpenPose) is used to extract posture key points of each human object, including head, neck, shoulder, elbow, wrist, hip, knee and ankle, etc. The positioning of the wrist key point is particularly concerned. The above processing procedure is repeatedly performed on all video frames in the plurality of continuous video frames, so as to generate a series of human body bounding box and posture key point data for each detected human object, and obtain a posture detection result. The posture detection result is temporarily stored in a memory buffer for subsequent shopping behavior analysis and commodity recognition processing.

[0022] In step S12, an upper body bounding box for the human object is determined according to the human body bounding box, and the upper body bounding box is tracked in the plurality of continuous video frames to obtain a motion trajectory for the same human object. wherein the upper body bounding box is the upper body region cropped from the complete human body bounding box, usually from head to waist, and the motion trajectory is the position sequence of the same person object in consecutive video frames, which can be used to analyze the speed, direction and behavior pattern.

[0023] In the embodiments of the present disclosure, each human body bounding box is represented in the form of rectangular coordinates, including center point coordinates (x_center, y_center), width (width) and height (height). The corresponding upper body bounding box is calculated according to the detected complete human body bounding box. The calculation rule can be as follows: the height of the human body bounding box is adjusted to 60% of its height, while the width remains unchanged, the center point is horizontally aligned and vertically moved by 10%, and then a multi-target tracking algorithm based on Kalman filtering and multi-feature matching is used. Specifically, the ByteTrack algorithm is used to associate the upper body bounding boxes in the consecutive multiple video frames, assign a unique identifier (track_id) to each detected upper body bounding box, and maintain identity consistency in subsequent video frames. The tracking process considers the appearance features (extracted by a convolutional neural network) and motion features (predicted position and speed by Kalman filtering) of the target; the position changes of the upper body bounding box in the consecutive multiple video frames are recorded to obtain the motion trajectory of the same person object.

[0024] In step S13, according to the posture detection result and the motion trajectory, the trajectory analysis and the posture analysis are performed to determine whether each of the person objects triggers the shopping behavior. If any of the person objects triggers the shopping behavior, the detection record corresponding to the shopping behavior triggering event of the person object is output and saved to obtain the shopping behavior detection result. wherein the motion trajectory analysis is a quantitative calculation process of the motion trajectory, including the extraction of features such as dwell time, speed, direction, and the posture analysis is a spatial relationship calculation based on key point coordinates to determine a specific behavior intention, such as a shopping behavior.

[0025] In the embodiments of the present disclosure, the coordinates of the wrist key point and the motion trajectory are obtained in real time, and the trajectory analysis and the posture analysis are performed on each of the person objects: the dwell time of the person object in the preset detection area in front of the checkout counter is calculated. When the continuous dwell time exceeds the preset time threshold, for example, 3 seconds, the first triggering condition is met; the relative position relationship between the wrist key point of the person object and the preset checkout counter interaction area is detected in real time. When the wrist key point is detected to cross the preset checkout counter boundary line in a specific direction, the second triggering condition is met; only when the same person object meets the above two conditions at the same time, it is determined that the person object triggers the shopping behavior. In a case where it is determined that the person object triggers the shopping behavior, a standardized detection record is generated for a shopping behavior trigger event of the person object, the detection record is saved, and a shopping behavior detection result is obtained, wherein the detection record at least includes a unique identity of the person object, a behavior trigger timestamp, a behavior type identifier, and associated multi-modal data is automatically saved, the multi-modal data includes multiple frames of video frames before and after a behavior trigger time, wrist key point coordinate data, a checkout counter interaction area coordinate, and the like, and the multi-modal data is encapsulated in a JSON format.

[0026] In step S14, in a case where the shopping behavior detection result represents that there is a shopping behavior trigger event, a wrist region image is determined according to the wrist key point corresponding to the video frame, and product recognition is performed on the wrist region image through open set object detection, a product recognition event record is output and saved, and a shopping settlement detection result is obtained.

[0027] The wrist region image is a region of a fixed size (such as 200x200 pixels) or a dynamic size (based on the size of the hand) constructed according to the wrist key point, and the image is obtained by cropping the original video frame according to the region, and the open set object detection is a detection technology that only needs to provide a text description or an example image, without the need to retrain a model, and can dynamically identify newly added products.

[0028] In the embodiment of the present disclosure, in a case where the shopping behavior detection result represents that there is a shopping behavior trigger event, the shopping behavior detection result is output through a message queue such as RabbitMQ, a shopping settlement detection module is triggered to start, a wrist region image is determined according to the wrist key point corresponding to the video frame, for example, a 200x200 pixel rectangular region can be constructed with the wrist key point as the center, the size can be adaptively adjusted according to the image resolution, the wrist region image is cropped from the original video frame, and illumination normalization and image enhancement processing are performed, an open set object detection architecture based on a visual-linguistic model is adopted, a backbone network such as ResNet-50 is used to extract features of a to-be-recognized product in the wrist region image, a prompt learning module is used to generate a learnable class prompt word, that is, a candidate product category, and a CLIP (Contrastive Language-Image Pre-Training) text encoder is used to work cooperatively to calculate a similarity matrix of the features of the to-be-recognized product and the features of each candidate product category, the candidate product category with the highest similarity is taken as the product category of the to-be-recognized product, a standardized product recognition event record is generated and saved, and a shopping settlement detection result is obtained.

[0029] The method in the technical solution has the following advantages: the method obtains continuous multiple video frames of a cash desk within a preset range, detects a human body bounding box and a posture key point of a human object in the video frame in which the human object exists, and obtains a posture detection result, wherein the posture key point includes a wrist key point; an upper body bounding box of the human object is determined according to the human body bounding box, and the upper body bounding box is tracked in the continuous multiple video frames to obtain a motion trajectory of the same human object; whether each human object triggers a shopping behavior is determined through trajectory analysis and posture analysis according to the posture detection result and the motion trajectory; if any human object triggers the shopping behavior, a shopping behavior trigger event of the human object is output and saved, a detection record corresponding to the shopping behavior trigger event is output and saved, and a shopping behavior detection result is obtained; in a case where the shopping behavior detection result represents that there is a shopping behavior trigger event, a wrist region image is determined according to the wrist key point corresponding to the video frame, and open set target detection is performed on the wrist region image to identify a commodity, output and save a commodity identification event record, and obtain a shopping settlement detection result. The user shopping behavior is dynamically analyzed according to the human object motion trajectory and the human body posture analysis, which can effectively distinguish the real shopping behavior from irrelevant background activities and improve the detection accuracy. The behavior trigger type commodity detection mechanism is introduced, and the commodity target identification is activated only when the shopping behavior is detected, thereby avoiding blind reasoning on all video frames, reducing the calculation overhead, and improving the real-time response capability of the system. The open set target detection is used, and a large amount of labeled data and repeated training are not required to identify diversified commodity categories, which can adapt to the scene of frequent commodity updates and dynamic changes, can significantly reduce the deployment cost and maintenance cost, and can improve the data management and review efficiency. In summary, the method can improve the accuracy of detection and identification while reducing the cost and improving the efficiency.

[0030] In a possible implementation, in step S14, the commodity identification on the wrist region image through open set target detection, output and save of the commodity identification event record, and obtaining of the shopping settlement detection result include: extracting features of a to-be-identified commodity belonging to a candidate identification region in the wrist region image; obtaining at least one candidate commodity category feature; obtaining a classification logic value by calculating a similarity between the features of the to-be-identified commodity and each candidate commodity category feature; normalizing all the classification logic values to obtain a probability value of the to-be-identified commodity belonging to each candidate commodity category; determining, according to the probability value, the candidate commodity category with the highest probability value as the commodity category of the to-be-identified commodity. Output and save the commodity identification event record including at least the commodity category, and obtain the shopping settlement detection result.

[0031] The candidate identification area is a to-be-detected area filtered through conditional constraints, the candidate commodity category feature is an embedding vector of a preset commodity category, and is generated through three mechanisms of text prompts, visual prompts and prompt-free modes, the classification logic value is a similarity original score of the feature of the to-be-identified commodity and the candidate commodity category feature, and the probability value is a confidence degree of the commodity belonging to each category after normalization.

[0032] In the embodiment of the present disclosure, the wrist area image is input into a YOLOE-v8L-seg model (yoloe-v8l-seg.pt) that introduces a zero-shot learning mechanism during pre-training, and the feature of the to-be-identified commodity is extracted from the wrist area image through a backbone network and a feature pyramid (Neck) to generate a deep feature vector ф(I)∈R d At the same time, the candidate commodity category feature ψ(c)∈R d is obtained through at least one of the three mechanisms of text prompts, visual prompts and prompt-free modes. The text prompts can encode the commodity category name (such as “Coke can” and “chip bag”) into a semantic vector; the visual prompts can generate a visually guided embedding representation through an example commodity image; and the prompt-free mode can automatically retrieve the candidate commodity category from a built-in general commodity feature library. Then, the system calculates the dot product similarity of the deep feature vector ф(I) of the to-be-identified commodity and each candidate commodity category feature ψ(c), obtains a classification logic value, and converts the classification logic value into a probability value P(c∣I) through a mathematical function Softmax normalization processing. The inference process can be represented as: P(c∣I)=Softmax(ф(I)T·ψ(c)); The candidate commodity category with the highest probability value is selected as the commodity category of the to-be-identified commodity. For example, the probability value of A brand cherry Coke is 0.08, the probability value of A brand original Coke is 0.9, and the probability value of B brand original Coke is 0.02. A brand original Coke is determined as the commodity category of the to-be-identified commodity, and a standardized commodity identification event record is generated to obtain the shopping settlement detection result.

[0033] The above technical solution can identify commodity categories that do not appear in the training set, does not need to collect or label new commodity category data for new commodities, adapts to dynamic retail environment, can significantly reduce deployment cost, and has a relatively simple identification process, thereby improving identification efficiency.

[0034] In a possible implementation, in step S14, the feature of the to-be-identified commodity belonging to the candidate identification region in the wrist region image is extracted, including: determining a candidate identification region according to the wrist region and a preset checkout counter interaction region; The candidate identification region satisfies all the following constraints: The ratio of the overlapping area of the candidate identification region and the wrist region to the area of the wrist region satisfies a first preset ratio constraint; The center point of the candidate identification region is located in the preset checkout counter interaction region, and the area ratio of the candidate identification region to the preset checkout counter interaction region satisfies a second preset ratio constraint; The item in the candidate identification region is a to-be-identified commodity determined according to a preset target commodity category set.

[0035] The wrist region is an image region determined according to a wrist key point detected in a human posture and a preset size (such as 200*200 pixels), the preset checkout counter interaction region is a coordinate mapping of a checkout counter physical space in an image, which can be a polygon or a rectangle, and the preset target commodity category set is a retail commodity classification system to be identified, such as 200+ beverage and snack categories.

[0036] In the embodiment of the present disclosure, the overlapping area (IoU) of the candidate identification region and the wrist region is not less than 60% of the area of the wrist region, that is, the overlapping ratio is greater than or equal to 0.6, which is used to ensure that the to-be-identified commodity is located in the hand operation range; the center point of the candidate identification region must be located in the checkout counter interaction region, and the area of the candidate identification region must not exceed 50% of the area of the checkout counter interaction region, which is used to avoid mis-detection of distant commodities or large interference objects; according to the preset target commodity category set, the items in the candidate identification region are classified, and only the candidate identification region belonging to the preset target commodity category set (such as 200+ retail categories such as beverages and snacks) is retained.

[0037] The above technical solution effectively excludes interference targets such as shelf background commodities and handheld non-commodity objects such as mobile phones held by human objects through spatial constraints and semantic constraints, excludes irrelevant candidate identification regions in advance, reduces the detection amount, and reduces the mis-detection rate.

[0038] In a possible implementation, in step S13, the candidate identification region and the wrist region are determined according to the posture detection result and the motion trajectory, through trajectory analysis and posture analysis, whether each human object triggers a shopping behavior, including: According to the motion trajectory, whether the center point of the upper body bounding box in the motion trajectory of the human object is in a preset checkout counter detection region is determined, to determine the stay time of the human object in the preset checkout counter detection region. According to the posture detection result, by determining the positional relationship between the wrist key point and the preset checkout counter interaction region, it is determined whether the character object has an interaction behavior with the checkout counter; If the stay time reaches the preset time threshold and it is determined that the character object has an interaction behavior with the checkout counter, it is determined that the character object triggers a shopping behavior.

[0039] Among them, the preset checkout counter detection region is a preset rectangle or polygon region, such as a 1.5x2m 2 range, used to define the effective trajectory analysis space.

[0040] In the embodiments of the present disclosure, according to the motion trajectory, it is calculated in real time whether the center point of the upper body bounding box of the character object is located in the preset checkout counter detection region, such as a 1.5m x 2m rectangular region in front of the checkout counter, and the continuous stay time is accumulated by a timing module; the wrist key point coordinates of the character object are extracted, combined with the boundary line of the preset checkout counter interaction region, for example, a virtual line segment with the checkout counter code scanner as the reference, the positional relationship between the wrist key point and the preset checkout counter interaction region is determined by calculating the spatial geometric relationship: if the scalar value is greater than the preset scalar threshold and the moving direction of the wrist key point is consistent with the preset interaction direction vector, such as moving from the side of the character object to the side of the cashier, it is determined that the interaction behavior occurs. When the same character object simultaneously satisfies the stay time not less than the preset time threshold (such as 3 seconds) and the interaction behavior exists, it is determined that the character object triggers a shopping behavior.

[0041] The above technical solution combines the stay time and the posture interaction double determination, which can significantly reduce the false detection rate, improve the accuracy and reliability of shopping behavior detection, and further improve the accuracy and reliability of shopping settlement fraud risk behavior detection.

[0042] In a possible implementation, in step S13, the determining, according to the posture detection result, whether the character object has an interaction behavior with the checkout counter by determining the positional relationship between the wrist key point and the preset checkout counter interaction region, comprises: determining a boundary point according to the preset checkout counter interaction region; calculating a scalar value for determining the positional relationship according to the coordinates of the boundary point and the wrist key point; If the scalar value satisfies the preset scalar threshold constraint, and the moving direction of the wrist key point relative to the boundary line of the preset checkout counter interaction region is consistent with the preset interaction direction, it is determined that the wrist key point crosses the boundary line, and it is confirmed that the interaction behavior occurs.

[0043] The boundary point is a point on a boundary line of a preset cash register interaction region, the scalar value is a numerical value determined through geometric operation (such as cross product) and represents the orientation relationship between the wrist key point and the boundary line, the moving direction is a displacement vector of the wrist key point in consecutive frames, and the preset interaction direction is a spatial motion vector conforming to the checkout operation in the shopping behavior.

[0044] In the embodiment of the present disclosure, according to the boundary line of the preset cash register interaction region, two boundary points P1 (x1, y1) and P2 (x2, y2) are determined, the wrist key point coordinate is P (x, y), the scalar value is Value, and the calculation formula of the scalar value can be: w (x w -y w )·(x w -x1)-(x2-x1)(y w -y1). The above technical solution realizes interaction behavior recognition through spatial geometric calculation, has fast calculation speed and high accuracy, can adapt to different cash register physical sizes, has strong generalization ability and deployment flexibility.

[0045] In a possible implementation, in step S13, if any of the person objects triggers the shopping behavior, a detection record corresponding to a shopping behavior triggering event of the person object is output and saved, and the detection record includes: In response to determining that the person object triggers the shopping behavior, a unique identity of the person object is acquired; According to the unique identity, at least one preceding video frame, an event triggering video frame, and at least one subsequent video frame corresponding to the moment when the shopping behavior triggering event occurs are acquired; The detection record including the acquired unique identity, the video frames, and metadata related to the shopping behavior triggering event is output and saved, and the metadata at least includes a timestamp and coordinates of the wrist key point.

[0046] The unique identity is a number used to uniquely identify and distinguish different person objects, and the metadata is structured information describing a behavior event.

[0047] ​In the embodiments of the present disclosure, when the shopping behavior of the character object is determined to exist, the unique identity of the character object is immediately obtained, and a plurality of video frames within a preset time based on the shopping behavior triggering event moment are extracted from the continuous multiple video frames, including a plurality of preceding video frames, an event triggering video frame, and a plurality of subsequent video frames after triggering, for example, 3 preceding video frames before triggering, 1 event triggering video frame, and 3 subsequent video frames after triggering, wherein the plurality of video frames within the preset time based on the shopping behavior triggering event moment can be dynamically adjusted according to the video frame rate, such as 30 fps / 60 fps; at the same time, a standardized detection record containing the unique identity, the preceding video frame, the event triggering video frame, the subsequent video frame, and related metadata is generated, wherein the metadata at least includes wrist key point coordinates and a timestamp, and finally the detection record can be output in a standardized format and stored in a distributed database and an object storage system.

[0048] The above technical solution realizes complete digital recording of the shopping behavior triggering event, guarantees data integrity, supports real-time processing in a high-concurrency scenario, and provides a standardized data basis for subsequent behavior analysis and system optimization.

[0049] In a possible implementation manner, in step S13, the upper body bounding box for the character object is determined according to the human body bounding box, and the upper body bounding box is tracked in the continuous multiple video frames to obtain a motion trajectory of the same character object, including: All the upper body bounding boxes in the current video frame are obtained; In the embodiments of the present disclosure, the current video frame is inferred in real time, and all visible upper body bounding boxes are detected, each detection result containing upper body bounding box coordinates (for example, center point x, y, width w, and height h) confidence score, and class label (fixed as upper_body); The trajectory positions of all the upper body bounding boxes in the current video frame are predicted according to the trajectory states of all the upper body bounding boxes in the previous video frame; In the embodiments of the present disclosure, according to the trajectory states of all the upper body bounding boxes in the previous video frame, a Kalman filter can be used to predict the trajectory positions: an 8-dimensional state vector is established: [x, y, w, h, vx, vy, vw, vh], a constant velocity model is used as a system dynamics model to predict the trajectory positions and sizes of the upper body bounding boxes of each trajectory in the current video frame, and the Mahalanobis distance between the predicted upper body bounding box and the detected upper body bounding box is calculated as a motion similarity measure. x y w h ​​​​ matching the upper body bounding box of the current video frame with the predicted trajectory position, determining the matched upper body bounding box and the predicted trajectory position; updating the motion trajectory according to the matched upper body bounding box and the predicted trajectory position, and obtaining the trajectory tracking result of the current video frame.

[0050] In the embodiments of the present disclosure, the Hungarian algorithm can be used for optimal matching, the intersection over union (IOU) between the upper body bounding box of the current video frame and the predicted trajectory position can be taken as the main association index, a minimum IOU threshold, for example, 0.3, can be set, and the matching pairs lower than the minimum IOU threshold can be removed, and then the matching can be performed in two stages: the upper body bounding box of the current video frame is associated with the predicted trajectory position in the first round of association, the high-confidence upper body bounding box with a confidence higher than a first threshold is used in this round of association, and the matching is performed based on the motion similarity between the upper body bounding box and the predicted trajectory position, i.e., the predicted upper body bounding box; for the trajectory position and the high-confidence upper body bounding box that are not matched in the first round of association, the second round of association is performed between the low-confidence upper body bounding box with a confidence lower than the first threshold but higher than a second threshold, and the matching is also performed based on the motion similarity; the high-confidence detection box that is still not matched after the second round of association is reserved as the initial value of the new trajectory, and all the low-confidence detection boxes that are not matched are discarded. updating the motion trajectory according to the matched upper body bounding box and the predicted trajectory position, updating the state of all matched trajectories, and outputting the trajectory tracking result of the current frame.

[0051] The above technical solution can effectively and accurately realize the association of the upper body bounding box by tracking the motion trajectory of the upper body bounding box, and improve the continuity and accuracy of the shopping behavior detection result.

[0052] It should be understood that, although Figure 1 The steps in the flowchart of the method can be displayed in sequence according to the arrows, but these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 1 At least part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0053] In one embodiment, as Figure 2As shown, a device for detecting shopping settlement fraud risk behavior is provided, comprising: The posture detection module 210 is configured to acquire a plurality of continuous video frames of the checkout counter within a preset range, and for the video frames in which a human object exists, detect a human body bounding box and posture key points of the human object in the video frames to obtain a posture detection result, wherein the posture key points include wrist key points; The trajectory tracking module 220 is configured to determine an upper body bounding box for the human object according to the human body bounding box, and track the upper body bounding box in the plurality of continuous video frames to obtain a motion trajectory for the same human object; The shopping behavior detection module 230 is configured to determine whether each human object triggers a shopping behavior through trajectory analysis and posture analysis according to the posture detection result and the motion trajectory, and if any human object triggers the shopping behavior, output and save a detection record corresponding to a shopping behavior trigger event of the human object to obtain a shopping behavior detection result; The shopping settlement detection module 240 is configured to, in a case where the shopping behavior detection result represents that there is a shopping behavior trigger event, determine a wrist region image according to the wrist key points corresponding to the video frames, and perform product identification on the wrist region image through open set target detection to output and save a product identification event record, thereby obtaining a shopping settlement detection result.

[0054] In a possible implementation, the shopping settlement detection module 240 is configured to: extract features of a to-be-identified product belonging to a candidate identification region in the wrist region image; obtain at least one candidate product category feature; obtain a classification logic value by calculating the similarity between the features of the to-be-identified product and each candidate product category feature; normalize all the classification logic values to obtain a probability value of the to-be-identified product belonging to each candidate product category; determine the candidate product category with the highest probability value as the product category of the to-be-identified product according to the probability value; output and save a product identification event record including at least the product category to obtain the shopping settlement detection result.

[0055] In a possible implementation, the shopping settlement detection module 240 is configured to: determine a candidate identification region according to the wrist region and a preset checkout counter interaction region; The candidate identification region satisfies all the following constraints: A ratio of an area of an overlapping region of the candidate recognition region and the wrist region to an area of the wrist region satisfies a first preset ratio constraint; A center point of the candidate recognition region is located within a preset cash register interaction region, and an area ratio of the candidate recognition region to the preset cash register interaction region satisfies a second preset ratio constraint; An item in the candidate recognition region is a to-be-recognized commodity determined according to a preset target commodity category set.

[0056] In a possible implementation, the shopping behavior detection module 230 is configured to: According to the motion trajectory, by determining whether a center point of the upper body bounding box in the motion trajectory of the character object is within a preset cash register detection region, a stay time of the character object within the preset cash register detection region is determined. According to the posture detection result, by determining a positional relationship between the wrist key point and the preset cash register interaction region, it is determined whether the character object has an interaction behavior with the cash register. If the stay time reaches a preset time threshold and it is determined that the character object has an interaction behavior with the cash register, it is determined that the character object triggers a shopping behavior.

[0057] In a possible implementation, the shopping behavior detection module 230 is configured to: According to the preset cash register interaction region, a boundary point is determined. According to the coordinates of the boundary point and the wrist key point, a scalar value for determining the positional relationship is calculated. If the scalar value satisfies a preset scalar threshold constraint, and a moving direction of the wrist key point relative to a boundary line of the preset cash register interaction region is consistent with a preset interaction direction, it is determined that the wrist key point crosses the boundary line, and it is confirmed that the interaction behavior occurs.

[0058] In a possible implementation, the shopping behavior detection module 230 is configured to: In response to determining that the character object triggers the shopping behavior, a unique identity of the character object is acquired. According to the unique identity, at least one preceding video frame, an event trigger video frame, and at least one subsequent video frame corresponding to a time point when the shopping behavior trigger event occurs are acquired. The detection record including the acquired unique identity, the video frames, and metadata related to the shopping behavior trigger event is output and saved, wherein the metadata at least includes a timestamp and coordinates of the wrist key point.

[0059] In a possible implementation, the trajectory tracking module 220 is configured to: obtain all the upper body bounding boxes in a current video frame; predict trajectory positions of all the upper body bounding boxes in the current video frame according to trajectory states of all the upper body bounding boxes in a previous video frame; match the upper body bounding boxes in the current video frame with the predicted trajectory positions, to determine the upper body bounding boxes and the predicted trajectory positions that are matched successfully; update the motion trajectory according to the upper body bounding boxes and the predicted trajectory positions that are matched successfully, to obtain a trajectory tracking result of the current video frame.

[0060] In a possible implementation, the apparatus further includes a recording tool module configured to: save and manage detection records of shopping behavior triggering events and records of commodity identification events, to provide complete basis for subsequent manual review.

[0061] Specific limitations of the apparatus for detecting shopping settlement fraud risk behavior can be seen from the limitations of the method for detecting shopping settlement fraud risk behavior, which will not be repeated here. Each module in the apparatus for detecting shopping settlement fraud risk behavior can be realized by software, hardware, or a combination thereof, in whole or in part. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0062] The embodiments of the present disclosure further provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the steps of the method in any of the foregoing embodiments.

[0063] The embodiments of the present disclosure further provide an electronic device, comprising: a memory having stored thereon a computer program; a processor configured to execute the computer program in the memory to implement the steps of the method in any of the foregoing embodiments.

[0064] Figure 3The illustrated detection shopping settlement fraud risk behavior apparatus 100 comprises a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, through a bus 1002. Optionally, the detection shopping settlement fraud risk behavior apparatus 100 can further comprise a communication component, which can be used for data interaction between the apparatus 100 and other devices, such as data sending or data receiving. It should be noted that the communication component is not limited to one in actual scheduling, and the structure of the detection shopping settlement fraud risk behavior apparatus 100 does not constitute a limitation to the embodiments of the present application.

[0065] The processor 1001 can be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, digital signal processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 1001 can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc.

[0066] The bus 1002 can comprise a channel for transmitting information between the above-mentioned components. The bus 1002 can be a PCI (Peripheral Component Interconnect, peripheral component interconnect) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus 1002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 3 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0067] The memory 1003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing program code, which can be read by a computer, without limitation.

[0068] The memory 1003 is configured to store program codes for implementing the embodiments of the present disclosure, and the processor 1001 is configured to control the execution of the program codes stored in the memory 1003. The processor 1001 is configured to execute the program codes stored in the memory 1003 to implement the steps of the above-mentioned method for detecting fraud risk behaviors in shopping settlement.

[0069] The embodiments of the present disclosure also provide a computer readable storage medium, which stores program codes. The program codes are executed by a processor to implement the steps of the above-mentioned method for detecting fraud risk behaviors in shopping settlement.

[0070] The preferred embodiments of the present disclosure are described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the specific details of the above-described embodiments. Within the technical concept of the present disclosure, various changes, modifications, replacements and variations can be made to the embodiments, and all these changes, modifications, replacements and variations shall be considered as the disclosed content of the present disclosure. In order to avoid unnecessary repetition, the present disclosure will not make further description on various possible combinations. The technical scope of the present disclosure shall not be limited to the content of the specification, but shall be determined by the scope of the claims.

[0071] In addition, it should be noted that each specific technical feature described in the above specific embodiments can be combined in any appropriate manner without contradiction, and shall be considered as the disclosed content of the present disclosure. In order to avoid unnecessary repetition, the present disclosure will not make further description on various possible combinations. The technical scope of the present disclosure shall not be limited to the content of the specification, but shall be determined by the scope of the claims.

Claims

1. A method for detecting fraudulent behavior in shopping settlement, characterized in that: The method comprises: Acquire a plurality of consecutive video frames of the cash register within a preset range, and for the video frames in which a person object exists, detect the body bounding box and posture key points of the person object in the video frames to obtain a posture detection result, wherein the posture key points include wrist key points; determining an upper body bounding box for a human object according to the human body bounding box, and tracking the upper body bounding box in the plurality of consecutive video frames to obtain a motion trajectory for the same human object; Based on the posture detection result and the motion trajectory, determining whether each of the human objects triggers a shopping behavior through trajectory analysis and posture analysis; if any of the human objects triggers the shopping behavior, outputting and saving a detection record corresponding to the shopping behavior triggering event for the human object, thereby obtaining a shopping behavior detection result; When the shopping behavior detection result indicates the presence of a shopping behavior triggering event, a wrist area image is determined based on the wrist key points corresponding to the video frame, and product identification is performed on the wrist area image through open set target detection. The product identification event record is output and saved to obtain a shopping settlement detection result.

2. The method according to claim 1, characterized in that The method of performing commodity recognition on the wrist area image through open set target detection, outputting and saving commodity recognition event records, and obtaining shopping settlement detection results includes: Extracting features of the commodity to be identified that belongs to the candidate identification area in the wrist area image; Obtain at least one candidate product category feature; Obtaining a classification logic value by calculating the similarity between the feature of the product to be identified and the feature of each candidate product category; Normalizing all the classification logic values ​​to obtain probability values ​​of the to-be-identified product belonging to each candidate product category; According to the probability values, determining the candidate commodity category with the highest probability value as the commodity category of the commodity to be identified; Output and save the commodity identification event record including at least the commodity category to obtain the shopping settlement detection result.

3. The method according to claim 2, characterized in that The extracting features of the commodity to be identified belonging to the candidate identification area in the wrist area image includes: Determining a candidate recognition area based on the wrist area and a preset cashier counter interaction area; The candidate recognition area satisfies all of the following constraints: The ratio of the overlapping area between the candidate recognition area and the wrist area to the area of ​​the wrist area satisfies a first preset ratio constraint; The center point of the candidate recognition area is located within the preset cashier interaction area, and the area ratio of the candidate recognition area to the preset cashier interaction area satisfies a second preset ratio constraint; The items in the candidate identification area are commodities to be identified that are determined according to a preset target commodity category set.

4. The method according to claim 1, characterized in that The determining, based on the posture detection result and the motion trajectory, whether each of the human objects triggers a shopping behavior through trajectory analysis and posture analysis includes: Determining, based on the motion trajectory, whether a center point of an upper body bounding box in the motion trajectory of the human object is within a preset checkout counter detection area, and determining a residence time of the human object in the preset checkout counter detection area; Determining whether the person object has an interactive behavior with the cashier counter by determining the positional relationship between the wrist key point and a preset cashier counter interaction area according to the posture detection result; If the stay time reaches a preset time threshold and it is determined that the character object has an interactive behavior with the checkout counter, it is determined that the character object triggers a shopping behavior.

5. The method according to claim 4, characterized in that: The determining, based on the posture detection result, whether the person object has an interactive behavior with the cashier counter by determining a positional relationship between the wrist key point and a preset cashier counter interaction area includes: Determining boundary points according to the preset cashier interaction area; Calculating a scalar value for determining a positional relationship according to the coordinates of the boundary point and the wrist key point; If the scalar value satisfies a preset scalar threshold constraint, and the movement direction of the wrist key point relative to the boundary line of the preset checkout counter interaction area is consistent with a preset interaction direction, it is determined that the wrist key point has crossed the boundary line, and an interaction behavior is confirmed to have occurred.

6. The method according to claim 1, characterized in that If any of the character objects triggers the shopping behavior, then for the shopping behavior triggering event of the character object, outputting and saving a detection record corresponding to the shopping behavior triggering event includes: In response to determining that the character object triggers the shopping behavior, obtaining a unique identity identifier of the character object; According to the unique identity, obtaining corresponding at least one preceding video frame, an event triggering video frame, and at least one subsequent video frame based on the shopping behavior triggering event moment; Output and save the detection record including the acquired unique identification, the video frame, and metadata related to the shopping behavior triggering event, wherein the metadata at least includes a timestamp and coordinates of the wrist key point.

7. The method according to any one of claims 1 to 6, characterized in that The step of determining an upper body bounding box for a human object based on the human body bounding box, and tracking the upper body bounding box in the plurality of consecutive video frames to obtain a motion trajectory for the same human object includes: Get all the upper body bounding boxes in the current video frame; Predicting the trajectory positions of all the upper body bounding boxes in the current video frame based on the trajectory states of all the upper body bounding boxes in the previous video frame; Matching the upper body bounding box of the current video frame with the predicted trajectory position, and determining a successfully matched upper body bounding box and the predicted trajectory position; According to the successfully matched upper body bounding box and the predicted trajectory position, the motion trajectory is updated to obtain the trajectory tracking result of the current video frame.

8. A device for detecting fraudulent behavior in shopping settlement, characterized in that: The device comprises: a posture detection module configured to acquire a plurality of consecutive video frames of the cash register within a preset range, and for the video frames in which a person object exists, detect a body bounding box and posture key points of the person object in the video frames to obtain a posture detection result, wherein the posture key points include a wrist key point; a trajectory tracking module configured to determine an upper body bounding box for a human object based on the human body bounding box, and track the upper body bounding box in the plurality of consecutive video frames to obtain a motion trajectory for the same human object; a shopping behavior detection module configured to determine, based on the posture detection result and the motion trajectory, whether each of the human objects triggers a shopping behavior through trajectory analysis and posture analysis; if any of the human objects triggers the shopping behavior, output and save a detection record corresponding to the shopping behavior triggering event for the human object, thereby obtaining a shopping behavior detection result; The shopping settlement detection module is configured to, when the shopping behavior detection result indicates the presence of a shopping behavior triggering event, determine a wrist area image based on wrist key points corresponding to the video frame, perform product recognition on the wrist area image through open set object detection, output and save a product recognition event record, and obtain a shopping settlement detection result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

Images