Behavior detection method, apparatus and device

Abstract

This application provides a behavior detection method, apparatus, and device. The method includes: an electronic device acquiring a real-time video stream sent by a camera module and extracting user feature information from the real-time video stream; determining a candidate user group whose feature information meets a first preset condition, and determining a candidate partial image corresponding to the candidate user group; determining a target category corresponding to the candidate partial image based on the candidate partial image and a target recognition model; determining a first user and a second user in the real-time video stream according to the feature information; if the target category is the first category, and the candidate user group includes the first user and the second user, then determining the target purchasing behavior corresponding to the candidate partial image. In this way, the electronic device can automatically detect purchasing behavior based on real-time video, which is more efficient, eliminates the need for manual spot checks, reduces labor costs, and improves the accuracy of purchasing behavior detection, thereby effectively reducing the risk of unauthorized transactions.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a behavior detection method, apparatus, and device. Background Technology

[0002] In offline shopping scenarios such as shopping malls and department stores, there is usually a positive correlation between mall counter rental fees and sales volume. Therefore, irregularly entered orders can lead to financial losses for the mall.

[0003] In related technologies, to reduce the risk of unauthorized sales, shopping malls (or brand operators, etc.) typically employ professionals to conduct random checks to minimize such behavior. However, this method is labor-intensive, and the efficiency and accuracy of manual checks are both low, making it ineffective in reducing the risk of unauthorized sales. Summary of the Invention

[0004] This application provides a behavior detection method, apparatus, and device to automatically identify customer purchasing behavior, improve the efficiency and accuracy of purchasing behavior detection, and thus effectively reduce the risk of unauthorized transactions.

[0005] In a first aspect, embodiments of this application provide a behavior detection method, including:

[0006] Acquire the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream;

[0007] Determine candidate user groups whose feature information satisfies the first preset condition, and determine candidate local images corresponding to the candidate user groups;

[0008] Based on the candidate local images and the target recognition model, the target category corresponding to the candidate local images is determined;

[0009] Based on the feature information, the first user and the second user in the real-time video stream are determined;

[0010] If the target category is the first category, and the candidate user group includes the first user and the second user, then the candidate local image corresponds to the target purchase behavior.

[0011] In one possible implementation, determining the candidate user group whose feature information satisfies a first preset condition includes:

[0012] Based on the feature information, the user's location information is determined;

[0013] If the location information of two users meets the preset location conditions in N consecutive image frames, then the two users are determined as the candidate user group; where N is an integer greater than 1.

[0014] In one possible implementation, determining the candidate user group whose feature information satisfies a first preset condition includes:

[0015] Based on the feature information, determine the user's coordinate information and the user's orientation angle information;

[0016] If, in N consecutive image frames, the coordinate information and orientation angle information of two users satisfy the preset position conditions, then the two users are identified as the candidate user group.

[0017] In one possible implementation, the method further includes:

[0018] Based on the real-time video stream, determine the target image region in the real-time video stream;

[0019] For the target image region, the candidate user group is detected and determined according to the first frame rate;

[0020] For image regions other than the target image region, the candidate user group is detected and determined according to a second frame rate; the first frame rate is greater than the second frame rate.

[0021] In one possible implementation, the target recognition model includes a target feature extraction model and a target classification model; determining the target category corresponding to the candidate local image based on the candidate local image and the target recognition model includes:

[0022] In the candidate local images, the user's hand detection box and the target bag detection box are identified and labeled to obtain the image to be identified;

[0023] The image to be identified is input into the target feature extraction model to determine the image features corresponding to the image to be identified, and at the same time, the heat map features corresponding to the image to be identified are determined; the heat map features are centered on the human hand detection frame and the target bag detection frame;

[0024] The image features and the heatmap features are fused to obtain the fused features;

[0025] The fused features are input into the target classification model to obtain the target category corresponding to the candidate local image.

[0026] In one possible implementation, the method further includes:

[0027] Obtain training samples; the training samples include images of bag-handling behavior labeled with hand detection boxes and bag detection boxes;

[0028] The training samples are input into a preset recognition model to determine the training category corresponding to the training samples;

[0029] The preset recognition model is iteratively trained according to the training category and the preset loss function to obtain the target recognition model.

[0030] In one possible implementation, determining the first user and the second user in the real-time video stream based on the feature information includes:

[0031] Based on the feature information, the same user is correlated and tracked in the real-time video stream to determine the user's motion trajectory;

[0032] If the motion trajectory meets the second preset condition, the user is identified as the first user;

[0033] If the motion trajectory does not meet the second preset condition, the user is identified as the second user.

[0034] In one possible implementation, the second preset condition includes:

[0035] The start and / or end times of the motion trajectory are included in a preset time period; and / or,

[0036] The duration of the motion trajectory is greater than a preset time threshold.

[0037] In one possible implementation, after determining the target category corresponding to the candidate local image, the method further includes:

[0038] If the target category is the first category, then for the candidate user group corresponding to the first category, determine the association between the user and the target bag;

[0039] If the association between the user and the target bag changes, then the target category corresponding to the candidate local image is determined to be correctly identified.

[0040] Secondly, embodiments of this application provide a behavior detection device, comprising:

[0041] An extraction module is used to acquire the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream;

[0042] The first determining module is used to determine the candidate user group whose feature information satisfies the first preset condition, and to determine the candidate local image corresponding to the candidate user group;

[0043] The second determining module is used to determine the target category corresponding to the candidate local image based on the candidate local image and the target recognition model;

[0044] The third determining module is used to determine the first user and the second user in the real-time video stream based on the feature information.

[0045] The fourth determining module is used to determine the target purchase behavior corresponding to the candidate local image if the target category is the first category and the candidate user group includes the first user and the second user.

[0046] In one possible implementation, the first determining module is specifically used for:

[0047] Based on the feature information, the user's location information is determined;

[0048] If the location information of two users meets the preset location conditions in N consecutive image frames, then the two users are determined as the candidate user group; where N is an integer greater than 1.

[0049] In one possible implementation, the first determining module is specifically used for:

[0050] Based on the feature information, determine the user's coordinate information and the user's orientation angle information;

[0051] If, in N consecutive image frames, the coordinate information and orientation angle information of two users satisfy the preset position conditions, then the two users are identified as the candidate user group.

[0052] In one possible implementation, the device is further used for:

[0053] Based on the real-time video stream, determine the target image region in the real-time video stream;

[0054] For the target image region, the candidate user group is detected and determined according to the first frame rate;

[0055] For image regions other than the target image region, the candidate user group is detected and determined according to a second frame rate; the first frame rate is greater than the second frame rate.

[0056] In one possible implementation, the target recognition model includes a target feature extraction model and a target classification model; the second determining module is specifically used for:

[0057] In the candidate local images, the user's hand detection box and the target bag detection box are identified and labeled to obtain the image to be identified;

[0058] The image to be identified is input into the target feature extraction model to determine the image features corresponding to the image to be identified, and at the same time, the heat map features corresponding to the image to be identified are determined; the heat map features are centered on the human hand detection frame and the target bag detection frame;

[0059] The image features and the heatmap features are fused to obtain the fused features;

[0060] The fused features are input into the target classification model to obtain the target category corresponding to the candidate local image.

[0061] In one possible implementation, the device is further used for:

[0062] Obtain training samples; the training samples include images of bag-handling behavior labeled with hand detection boxes and bag detection boxes;

[0063] The training samples are input into a preset recognition model to determine the training category corresponding to the training samples;

[0064] The preset recognition model is iteratively trained according to the training category and the preset loss function to obtain the target recognition model.

[0065] In one possible implementation, the third determining module is specifically used for:

[0066] Based on the feature information, the same user is correlated and tracked in the real-time video stream to determine the user's motion trajectory;

[0067] If the motion trajectory meets the second preset condition, the user is identified as the first user;

[0068] If the motion trajectory does not meet the second preset condition, the user is identified as the second user.

[0069] In one possible implementation, the second preset condition includes:

[0070] The start and / or end times of the motion trajectory are included in a preset time period; and / or,

[0071] The duration of the motion trajectory is greater than a preset time threshold.

[0072] In one possible implementation, the device is further used for:

[0073] If the target category is the first category, then for the candidate user group corresponding to the first category, determine the association between the user and the target bag;

[0074] If the association between the user and the target bag changes, then the target category corresponding to the candidate local image is determined to be correctly identified.

[0075] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0076] The memory stores computer-executed instructions;

[0077] The processor executes computer execution instructions stored in the memory, causing the processor to perform the behavior detection method described in either the first or second aspect.

[0078] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the behavior detection method described in either the first or second aspect.

[0079] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the behavior detection method shown in either the first or second aspect.

[0080] In this embodiment, the electronic device acquires a real-time video stream sent by a camera module and extracts user feature information from the real-time video stream; determines a candidate user group whose feature information meets a first preset condition, and determines a candidate partial image corresponding to the candidate user group; based on the candidate partial image and a target recognition model, determines the target category corresponding to the candidate partial image; and determines a first user and a second user in the real-time video stream according to the feature information. If the target category is the first category, and the candidate user group includes both the first user and the second user, then the target purchase behavior corresponding to the candidate partial image is determined. In this embodiment, the electronic device performs a preliminary screening of bag-handling behavior based on the feature information extracted from the real-time video to determine candidate partial images, and then accurately identifies the candidate partial images to determine whether the target category corresponding to the candidate partial image is bag-handling behavior; simultaneously, the electronic device can identify the first user (salesperson) and the second user (customer) based on the feature information. When the target category corresponding to the candidate partial image is bag-handling behavior and the candidate user group includes both salesperson and customer, the electronic device determines that a purchase behavior has occurred. In this way, electronic devices can automatically detect purchasing behavior based on real-time video, which is more efficient and eliminates the need for manual spot checks, thus reducing labor costs. At the same time, by recognizing bag delivery behavior and identifying store clerks and customers, the accuracy of purchasing behavior detection can be improved. Subsequently, purchasing behavior can be used to accurately identify unauthorized orders, thereby effectively reducing the risk of unauthorized orders. Attached Figure Description

[0081] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0082] Figure 1 A schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application;

[0083] Figure 2 A flowchart illustrating an exemplary embodiment of this application for behavior detection method;

[0084] Figure 3 A flowchart illustrating another behavior detection method provided for an exemplary embodiment of this application;

[0085] Figure 4 A schematic diagram illustrating user orientation relationship calculation as an exemplary embodiment of this application;

[0086] Figure 5 A schematic diagram of an algorithm for recognizing bag-handling behavior provided as an exemplary embodiment of this application;

[0087] Figure 6 A schematic diagram of a technology chain for behavior detection provided for an exemplary embodiment of this application;

[0088] Figure 7 A schematic diagram of the structure of a behavior detection device provided for an exemplary embodiment of this application;

[0089] Figure 8 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0091] As consumer spending power continues to rise, the number of offline shopping malls is constantly expanding. The main source of revenue for department stores is the rent from their individual counters. Generally, rent and counter sales are positively correlated. However, if unauthorized sales occur, counters can underreport sales figures, causing financial losses to the mall. Unauthorized sales refer to situations where sales staff (or "shop assistants," "sales personnel," etc.) fail to enter sales orders into the mall's designated sales system or use third-party sales equipment without mall permission, providing false sales data and resulting in losses for the mall. Furthermore, malls frequently run promotional activities such as points and rebates, allowing sales staff to record sales during these promotions and profit from the difference. Additionally, if product quality issues arise, the lack of recorded sales can lead to risks related to after-sales issues between the mall and the counter.

[0092] In order to reduce the risk of unauthorized transactions, shopping malls often conduct random checks by their staff. However, this method is labor-intensive and inefficient. Furthermore, due to the lack of comprehensiveness and accuracy of the checks, it cannot effectively reduce the risk of unauthorized transactions.

[0093] In this embodiment, the premise of identifying "fly-by-night" sales risks is the accurate identification of purchasing behavior. In the purchasing scenario of offline shopping malls and department stores, after a customer purchases goods, the salesperson will organize the goods and place them in their own brand's shopping bag (target bag) and hand it to the customer. Based on this, electronic devices can identify the actions of the salesperson and the customer handing the bag based on real-time video streams captured by cameras and visual algorithms. This can accurately locate the purchasing behavior, and then, based on the comparison between the purchasing behavior and sales records, the "fly-by-night" behavior can be effectively identified, reducing the risk of "fly-by-night" sales. At the same time, it eliminates the need for manual spot checks, improving the efficiency of identifying "fly-by-night" behavior and reducing labor costs.

[0094] Figure 1 This is a schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application. For example... Figure 1 As shown, this includes mall staff member 101 and electronic device 102. Electronic device 102 can be a mobile phone, computer, or other similar device. Figure 1 As shown, in related technologies, to reduce the risk of unauthorized transactions, shopping mall staff typically conduct random checks to confirm such transactions. This method is inefficient, has high labor costs, and cannot accurately identify a large number of unauthorized transactions, thus failing to effectively reduce the risk.

[0095] In this embodiment, the electronic device 102 acquires a real-time video stream and detects the bag-handling behavior in the video stream using a visual algorithm to determine the purchasing behavior between the store clerk and the customer. This purchasing behavior can then be used to determine any instances of unauthorized order fulfillment. In this way, the electronic device 102 can automatically and accurately identify purchasing and unauthorized order fulfillment without the need for manual checks, thus improving the efficiency of determining such behavior and reducing the risk of unauthorized order fulfillment.

[0096] The technical solutions shown in this application will be described in detail below through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other, and the same or similar content will not be described again in different embodiments.

[0097] Figure 2 This is a flowchart illustrating a behavior detection method provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 2 The behavior detection method may include:

[0098] S201. Obtain the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream.

[0099] The execution subject in this application embodiment can be an electronic device or a behavior detection device installed in an electronic device. The behavior detection device can be implemented by software or by a combination of software and hardware. For ease of understanding, the following description will use an electronic device as the execution subject.

[0100] In this embodiment, the camera module can refer to a camera or similar device used for video capture, and may include at least one camera, such as cameras positioned at different locations. The real-time video stream can refer to the real-time video stream captured and uploaded by the camera module. The user can refer to any person included in the real-time video stream, specifically including customers and shop assistants. Feature information can refer to the user's human body feature information, specifically including key point information and re-identification (Reid) features.

[0101] In this step, the electronic device communicates with the camera module, which can be a wired connection or a wireless connection; this embodiment of the application does not limit this. The electronic device can acquire the real-time video stream sent by the camera module and perform detection and recognition on the real-time video stream, extracting the feature information of each user in the real-time video stream. For example, the electronic device can detect the user's human body bounding box in each image frame of the real-time video stream based on multi-object tracking algorithms, such as the real-time online tracking algorithm DeepSort and multi-object tracking networks (Joint Detection and Embedding, JDE), and extract the feature information of the human body bounding box.

[0102] S202. Determine the candidate user group whose feature information meets the first preset condition, and determine the candidate local image corresponding to the candidate user group.

[0103] In this embodiment, the first preset condition can refer to a pre-set bag-passing behavior screening condition, specifically, it can refer to the location information of two users satisfying preset location conditions in N consecutive image frames. The candidate user group can refer to two users who may exhibit bag-passing behavior. The candidate local image can refer to a sequence of local images corresponding to the candidate user group that may exhibit bag-passing behavior.

[0104] In this step, after extracting the feature information of each user from the real-time video stream, the electronic device can perform preliminary screening of bag-handling behavior to reduce computational load and save computing resources. Since store clerks and customers are usually close together and face-to-face when handing over bags, the electronic device can determine the location information of each user based on the feature information, such as foot coordinates and orientation angle. Then, two users whose location information meets preset conditions (similar foot coordinates, face-to-face orientation, etc.) are identified as candidate user groups. Subsequently, the electronic device can use image cropping algorithms to crop and save the corresponding local images from the image frames of the real-time video stream, obtaining candidate local images for the candidate user groups, which serve as candidate sequences for bag-handling behavior detection and recognition. In this way, by performing preliminary screening of users engaging in bag-handling behavior in the real-time video stream, the electronic device can filter out users who do not belong to the bag-handling behavior category, reducing the computational load for subsequent bag-handling behavior recognition, saving system computing resources, and improving the accuracy of bag-handling behavior detection.

[0105] S203. Based on the candidate local images and the target recognition model, determine the target category corresponding to the candidate local images.

[0106] In this embodiment, the target recognition model can refer to a pre-trained bag-handling behavior recognition model. Specifically, this target recognition model can include a target feature extraction model and a target classification model, capable of extracting and classifying image features from candidate local images. The target category can refer to the category corresponding to the candidate local image, which can include a first category and a second category. The first category can be a candidate local image that includes bag-handling behavior, and the second category can be a candidate local image that does not include bag-handling behavior. Specifically, after determining the candidate local images corresponding to the candidate user group, the electronic device can input the candidate local images into the target recognition model for detection and recognition. The target recognition model can output the target category corresponding to the candidate local image.

[0107] S204. Based on the feature information, determine the first user and the second user in the real-time video stream.

[0108] In this embodiment, the first user can refer to a store clerk (salesperson, sales assistant, etc.) in the real-time video stream. The second user can refer to a customer (consumer, pedestrian, etc.) in the real-time video stream. In actual offline scenarios, customers walking together may pass bags to each other, but this behavior cannot be definitively identified as a purchase. To improve the accuracy of purchase determination, the electronic device can identify each user in the real-time video stream based on feature information, i.e., determine whether a user in the real-time video stream is the first user or the second user. Specifically, the electronic device can use a multi-target tracking algorithm to determine the behavioral trajectory of each user, and then determine the first user based on the time and location characteristics of the behavioral trajectory, while other users who do not belong to the first user are the second users.

[0109] S205. If the target category is the first category, and the candidate user group includes the first user and the second user, then determine the target purchase behavior corresponding to the candidate local image.

[0110] In this embodiment, the target purchasing behavior can refer to the purchasing behavior corresponding to the purchase event between the customer and the store clerk. After the electronic device performs bag-handling behavior recognition and user identity recognition, if the target category corresponding to the candidate partial image is the first category, that is, the candidate partial image includes bag-handling behavior; and the candidate user group corresponding to the candidate partial image includes the first user and the second user, that is, the two users in the candidate partial image are the store clerk and the customer, then the electronic device can determine the target purchasing behavior corresponding to the candidate partial image. In this way, the electronic device can accurately locate the purchasing behavior in the real-time video stream by detecting and recognizing the real-time video stream based on visual algorithms, and achieve accurate recognition of the purchasing behavior. Subsequently, it can determine the "fly-by-night" behavior based on the purchasing behavior and sales data, which can effectively reduce the risk of "fly-by-night" behavior.

[0111] In this embodiment, the electronic device acquires a real-time video stream sent by a camera module and extracts user feature information from the real-time video stream; determines a candidate user group whose feature information meets a first preset condition, and determines a candidate partial image corresponding to the candidate user group; based on the candidate partial image and a target recognition model, determines the target category corresponding to the candidate partial image; and determines a first user and a second user in the real-time video stream according to the feature information. If the target category is the first category, and the candidate user group includes both the first user and the second user, then the target purchase behavior corresponding to the candidate partial image is determined. In this embodiment, the electronic device performs a preliminary screening of bag-handling behavior based on the feature information extracted from the real-time video to determine candidate partial images, and then accurately identifies the candidate partial images to determine whether the target category corresponding to the candidate partial image is bag-handling behavior; simultaneously, the electronic device can identify the first user (salesperson) and the second user (customer) based on the feature information. When the target category corresponding to the candidate partial image is bag-handling behavior and the candidate user group includes both salesperson and customer, the electronic device determines that a purchase behavior has occurred. In this way, electronic devices can automatically detect purchasing behavior based on real-time video, which is more efficient and eliminates the need for manual spot checks, thus reducing labor costs. At the same time, by recognizing bag delivery behavior and identifying store clerks and customers, the accuracy of purchasing behavior detection can be improved. Subsequently, purchasing behavior can be used to accurately identify unauthorized orders, thereby effectively reducing the risk of unauthorized orders.

[0112] Based on the above embodiments, Figure 3 A flowchart illustrating another behavior detection method provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 3 The behavior detection method may include:

[0113] S301. Obtain the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream.

[0114] S302. Determine the user's location information based on the feature information; if the location information of two users meets the preset location conditions in N consecutive image frames, then the two users are determined as a candidate user group; where N is an integer greater than 1.

[0115] In this embodiment, location information can refer to information such as the user's foot coordinates and body orientation angle. Preset location conditions can refer to pre-set location relationship conditions, specifically, the distance between the foot coordinates of two users is less than a preset distance threshold, or the two users are facing each other. Specifically, after determining the user's feature information in the real-time video stream, the electronic device can determine the user's location information based on key human body features, and subsequently identify users whose location information meets the preset location conditions as a candidate user group that may engage in bag-passing behavior.

[0116] In one possible implementation, step S302 can be specifically achieved through the following steps (1) to (2):

[0117] (1) Based on the feature information, determine the coordinate information of the user and the orientation angle information of the user.

[0118] In this embodiment, the coordinate information can refer to the user's foot coordinates, specifically the world coordinates or image coordinates of the user's foot points. Specifically, the electronic device can determine the image coordinates of the user's foot points based on key human body points and other information in the feature information. When the camera module is pre-configured with calibration information (the correspondence between image pixel distance and actual scene distance), the electronic device can convert the image coordinates of the user's foot points into world coordinates in the world coordinate system and use these world coordinates as the user's corresponding coordinate information. When the camera module is not configured with calibration information, the electronic device can directly use the image coordinates of the user's foot points as the user's corresponding coordinate information.

[0119] Orientation angle information can refer to the angle between the perpendicular line from the line containing the two shoulders of the human body and the horizontal line (X-axis of the world coordinate system). Electronic devices can obtain the user's orientation angle information through a human orientation multi-classification model. Of course, electronic devices can also use other methods to determine the user's coordinate information and orientation angle information, and the specific settings can be flexibly configured based on actual needs. This application embodiment does not limit this.

[0120] (2) In N consecutive image frames, if the coordinate information and orientation angle information of two users meet the preset position conditions, then the two users are determined as the candidate user group.

[0121] In this embodiment of the application, after determining the user's coordinate information and orientation angle information, the electronic device can further determine whether the position information between the two meets the preset position conditions. Specifically, it can determine whether the distance between the coordinate information of the two users is less than a preset distance threshold, and whether the orientation angle information of the two users meets the face-to-face relationship.

[0122] In this step, the electronic device can calculate the distance between the coordinates of two users. If this distance is less than a preset distance threshold, the electronic device can determine that the coordinates of the two users satisfy a preset positional relationship. For example, assuming the coordinates of the foot points of the two users' body frames in the world coordinate system are (x1, y1) and (x2, y2) respectively, and the preset distance threshold is T1, the distance T between the coordinates of the two users can be calculated using... The calculation is performed. When the distance T is less than the preset distance threshold T1, the electronic device can determine that the coordinate information between the two users satisfies the preset positional relationship.

[0123] Simultaneously, the electronic device can determine the orientation relationship between two users based on their orientation angle information. If this orientation relationship is less than a preset angle threshold, the electronic device can determine that the two users are face-to-face, meaning their orientation angle information satisfies a preset positional relationship. For example, assuming the human body orientation angle information in the two users' frames is θ1 and θ2 respectively, and the preset angle threshold is T... θ The orientation relationship θ between two users' orientation angles can be calculated using θ = fabs(θ1 - (π - θ2)). When the orientation relationship θ is less than a preset angle threshold T... θ At that time, the electronic device can determine that the orientation angle information between two users meets the preset positional relationship.

[0124] For example, Figure 4 This is a schematic diagram illustrating user orientation relationship calculation as an exemplary embodiment of this application. Figure 4 As shown, the angle between the perpendicular line from the line connecting user A's left and right shoulders and the horizontal line is θ1, which represents user A's orientation angle. The angle between the perpendicular line from the line connecting user B's left and right shoulders and the horizontal line is θ2, which represents user B's orientation angle. The electronic device can determine the orientation relationship θ between user A and user B based on θ1 and θ2. When θ is less than a preset angle threshold T... θ At that time, the electronic device can determine that the orientation angle information between user A and user B meets the preset positional relationship.

[0125] In this embodiment, the electronic device determines the user's location information based on the user's characteristic information. By using methods such as distance calculation and angle calculation, it determines the candidate user group that may have bag-handling behavior based on whether the location information meets the preset location conditions. This can accurately determine the relative position between users, improve the accuracy of determining the candidate user group for bag-handling behavior, and thus improve the accuracy of subsequent purchase behavior identification. At the same time, the filtering can also reduce the amount of calculation and save system resources.

[0126] In one possible implementation, the behavior detection method may further include the following steps (3) to (5):

[0127] (3) Determine the target image region in the real-time video stream based on the real-time video stream.

[0128] In this embodiment, the target image region can refer to an area captured by the real-time video stream where the bag-handing behavior has a high probability of occurring, such as the cashier area or a waiting area (sofa lounge area). The electronic device can determine the target image region in the real-time video stream based on image detection algorithms and target recognition algorithms.

[0129] (4) For the target image region, detect and determine the candidate user group according to the first frame rate.

[0130] (5) For image regions other than the target image region, detect and determine the candidate user group according to the second frame rate; the first frame rate is greater than the second frame rate.

[0131] In this embodiment, the frame rate refers to the rate at which the electronic device performs image detection on the real-time video stream. A higher frame rate means the electronic device detects more image frames per unit time. Since the real-time video stream has a large shooting range, the computational burden on the electronic device is significant when detecting images frame by frame. In offline shopping malls and department stores, the act of handing bags between staff and customers is more likely to occur in the cashier area and waiting areas. Based on this, the electronic device can perform detection and recognition at a higher first frame rate for target image areas such as the cashier area and waiting areas; and at a lower second frame rate for other image areas outside the target image areas. This allows the electronic device to use different frame rates to detect and recognize different image areas, which can improve the recall rate of candidate user groups, save system resources, and reduce the computational burden on the electronic device.

[0132] S303. Determine the candidate local images corresponding to the candidate user groups.

[0133] In this embodiment, when the location information of two users meets preset location conditions in N consecutive image frames, the electronic device can determine these two users as a candidate user group. The electronic device can crop the local images corresponding to the candidate user group in the N image frames to obtain candidate local images, and then store the candidate local images in the form of an image sequence. Subsequently, the candidate local images can be used to identify bag-handling behavior.

[0134] S304. In the candidate local images, determine and label the user's hand detection box and the target bag detection box to obtain the image to be recognized.

[0135] In this embodiment, the human hand detection bounding box can be used to mark the hand area of ​​a human body. The target bag detection bounding box can be used to mark the target bag, which can refer to a brand bag corresponding to the current store as captured by the real-time video stream. The image to be identified can refer to a candidate partial image marked by an image detection model.

[0136] In this step, after identifying candidate local images, the electronic device can input these images into the image detection model to obtain a sequence of images to be recognized, labeled with bounding boxes for human hands and target bags. Subsequently, the device can perform bag-handling behavior recognition based on this sequence and the target recognition model. In this way, by using information about human hands and target bags as prior information when recognizing bag-handling behavior, the electronic device can further improve the accuracy of bag-handling behavior recognition based on image sequences.

[0137] During the process of determining and labeling the target bag detection box, the electronic device can also detect the target bag to determine whether it belongs to the current store. Specifically, the electronic device can extract the image features corresponding to the target bag in the candidate local image based on the image detection model, and then compare and match them with the pre-configured features of the current store's bags. If the two do not match, the target bag in the candidate local image is not a bag from the current store, and there is no target purchase behavior based on the bag-handling action in the candidate local image, so there is no need for further identification and judgment processes. This reduces the amount of computation and saves system computing resources. If the two match, the target bag in the candidate local image is a bag from the current store, and the electronic device can continue to perform the subsequent identification and judgment processes.

[0138] S305. Input the image to be identified into the target feature extraction model to determine the image features corresponding to the image to be identified, and at the same time determine the heat map features corresponding to the image to be identified; the heat map features are centered on the human hand detection box and the target bag detection box.

[0139] In this embodiment, the target recognition model includes a target feature extraction model and a target classification model. The target feature extraction model can be created based on Two-Stream Convolutional Networks (BRNNs). One branch of the network is used to extract image features from the image to be recognized, while the other branch is used to extract heatmap features of the image to be recognized, centered on the human hand detection box and the target bag detection box. The target classification model can be created based on a convolutional neural network (CNN) for feature classification. Of course, the target recognition model can also be constructed using other models or based on other algorithms. The specific configuration can be flexibly set according to actual needs. This embodiment does not limit the model composition or the algorithm used for target recognition.

[0140] S306. The image features and heatmap features are fused to obtain fused features; the fused features are input into the target classification model to obtain the target category corresponding to the candidate local image.

[0141] In this embodiment, the fusion feature refers to the feature obtained by fusing image features and heatmap features of the image to be identified. Specifically, the electronic device can first extract image features of the image to be identified using a target feature extraction model, and simultaneously extract heatmap features of the image to be identified, centered on the human hand detection box and the target bag detection box. Then, the electronic device can fuse the image features and the heatmap features to obtain the fused feature. This fused feature can then be input into a target classification model to obtain the target category corresponding to the image to be identified, i.e., the target category corresponding to the candidate local image. In this way, the electronic device uses the human hand and the target bag as prior information, and determines the target category corresponding to the candidate local image through feature extraction and classification, improving the accuracy of bag-handling behavior detection.

[0142] In one possible implementation, the target recognition model can be trained as follows:

[0143] (6) Obtain training samples; training samples include images of bag handing behavior labeled with hand detection boxes and bag detection boxes.

[0144] In this embodiment, training samples refer to samples used for model training, which may include images of bag-handling behavior pre-annotated with human hand detection boxes and bag detection boxes. Training samples may include bag-handling behavior images as positive samples, and may also include other non-bag-handling behavior images as negative samples. This embodiment does not limit the specific configuration of the training samples.

[0145] (7) Input the training samples into the preset recognition model and determine the training category corresponding to the training samples.

[0146] In this embodiment, the preset recognition model can refer to the recognition model to be trained, specifically including a preset feature extraction model and a preset classification model. The training category can refer to the category corresponding to the training sample, specifically including a first category belonging to the bag-handling behavior and a second category not belonging to the bag-handling behavior.

[0147] (8) Based on the training category and the preset loss function, the preset recognition model is iteratively trained to obtain the target recognition model.

[0148] In this embodiment, after determining the training category corresponding to the training sample, the electronic device can determine the loss value based on the training category and a preset loss function, and iteratively train the preset recognition model based on the loss value to finally obtain the trained target recognition model. The specific form of the preset loss function and the specific method of iterative training can be flexibly set according to actual needs, and this embodiment does not limit them.

[0149] For example, Figure 5This is a schematic diagram of an algorithm for recognizing bag-handling behavior, provided as an exemplary embodiment of this application. Figure 5 As shown, the electronic device first detects and identifies candidate local image sequences based on an image detection model, marking human hand detection boxes and target bag detection boxes in the candidate local image sequences to obtain the image sequence to be identified. Then, the electronic device can input the image sequence to be identified into a two-stream network of a target feature extraction model. One network branch extracts the image features of the image sequence to be identified; the other network branch generates a heatmap centered on the human hand detection box and the target bag detection box, and extracts the corresponding heatmap features of the image sequence to be identified. The electronic device can then fuse the image features and heatmap features corresponding to the image to be identified to obtain fused features. Finally, the fused features can be input into a target classification model to obtain the target category corresponding to the image to be identified.

[0150] In this embodiment, when a bag appears in a crowd, it is easy to misidentify nearby people and bags as bag-handing behavior, resulting in misidentification. The electronic device can filter out these misidentifications through post-processing. In one possible implementation, after step S304, the behavior detection method may further include the following steps:

[0151] (9) If the target category is the first category, then for the candidate user group corresponding to the first category, determine the association between the user and the target bag.

[0152] (10) If the relationship between the user and the target bag changes, the target category corresponding to the candidate local image is determined to be correctly identified.

[0153] In this embodiment, the association relationship can refer to the matching relationship between the user and the target bag, such as whether the target bag belongs to the user's possession. When the electronic device identifies that the target category corresponding to the candidate partial image is the first category, that is, the candidate partial image includes the bag-handling behavior, the electronic device can track and detect the candidate user group corresponding to the candidate partial image to determine the association relationship between the user and the target bag. If the association relationship between the user and the target bag changes, for example, the target bag changes from being an possession of user A to being an possession of user B, then it is determined that the target bag has been transferred between user A and user B, and the identification result that the target category corresponding to the candidate partial image is the first category is accurate. If the association relationship between the user and the target bag does not change, then it is determined that the target bag has not been transferred between the two users, and the identification result that the target category corresponding to the candidate partial image is the first category is incorrect, which is a misidentification. In this way, by tracking and detecting the candidate user group that includes the bag-handling behavior, the electronic device can further confirm whether the target bag has been transferred between the two users, filter out misidentified bag-handling behaviors, and ensure the accuracy of bag-handling behavior identification.

[0154] It should be emphasized that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0155] S307. Based on feature information, perform correlation tracking on the same user in the real-time video stream to determine the user's motion trajectory.

[0156] In this embodiment, the motion trajectory can refer to a relatively complete movement trajectory of the user. Due to factors such as occlusion, posture changes, entering blind spots, or entering a fitting room, the trajectory obtained by the electronic device when tracking and detecting a user based on a multi-target tracking algorithm is usually not a complete trajectory, requiring the association of trajectory segments of the same person. Furthermore, when the camera module includes multiple cameras at different locations, the electronic device also needs to associate trajectory segments of the same user under different cameras. The electronic device can obtain the complete motion trajectory of the same user based on multiple trajectory segments of the user, the REID features of the user's body bounding box, and feature similarity matching. Of course, the electronic device can also use other methods to obtain the user's motion trajectory, and this embodiment does not limit this method.

[0157] S308. If the motion trajectory meets the second preset condition, the user is identified as the first user; if the motion trajectory does not meet the second preset condition, the user is identified as the second user.

[0158] In this embodiment, the second preset condition can refer to a pre-set judgment condition for the first user, i.e., the store clerk. This second preset condition can be set based on time characteristics and movement location characteristics, where time characteristics can include start time, end time, and duration, and movement location characteristics can refer to the movement trajectory passing through a specific location (store entrance), etc. After determining the user's movement trajectory, the electronic device can further determine whether the movement trajectory meets the second preset condition. If the user's movement trajectory meets the second preset condition, the electronic device can determine that the user is the first user; if the user's movement trajectory does not meet the second preset condition, the electronic device can determine that the user is the second user.

[0159] In one possible implementation, the second preset condition includes:

[0160] The start and / or end times of the motion trajectory are included in the preset time period; and / or,

[0161] The duration of the motion trajectory exceeds a preset time threshold.

[0162] In this embodiment, the start time can refer to the time when the user's movement trajectory begins to appear. The end time can refer to the time when the user's movement trajectory ends. The preset time period can refer to the non-business hours of the shopping mall counter, such as after the shopping mall counter closes for business and before it opens for business. The preset time threshold can refer to a pre-set threshold value for the duration, specifically 3 hours, 4 hours, etc.

[0163] In this step, in one approach, when determining whether a user's movement trajectory meets the second preset condition, the electronic device can determine whether the start time and / or end time of the user's movement trajectory are included within a preset time period. If the start time of the user's movement trajectory is before the mall counter opens for business, and / or the end time of the user's movement trajectory is after the mall counter closes for business, then the electronic device can determine that the user is the first user, i.e., a salesperson. In another approach, when determining whether a user's movement trajectory meets the second preset condition, the electronic device can determine whether the duration of the user's movement trajectory is greater than a preset time threshold. If the duration of the user's movement trajectory is greater than the preset time threshold, the electronic device can determine that the user is a salesperson.

[0164] In this embodiment of the application, the electronic device determines the user's identity based on whether the user's movement trajectory meets the second preset condition, which can accurately determine whether the user is a store clerk or a customer. Subsequently, the purchase behavior can be determined based on the user's identity, which can improve the accuracy of the purchase behavior determination.

[0165] S309. If the target category is the first category, and the candidate user group includes the first user and the second user, then determine the target purchase behavior corresponding to the candidate local image.

[0166] In this embodiment, when the target category corresponding to the candidate partial image is the first category, meaning the candidate partial image includes the bag-handing behavior, and the two users in the candidate user group are a customer and a salesperson, the electronic device can determine that the candidate partial image corresponds to the target purchasing behavior, and that a purchasing event exists between the customer and the salesperson. Subsequently, a comparative analysis can be performed based on the target purchasing behavior and sales data to comprehensively and accurately identify any instances of unauthorized transactions, thereby reducing the risk of unauthorized transactions.

[0167] Based on any of the above embodiments, Figure 6 This is a schematic diagram of a technology chain for behavior detection provided as an exemplary embodiment of this application. For example... Figure 6As shown, the electronic device tracks and detects pedestrians (i.e., users) in a real-time video stream, extracting the feature information of each user. Based on this feature information, the electronic device determines the user's location information and identifies two users whose location information meets preset location conditions within N consecutive frames as a candidate user group. This candidate group is then cropped to obtain a candidate local image sequence, achieving preliminary screening of users engaging in bag-passing behavior. Subsequently, the electronic device can input the candidate local image sequence into a target recognition model to determine the target category corresponding to the candidate local image sequence and whether the candidate local image sequence includes bag-passing behavior.

[0168] Simultaneously, the electronic device performs correlation matching on trajectory segments of the same user in the real-time video stream to obtain the complete motion trajectory of the same user, thus merging pedestrians. Then, the electronic device can determine the user's identity based on whether the user's motion trajectory meets a second preset condition, identifying the first user (shop assistant) and the second user (customer). When the candidate local image sequence includes the act of handing over a bag, and the two users in the candidate user group are a shop assistant and a customer, the electronic device can determine that the candidate local image sequence corresponds to the target purchasing behavior, indicating that a purchase event has occurred between the shop assistant and the customer.

[0169] In this embodiment, the electronic device identifies shopping behavior by recognizing the bag-handing actions of shop assistants and customers. This method is highly feasible, has a short identification link, and results in higher accuracy in determining purchase behavior, thus improving the identification of the risk of unauthorized transactions. Furthermore, this embodiment uses foot coordinate information and the angle of the human body to initially screen users who hand over bags, which can improve the accuracy of purchase behavior recognition while reducing the computational burden on the system. In addition, when recognizing bag-handling behavior, the electronic device uses the human hand and the target bag as prior information, which can further improve the accuracy of purchase behavior recognition based on real-time video streams.

[0170] Figure 7 Please refer to the structural schematic diagram of a behavior detection device provided for an exemplary embodiment of this application. Figure 7 The behavior detection device 70 includes:

[0171] Extraction module 71 is used to acquire the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream;

[0172] The first determining module 72 is used to determine the candidate user group whose feature information meets the first preset condition, and to determine the candidate local image corresponding to the candidate user group.

[0173] The second determining module 73 is used to determine the target category corresponding to the candidate local image based on the candidate local image and the target recognition model;

[0174] The third determining module 74 is used to determine the first user and the second user in the real-time video stream based on the feature information;

[0175] The fourth determination module 75 is used to determine the target purchase behavior corresponding to the candidate local image if the target category is the first category and the candidate user group includes the first user and the second user.

[0176] In one possible implementation, the first determining module 72 is specifically used for:

[0177] Based on the feature information, determine the user's location information;

[0178] If the location information of two users meets the preset location conditions in N consecutive image frames, then the two users are identified as candidate user groups; where N is an integer greater than 1.

[0179] In one possible implementation, the first determining module 72 is specifically used for:

[0180] Based on the feature information, determine the user's coordinate information and the user's orientation angle information;

[0181] If the coordinate information and orientation angle information of two users meet the preset position conditions in N consecutive image frames, then the two users are identified as the candidate user group.

[0182] In one possible implementation, the device 70 is further used for:

[0183] Based on the real-time video stream, determine the target image region in the real-time video stream;

[0184] For the target image region, detect and determine the candidate user group according to the first frame rate;

[0185] For image regions other than the target image region, candidate user groups are detected and determined according to the second frame rate; the first frame rate is greater than the second frame rate.

[0186] In one possible implementation, the target recognition model includes a target feature extraction model and a target classification model; the second determining module 73 is specifically used for:

[0187] In the candidate local images, the user's human hand detection box and the target bag detection box are identified and labeled to obtain the image to be recognized;

[0188] The image to be identified is input into the target feature extraction model to determine the image features corresponding to the image to be identified, and at the same time, the heat map features corresponding to the image to be identified are determined; the heat map features are centered on the human hand detection box and the target bag detection box;

[0189] Image features and heatmap features are fused to obtain fused features;

[0190] The fused features are input into the target classification model to obtain the target category corresponding to the candidate local image.

[0191] In one possible implementation, the device 70 is further used for:

[0192] Obtain training samples; training samples include images of bag-handling behavior labeled with hand detection boxes and bag detection boxes;

[0193] The training samples are input into the preset recognition model to determine the training category corresponding to the training samples;

[0194] Based on the training category and the preset loss function, the preset recognition model is iteratively trained to obtain the target recognition model.

[0195] In one possible implementation, the third determining module 74 is specifically used for:

[0196] Based on feature information, the same user is correlated and tracked in real-time video streams to determine the user's motion trajectory;

[0197] If the movement trajectory meets the second preset condition, the user will be identified as the first user;

[0198] If the motion trajectory does not meet the second preset condition, the user will be identified as the second user.

[0199] In one possible implementation, the second preset condition includes:

[0200] The start and / or end times of the motion trajectory are included in the preset time period; and / or,

[0201] The duration of the motion trajectory exceeds a preset time threshold.

[0202] In one possible implementation, the device 70 is further used for:

[0203] If the target category is the first category, then for the candidate user group corresponding to the first category, determine the association between the user and the target bag;

[0204] If the relationship between the user and the target bag changes, it is determined that the target category corresponding to the candidate local image is correctly identified.

[0205] The behavior detection device 70 provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0206] Figure 8For a schematic diagram of an electronic device provided as an exemplary embodiment of this application, please refer to [link / reference]. Figure 8 The electronic device 80 may include a processor 81 and a memory 82. Exemplarily, the processor 81 and the memory 82 are interconnected via a bus 83.

[0207] Memory 82 stores instructions executed by the computer;

[0208] The processor 81 executes computer execution instructions stored in the memory 82, causing the processor 81 to perform the behavior detection method as shown in the above method embodiment.

[0209] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the behavior detection method of the above-described method embodiments.

[0210] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the behavior detection method shown in the above method embodiments.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] In a typical configuration, a computing device includes one or more processors, input / output interfaces, network interfaces, and memory.

[0216] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0217] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0218] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0219] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of behavior detection, the method comprising: include: Acquire the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream; A candidate user group whose feature information satisfies a first preset condition is identified, and a candidate local image corresponding to the candidate user group is identified; wherein, the first preset condition refers to a pre-set bag-passing behavior screening condition; and the candidate local image refers to a sequence of local images corresponding to the candidate user group that may exhibit bag-passing behavior. Based on the candidate local images and the target recognition model, the target category corresponding to the candidate local images is determined; Based on the feature information, the first user and the second user in the real-time video stream are determined; If the target category is the first category, and the candidate user group includes the first user and the second user, then the candidate local image corresponds to the target purchase behavior. The target recognition model includes a target feature extraction model and a target classification model; determining the target category corresponding to the candidate local image based on the candidate local image and the target recognition model includes: In the candidate local images, the user's hand detection box and the target bag detection box are identified and labeled to obtain the image to be identified; the image to be identified is input into the target feature extraction model to determine the image features corresponding to the image to be identified, and at the same time, the heat map features corresponding to the image to be identified are determined; the heat map features are centered on the human hand detection box and the target bag detection box; The image features and the heatmap features are fused to obtain fused features; the fused features are input into the target classification model to obtain the target category corresponding to the candidate local image.

2. The method according to claim 1, characterized in that, The step of determining the candidate user group whose feature information meets the first preset condition includes: Based on the feature information, the user's location information is determined; If the location information of two users meets the preset location conditions in N consecutive image frames, then the two users are determined as the candidate user group; where N is an integer greater than 1.

3. The method according to claim 2, characterized in that, The step of determining the candidate user group whose feature information meets the first preset condition includes: Based on the feature information, determine the user's coordinate information and the user's orientation angle information; If, in N consecutive image frames, the coordinate information and orientation angle information of two users satisfy the preset position conditions, then the two users are identified as the candidate user group.

4. The method according to claim 1, characterized in that, The method further includes: Based on the real-time video stream, determine the target image region in the real-time video stream; For the target image region, the candidate user group is detected and determined according to the first frame rate; For image regions other than the target image region, the candidate user group is detected and determined according to a second frame rate; the first frame rate is greater than the second frame rate.

5. The method according to claim 1, characterized in that, The method further includes: Obtain training samples; the training samples include images of bag-handling behavior labeled with hand detection boxes and bag detection boxes; The training samples are input into a preset recognition model to determine the training category corresponding to the training samples; The preset recognition model is iteratively trained according to the training category and the preset loss function to obtain the target recognition model.

6. The method according to claim 1, characterized in that, The step of determining the first user and the second user in the real-time video stream based on the feature information includes: Based on the feature information, the same user is correlated and tracked in the real-time video stream to determine the user's motion trajectory; If the motion trajectory meets the second preset condition, the user is identified as the first user; If the motion trajectory does not meet the second preset condition, the user is identified as the second user.

7. The method according to claim 6, characterized in that, The second preset condition includes: The start and / or end times of the motion trajectory are included in a preset time period; and / or, The duration of the motion trajectory is greater than a preset time threshold.

8. The method according to claim 1, characterized in that, After determining the target category corresponding to the candidate local image, the method further includes: If the target category is the first category, then for the candidate user group corresponding to the first category, determine the association between the user and the target bag; If the association between the user and the target bag changes, then the target category corresponding to the candidate local image is determined to be correctly identified.

9. A behavior detection device, characterized in that, include: An extraction module is used to acquire the real-time video stream sent by the camera module and extract the user's feature information from the real-time video stream; The first determining module is used to determine candidate user groups whose feature information satisfies a first preset condition, and to determine candidate local images corresponding to the candidate user groups; wherein, the first preset condition refers to a pre-set bag-passing behavior screening condition; and the candidate local image refers to a sequence of local images corresponding to the candidate user groups that may exhibit bag-passing behavior. The second determining module is used to determine the target category corresponding to the candidate local image based on the candidate local image and the target recognition model; The third determining module is used to determine the first user and the second user in the real-time video stream based on the feature information. The fourth determining module is used to determine the target purchase behavior corresponding to the candidate local image if the target category is the first category and the candidate user group includes the first user and the second user. The target recognition model includes a target feature extraction model and a target classification model; the second determining module is specifically used for: In the candidate local images, the user's hand detection box and the target bag detection box are identified and labeled to obtain the image to be identified; the image to be identified is input into the target feature extraction model to determine the image features corresponding to the image to be identified, and at the same time, the heat map features corresponding to the image to be identified are determined; the heat map features are centered on the human hand detection box and the target bag detection box; The image features and the heatmap features are fused to obtain fused features; the fused features are input into the target classification model to obtain the target category corresponding to the candidate local image.

10. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the behavior detection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the behavior detection method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program, which, when executed by a computer, implements the behavior detection method as described in any one of claims 1 to 8.

Images