A customer behavior analysis method applied to a physical store and an electronic device

Abstract

The application provides a customer behavior analysis method applied to a physical store and an electronic device. According to an embodiment of the application, first, according to a video stream collected by a camera in the physical store, a plurality of action sequences generated by customers when interacting with goods are generated, the action sequences are composed of a plurality of atomic actions with a set granularity in time sequence, then the behavior type indicated by the action sequence is recognized according to the execution sequence of the atomic actions, finally, customer behavior analysis is performed according to the behavior type of the plurality of customers. The scheme only needs to deploy a conventional camera and a corresponding posture analysis algorithm to complete the judgment of customer behavior, without the need for a high-density, high-resolution camera array, nor the need for relying on a high-precision hand tracking algorithm, thereby greatly reducing the equipment cost and deployment cost, having stronger compatibility and scalability, and being suitable for low-cost rapid deployment in traditional chain supermarkets, convenience stores and other widely used scenarios.

Description

Technical Field

[0001] This application relates to the field of commodity transaction technology, and in particular to a customer behavior analysis method, electronic device, storage medium, and computer program product applied to physical stores. Background Technology

[0002] As the digital transformation of offline retail stores continues to deepen, merchants not only need to optimize product display, customer flow layout and marketing strategies through refined analysis of customer behavior data, but also urgently need to establish an effective loss prevention management mechanism to reduce asset losses caused by missed or incorrect scanning or abnormal taking of goods.

[0003] One existing technical solution involves densely deploying high-resolution cameras (i.e., a hundred-camera array) on the ceiling, combined with 3D hand tracking algorithms to identify every instance of a customer taking or putting away an item in the store. This solution requires high hardware computing power and image resolution, and necessitates a complete overhaul of the shelving structure, weighing shelves, and checkout system, resulting in high equipment and deployment costs. Therefore, there is an urgent need for a cost-effective, easy-to-deploy, and efficient customer behavior analysis solution to address the technical challenges of high hardware investment, complex modifications, and difficulty in large-scale application of existing technologies. Summary of the Invention

[0004] This application provides a customer behavior analysis method, electronic device, storage medium, and computer program product for use in physical stores to solve one or more of the aforementioned technical problems.

[0005] In a first aspect, embodiments of this application provide a customer behavior analysis method applied to physical stores, comprising: generating multiple action sequences of customer interactions with products in the physical store based on video streams captured by cameras in the physical store, wherein the action sequences consist of multiple atomic actions with a set granularity; identifying the behavior type of customer interaction with products indicated by the action sequences based on the execution order of the atomic actions in the action sequences; and performing customer behavior analysis based on the behavior types of the multiple customer interactions with products.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements the above-described method when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0008] Fourthly, embodiments of this application provide a computer program product, wherein the computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0009] According to the embodiments of this application, firstly, based on the video stream captured by cameras in physical stores, multiple action sequences generated when customers interact with products are produced. These action sequences consist of multiple atomic actions of a set granularity arranged chronologically. Then, the behavior type indicated by the action sequence is identified according to the execution order of the atomic actions. Finally, customer behavior analysis is performed based on the behavior types of multiple customers. Compared to traditional solutions that rely solely on a single action (such as only recognizing picking up) for coarse-grained judgment, this solution constructs action sequences containing multiple atomic actions and utilizes the correlation and execution order between these actions to identify customer behavior types. This allows for more accurate capture of customer behavioral intentions and significantly improves the accuracy of behavior analysis. Furthermore, this solution only requires the deployment of conventional cameras and corresponding posture analysis algorithms to complete customer behavior judgment, eliminating the need for high-density, high-resolution camera arrays and high-precision hand tracking algorithms. This significantly reduces equipment and deployment costs, providing stronger compatibility and scalability, and is suitable for low-cost, rapid deployment in a wide range of scenarios such as traditional chain supermarkets and convenience stores. The customer behavior analysis data can provide quantitative and traceable data support for business decisions such as adjusting store merchandise displays, formulating promotional strategies, optimizing customer flow, and managing inventory.

[0010] Especially in scenarios involving damage assessment, traditional solutions often only identify a single customer action of picking up the product. Once contact with the product is detected without a corresponding checkout action, it is considered a potential damage risk. This is prone to misjudgment due to customers simply putting the product back on the shelf, without actually taking it. This application, by analyzing action sequences containing multiple atomic actions, can accurately distinguish whether a customer has actually taken the product, effectively eliminating situations where the product is put back after contact without generating any sales. This significantly improves the accuracy of damage assessment and provides more accurate and comprehensive data support for product display optimization and personalized marketing.

[0011] Furthermore, the action sequences obtained by analyzing the video stream in this embodiment can include multiple sequences. That is, it can continuously detect multiple behaviors of a customer throughout the entire process of entering a physical store, from entering, passing through, stopping, taking goods, to checking out. For example, if the behavior type indicated by a certain action sequence among the multiple action sequences is determined to be taking goods, then it checks whether there is a payment action sequence after the action sequence of taking goods. If there is no payment action sequence after the action sequence of taking goods, an abnormal behavior prompt is generated. Compared to the traditional approach of first identifying customer contact with goods and then manually verifying the same customer's checkout video, this application, through continuous detection of action sequences and analysis of behavior types, can determine in real time and automatically whether there is a corresponding payment behavior after the taking behavior, without the need for manual review of video for comparison afterward. This significantly improves the efficiency of abnormal behavior detection, reduces the false alarm rate, shortens the time window from the occurrence of an anomaly to its discovery, and avoids problems such as long loss recovery cycles and high labor costs caused by low detection efficiency.

[0012] This application embodiment first uses a spatiotemporal graph convolutional network to perform preliminary screening of the video stream, determining whether customer hand-product interaction occurs in each video frame combination, filtering out invalid frame combinations where no interaction occurs. For the remaining frame combinations confirmed to have interaction after screening, a more accurate visual model is then invoked to further identify the specific type of atomic action. Through this cascaded scheme of small-model screening combined with large-model fine-tuning, under the condition of limited edge computing resources, it effectively balances processing timeliness, recall rate, and recognition accuracy, achieving efficient understanding of video behavior, reducing the consumption of computing resources while ensuring the accuracy of key action recognition.

[0013] Furthermore, this application embodiment can also identify the target product corresponding to the abnormal prompt for subsequent continuous detection of the target product. Compared with the traditional method of directly identifying products based on product images, which has the problem of low recognition rate, this application embodiment adopts a combination of spatial positioning and shelf mapping. First, it identifies the first position coordinates of the target customer's hand in the target video stream in the camera coordinate system and converts them into the second position coordinates in the physical store map coordinate system. Then, based on the second position coordinates and the preset shelf layout data in the map coordinate system, it determines the target shelf location where the customer interacts. Finally, it matches the corresponding placed product from the product storage database based on the target shelf location as the target product. By indirectly determining the interacted product through the geometric relationship between the customer's hand position and the shelf space, it avoids recognition errors caused by factors such as product occlusion, light changes, and similar packaging, and is less affected by environmental interference.

[0014] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0016] Figure 1 A schematic diagram illustrating an example embodiment of this application is shown; Figure 2 A flowchart of a customer behavior analysis method for physical stores provided in an embodiment of this application is shown; Figure 3 This invention illustrates a structural block diagram of a customer behavior analysis device for physical stores, as provided in an embodiment of this application; and Figure 4 A block diagram of an electronic device used to implement embodiments of this application is shown. Detailed Implementation

[0017] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0018] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0019] This application provides a customer behavior analysis method for physical stores. The store is equipped with one or more cameras. Based on the video stream data collected by the cameras, multiple action sequences are generated representing customer interactions with products within the store. The video stream data used for analysis can be real-time or historical video stream data collected by the cameras. Customer interactions with products can include picking up or putting away products. These interactions are divided into multiple actions, called atomic actions, according to a predefined granularity. An action sequence is composed of multiple atomic actions combined in chronological order. In one example, the atomic action could be divided into four smallest granular actions: reaching out, touching, taking down, and withdrawing. These four atomic actions are arranged in chronological order to form an action sequence. In other words, this action sequence is not just a single action but depicts the continuous process of a customer's complete interaction from approaching to leaving the product. Through this step, structured behavioral data units that can be further identified and classified by the algorithm are obtained based on the original continuous video stream analysis, providing a data foundation for subsequent identification of specific behavior types.

[0020] The granularity of atomic actions refers to the fineness with which an interaction is broken down into actions, and can be customized according to business needs and computing power. For example, coarse granularity can be divided into three atomic actions: touching a product, moving a product, and releasing a product; medium granularity can be divided into actions such as reaching out, touching, taking down, and retracting; and finer granularity can include micro-actions such as fingertip touch, finger closing, and wrist rotation. The definition level of atomic actions can be flexibly set according to the actual application scenario, and the solution in this application is not limited to any fixed action division method.

[0021] To balance computational resource consumption with the continuity of behavior recognition, this solution samples the video stream captured by the camera. For example, the original video stream is downsampled by extracting one frame every three frames, meaning that every two frames are retained as the input frame sequence for subsequent analysis, while the two frames in between are discarded. This sampling strategy effectively reduces the total number of video frames to be processed while preserving the temporal continuity of key customer-product interaction actions, thereby reducing computational resource consumption and improving the real-time performance of behavior analysis.

[0022] In the process of generating action sequences based on video streams captured by cameras in physical stores, the first step is to identify the customer's hand posture information in multiple video frames. Hand posture information refers to structured data extracted from the customer's hand area in the video frame images, describing the hand's shape and spatial posture, including key hand coordinates and gesture states. Then, video frames are combined according to a set time window, which can correspond to the duration of an atomic action, such as 8 frames or 0.5 seconds. Based on the hand posture information corresponding to multiple video frames in the combined video frames, the type of atomic action corresponding to the video frame combination is identified. That is, based on the temporal changes in hand posture within multiple frames of the video frame combination, the type of atomic action occurring within the time window is identified. For example, based on whether the customer has a product in their hand and the directionality of their hand relative to the shelf, the customer's act of taking a product can be divided into four atomic action types: reaching out (reaching out with a product in hand), touching (reaching out with a product in hand), taking down (removing the product from hand), and retracting the hand (removing the product empty-handed). The atomic action types are sorted according to the time sequence corresponding to the combination of video frames to form the action sequence (e.g., reaching out - touching - taking down - withdrawing). The action sequence indicates the complete action type corresponding to the interaction by the changing trend of the multiple atomic actions it includes in the time sequence.

[0023] When extracting video frames within a set time window, this application supports two sliding window strategies. The first is an overlapping sliding window, where the window size is fixed (e.g., 8 frames) and slides sequentially in small increments (e.g., 1 frame), generating consecutive overlapping video frame combinations such as 1-8 frames, 2-9 frames, and 3-10 frames. This strategy can capture subtle changes in motion and avoid missed detections due to key actions falling within the window boundaries, making it suitable for scenarios requiring fine-grained motion analysis. The second is a non-overlapping sliding window, where the window size is fixed (e.g., 8 frames), but the windows do not overlap, extracting consecutive but non-overlapping video frame combinations such as 1-8 frames, 9-16 frames, and 17-24 frames. This strategy requires less computation and processes faster, making it suitable for scenarios with smooth motion changes or high real-time requirements. In practice, the two strategies can be flexibly selected or dynamically switched based on available computing resources and the required accuracy of motion recognition.

[0024] The aforementioned hand pose recognition based on video frames can be achieved using the YOLOPose pose detection algorithm. One implementation involves first detecting the bounding boxes of all human figures and their corresponding full-body keypoints (including the head, torso, and limbs) from the image frame. Then, based on the spatial position of the wrist or arm among the keypoints, the corresponding hand region is cropped, and further, the fine coordinates of the hand keypoints within that region are extracted. This approach utilizes full-body pose information to assist hand localization and is suitable for complex scenes with multiple people interspersed. Another implementation uses a hand detection model to first identify the area where the hand is located in the image, and then performs hand keypoint detection and pose analysis separately for that area. This approach provides more direct hand localization and has a relatively lower computational load. In practice, the two strategies can be flexibly selected or combined depending on the computing resources and scene complexity of the actual deployment environment.

[0025] In multi-camera scenarios, hand pose information of the same customer is integrated across frames. After calling a pose detection algorithm to identify the hand pose information corresponding to multiple video frames, the hand pose information of the same customer from multiple cameras is stitched together to obtain the hand pose information of the customer in multiple video frames. When stitching the hand pose information of the same customer from multiple cameras, ReID (Person Re-Identification) technology is used to extract the global appearance features of the customer (such as clothing color, body shape, etc.), and the hand pose information of the same customer in different frames is associated and stitched together to form a temporally continuous and spatially consistent sequence of hand pose information of the customer.

[0026] When identifying the atomic action type corresponding to a combination of video frames, the hand posture information corresponding to multiple consecutive video frames in the combination can be input into a pre-trained visual model. The model then analyzes the spatiotemporal dynamics of the hand posture and outputs the atomic action type corresponding to the combination of video frames, such as touching, taking down, or retracting the hand. The visual model used can be a CSN (Channel-Separable Network), a lightweight convolutional neural network structure specifically designed for video action recognition. By grouping and separating the feature calculations between channels, it significantly reduces the computational complexity and number of parameters of the model while maintaining high action recognition accuracy. It is suitable for deployment on edge servers with limited computing power, and can balance inference speed and recognition accuracy when processing multiple video streams in real time, achieving efficient and accurate recognition of customers' atomic hand movements. This provides a reliable, fine-grained data foundation for subsequent construction of action sequences and behavior type judgments.

[0027] Before using a visual model for action recognition, a lightweight model can be used to pre-screen video frame combinations. First, a spatiotemporal graph convolutional network is used to quickly identify hand pose information in the video frame combinations, determining whether there are any hand interactions with the product (such as reaching out, touching, or taking something down). Video frame combinations without such interactions are then removed. To ensure that no potential positive samples are missed, a very low confidence threshold (e.g., 0.1) can be used for the pre-screening model when removing video frame combinations without hand interactions. Even if the model's confidence in a particular interaction is low, it is still considered "interaction exists" and retained for later processing. Negative sample frame combinations that clearly lack any hand interaction, such as hands not moving or moving away from the product area, can be directly filtered out. Through the cascaded architecture of small-model screening combined with large-model fine-tuning, the lightweight spatiotemporal graph convolutional network can quickly eliminate a large number of trivial windows with no interaction, significantly reducing the amount of data that needs to be processed by subsequent fine-grained models. While ensuring high recall and not missing any possible interactive behaviors, it effectively reduces the overall computational load and improves the real-time performance of behavioral analysis, making it particularly suitable for deployment in edge computing environments with limited computing power. Specifically, the spatiotemporal graph convolutional network can be the ST-GCN++ network (Spatial Temporal Graph Convolutional Network++, an improved spatiotemporal graph convolutional network), which can enhance the accuracy and robustness of modeling high-order relationships between nodes in the spatiotemporal sequence for hand key points.

[0028] Figure 1 The diagram illustrates an example of an embodiment of this application. As shown, the video stream is first decoded into consecutive video frames and input to the "Detection Model + ReID" module. The module outputs a temporal sequence of hand bounding boxes and key points using object detection and person re-identification technologies. Subsequently, this sequence enters two parallel processing branches: one branch is the "Small Model - Temporal Key Points" branch, a lightweight spatiotemporal graph convolutional network used to predict the key point sequence of the video frame combination, quickly identifying whether there is a hand-product interaction action; the other branch is the "Large Model - Video Frame Sequence" branch, which uses a more precise visual model to perform fine-grained action recognition on the video frame combination. If the small model's prediction result is "No" (i.e., no interaction action detected), the data for that video frame combination is not further processed and is directly filtered; if the prediction result is "Yes," the large model is further invoked to perform a second judgment on the same video frame combination. If the judgment result output by the large model is "Yes," the behavioral event is reported; if it is "No," it is not reported.

[0029] After generating atomic action types, this application further optimizes the temporal classification results through post-processing to eliminate noise and merge continuous action segments. Because frame data overlaps between adjacent sliding windows, the same action type will appear consecutively in the classification results of multiple adjacent windows, forming a continuous and consistent recognition output. If the classification result of a window is inconsistent with the categories of its preceding and following windows, the result is determined to be noise or a misclassification and is discarded. If the classification result of a window is consistent with the category of its preceding or following window, it is merged into a continuous action segment to reduce fragmented output. Based on the above design, for each video frame combination, it is determined whether the atomic action type corresponding to the current video frame combination is consistent with the atomic action type of the preceding and following video frame combinations in time sequence. If the atomic action type of the current frame combination is inconsistent with both preceding and following frame combinations, it is considered noise and the classification result of the current frame combination is deleted. If the atomic action type of the current frame combination is consistent with the atomic action type of the preceding or following frame combination, it is merged into an adjacent frame combination with the consistent action type, forming a longer continuous action segment. Through the above elimination and merging mechanism, isolated misclassification points in the output of the classification model can be effectively filtered out, and recognition results that are sequential and of the same type can be integrated into complete action segments, thereby improving the continuity and accuracy of action sequences and providing a more reliable data foundation for subsequent behavior type recognition.

[0030] Based on the execution order of the atomic actions in the above action sequence, the type of customer interaction indicated by the action sequence can be identified. After obtaining an action sequence composed of multiple atomic actions in chronological order, the type of complete interaction between the customer and the product represented by the sequence can be determined based on the execution order of the atomic actions in the action sequence. For example, an action sequence consisting of four actions—reaching out, picking up, moving, and putting in the shopping basket—can be identified as a purchase behavior; while a sequence consisting of reaching out, picking up, moving, and putting back might indicate that the customer picked up the item and then put it back, and is identified as a non-purchase behavior. Through this sequence-based identification method, the customer's actual intention can be inferred from the temporal combination of atomic actions, rather than relying solely on a single action, thereby achieving a semantic-level understanding of customer behavior and providing a more accurate basis for subsequent business analysis or loss prevention judgment.

[0031] This mechanism extracts the atomic action types corresponding to two adjacent atomic actions in an action sequence and matches them with the action type combinations corresponding to preset behavior types. The preset behavior types define the standard atomic action combination sequences for each complete behavior. Then, based on the matching results, the behavior type of the customer-product interaction indicated by the action sequence is determined. If a match is successful, the type of customer-product interaction indicated by the action sequence is determined. This matching mechanism based on adjacent action type combinations takes into account the temporal relationship between atomic actions, allowing for finer-grained identification of customer behavioral intentions and avoiding ambiguity caused by relying solely on a single atomic action. In the scenario of goods damage detection, the action of simply taking something out cannot determine whether the customer ultimately buys it or puts it back, but combining this with the subsequent actions of putting it in the shopping basket or putting it back allows for accurate judgment.

[0032] If the preceding atomic action is reaching out with no item in hand, and the following atomic action is retracting the hand with the item in hand, it is considered a valid take-up action. If the preceding atomic action is reaching out with the item in hand, and the following atomic action is retracting the hand with no item in hand, it is considered a return action. The timing of the action can also be considered here. For example, if the following atomic action is retracting the hand with no item in hand, and the hand remains stationary for an extended period (e.g., more than 3 seconds) without moving, it is considered a return action.

[0033] Based on the behavioral types of multiple customers interacting with products obtained from the above analysis, further customer behavior analysis can be performed. Compared with traditional solutions that rely on a single action for coarse-grained judgment, this solution constructs an action sequence containing multiple atomic actions and uses the correlation and execution order between atomic actions to identify customer behavior types. This can more accurately capture customer behavioral intentions and significantly improve the accuracy of behavior analysis.

[0034] When performing customer behavior analysis based on the interaction types of multiple customers interacting with products, one or more of the following data can be collected: the number of interactions, duration, frequency, outcome (successful purchase or return), and changes in product quantity (increase or decrease due to picking up items) for the same interaction type among multiple customers. This quantifies the activity level and interaction intensity of customers in different shelf areas. It also allows for the identification of interactions corresponding to the product picking type, and the collection of customer characteristics (e.g., age group, gender, time of visit) and / or picking preferences (e.g., preferred product categories, brands). Furthermore, it allows for the identification of individual products and the counting of interactions from a product perspective, including the number of times a product is picked up, the number of times it is returned, and the frequency of trial picking (returning items after multiple pick-ups without purchase). This can generate a product hesitation index or popularity index, quantifying the impact of product display location on customer attention. For example, identifying products with high pick-up rates but low conversion rates indicates that the display location attracts significant attention but results in poor sales. This data can serve as a basis for subsequent business decisions and can be used in scenarios such as product mix optimization, personalized marketing push, and shelf layout adjustment.

[0035] It can also identify the products corresponding to interactive behaviors, and count interactive behaviors at the product level from the product dimension. This enables business analysis, such as statistics on the interaction frequency of popular shelf products, conversion rates corresponding to product picking operations, statistics on products with high picking rates but low conversion rates, or loss prevention management by locating products with abnormal picking behaviors. It provides an interpretable chain of behavioral evidence, enhancing the credibility and practicality of data output.

[0036] This application embodiment analyzes video streams to obtain action sequences that can include multiple sequences. That is, it can continuously detect multiple behaviors of a customer throughout the entire process of entering a physical store, from entering, passing through, stopping, taking goods, to checking out. For example, if the behavior type indicated by a certain action sequence is determined to be taking goods, it checks whether there is a payment action sequence after the action sequence for taking goods. If there is no payment action sequence after the action sequence for taking goods, an abnormal behavior prompt is generated. Compared to the traditional approach of first identifying customer contact with goods and then manually verifying the same customer's checkout video, this application, through continuous detection of action sequences and analysis of behavior types, can determine in real time and automatically whether there is a corresponding payment behavior after the taking behavior. This eliminates the need for manual review of video footage afterward, significantly improving the efficiency of abnormal behavior detection, reducing false alarm rates, shortening the time window from the occurrence of an anomaly to its discovery, and avoiding problems such as long loss recovery cycles and high labor costs caused by low detection efficiency.

[0037] Depending on the scope of analysis, data from one or more cameras can be analyzed independently or jointly. For example, when analyzing customer behavior regarding picking up goods from a specific shelf, only the video stream captured by a single camera corresponding to that shelf area needs to be analyzed. When analyzing the entire customer behavior within a physical store, from entering, passing through, stopping, picking up goods, to checkout, video stream data from multiple cameras can be acquired for joint analysis. This application eliminates the need for deploying high-density, high-resolution camera arrays; it relies solely on existing ordinary cameras and edge computing resources in the store to meet data acquisition and computation needs. This effectively reduces network latency and bandwidth consumption caused by uploading video data to the cloud, while also reducing dependence on cloud computing power, significantly lowering equipment and deployment costs. It also offers stronger compatibility and scalability, making it suitable for low-cost, rapid deployment in a wide range of scenarios, including traditional chain supermarkets and convenience stores. Taking a single small to medium-sized physical store as an example, configuring one edge server can cover the entire area of ​​the store. This edge server can deploy three GPU cards, each processing 80 channels of 1080P resolution video streams at 25 frames per second. In actual analysis, when analyzing multiple customers across the entire store simultaneously, a fixed time window (e.g., one hour) can be set. Only the video stream within that time window is extracted for customer behavior analysis, thereby effectively controlling the amount of data processed at one time, avoiding overloading computing resources, and achieving a balanced distribution of computing load while ensuring real-time analysis.

[0038] After generating an abnormal behavior alert, the product associated with the abnormal behavior can be identified as the target product. Then, based on the target product, the behavior types of multiple customer interactions with the target product (such as taking, putting back, and purchasing) can be summarized to conduct focused customer behavior analysis for that specific product. For example, the conversion rate of the product being taken and purchased, the frequency of trial taking, and the distribution of customer characteristics can be analyzed to help determine whether there are display problems, pricing problems, or loss risks for the product.

[0039] In this process, when identifying the product corresponding to the abnormal behavior prompt as the target product, the following steps are taken: First, the original target video stream corresponding to the action sequence that generated the abnormal behavior prompt is located. Second, the first position coordinates of the target customer's hand in the video stream under the camera coordinate system are identified, and then converted into second position coordinates under the physical store map coordinate system using a pre-calibrated coordinate transformation matrix. Next, based on these second position coordinates and pre-set shelf layout data in the map coordinate system (e.g., shelf boundaries, shelf positions, product display areas), the target shelf position corresponding to the target customer's interaction is determined. Finally, the product placed at that position is matched from the product storage database based on the target shelf position and designated as the target product. Through this series of spatial positioning and mapping steps, the customer's hand movement position can be mapped to the actual product entity, providing a basis for subsequent behavioral analysis of that product. The aforementioned method of combining spatial positioning with shelf mapping indirectly determines the interacting product through the geometric relationship between the customer's hand position and the shelf space. It does not rely on image recognition of the product's appearance, thus effectively avoiding recognition errors caused by factors such as product obstruction, changes in lighting, and similar packaging. It has the technical advantages of being less affected by environmental interference and accurate positioning, and supports product-level interaction behavior counting, providing an interpretable chain of behavioral evidence for business analysis and loss prevention management.

[0040] The execution entity in this application embodiment can be an application, service, instance, functional module in software form, virtual machine (VM), container, or cloud server, or hardware device with data processing function (such as server or terminal device) or hardware chip (such as CPU, GPU, FPGA, NPU, AI accelerator card, or DPU). The device for providing the service can be deployed on the computing device of the application providing the corresponding service or on a cloud computing platform providing computing power, storage, and network resources. The cloud computing platform can provide services in the following modes: IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), or DaaS (Data as a Service). Taking the platform providing SaaS (Software as a Service) as an example, the cloud computing platform can utilize its own computing resources to provide the execution of one or more of the above steps, and the specific application architecture can be built according to service requirements.

[0041] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0042] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0043] This application provides a customer behavior analysis method applicable to physical stores, such as... Figure 2 The diagram shows a flowchart of a customer behavior analysis method 200 applied to a physical store according to an embodiment of this application. The method 200 may include: in step 201, generating multiple action sequences of customer interactions with products in the physical store based on video streams captured by cameras in the physical store, wherein the action sequences consist of multiple atomic actions with a set granularity; in step 202, identifying the behavior type of customer interaction with products indicated by the action sequences based on the execution order of the atomic actions in the action sequences; and in step 203, performing customer behavior analysis based on the behavior types of the multiple customer interactions with products.

[0044] In one embodiment, generating a sequence of actions including multiple customer interactions with products in the physical store based on video streams captured by cameras in the physical store includes: identifying hand gesture information corresponding to customers in multiple video frames of the video stream; extracting video frame combinations according to a set time window; identifying atomic action types corresponding to the video frame combinations based on the hand gesture information corresponding to multiple video frames in the video frame combinations; and arranging the atomic action types according to the time sequence corresponding to the video frame combinations to form a sequence of actions including customer interactions with products.

[0045] In one embodiment, identifying the hand gesture information of the customer in multiple video frames of the video stream includes: calling a gesture detection algorithm to identify the hand gesture information corresponding to each of the multiple video frames; and stitching together the hand gesture information of the same customer from multiple cameras to obtain the hand gesture information of the customer in multiple video frames.

[0046] In one embodiment, identifying the atomic action type corresponding to the video frame combination based on the hand posture information corresponding to multiple video frames in the video frame combination includes: calling a visual model to perform action recognition on the hand posture information corresponding to multiple video frames in the video frame combination to obtain the atomic action type corresponding to the video frame combination.

[0047] In one embodiment, before calling the visual model to perform action recognition on the hand pose information corresponding to multiple video frames in the video frame combination, the step of identifying the atomic action type corresponding to the video frame combination based on the hand pose information corresponding to multiple video frames in the video frame combination further includes: calling a spatiotemporal graph convolutional network to perform action recognition on the hand pose information corresponding to multiple video frames in the video frame combination, determining whether there is an action in which the hand interacts with the product; and deleting video frame combinations in which there is no action in which the hand interacts with the product.

[0048] In one embodiment, the step of generating multiple action sequences of customer interactions with products in the physical store based on video streams captured by cameras in the physical store further includes: determining whether the atomic action types corresponding to the video frame combination are consistent with the previous and subsequent video frame combinations in time sequence; deleting the currently determined video frame combination if the atomic action types are inconsistent with both the previous and subsequent video frame combinations; and merging the currently determined video frame combination with the previous and / or subsequent video frame combinations if the atomic action types are consistent with both the previous and subsequent video frame combinations.

[0049] In one embodiment, identifying the behavior type of customer-product interaction indicated by the action sequence based on the execution order of atomic actions in the action sequence includes: extracting the atomic action types corresponding to two adjacent atomic actions in the action sequence, and matching them with the action types corresponding to preset behavior types; determining the behavior type of customer-product interaction indicated by the action sequence based on the matching result.

[0050] In one embodiment, the customer corresponds to multiple action sequences in the physical store. The step of performing customer behavior analysis based on the behavior types of the multiple customers interacting with the products includes: if the behavior type indicated by a certain action sequence among the multiple action sequences is determined to be the product-taking type, then checking whether there is a payment type action sequence after the product-taking type action sequence; if there is no payment type action sequence after the product-taking type action sequence, then generating an abnormal behavior prompt.

[0051] In one embodiment, the method further includes: identifying the product corresponding to the abnormal behavior prompt as the target product, and performing customer behavior analysis for the target product based on the behavior types of the multiple customers interacting with the target product.

[0052] In one embodiment, determining the product corresponding to the abnormal behavior prompt as the target product includes: searching for the target video stream corresponding to the action sequence of the abnormal behavior prompt; identifying the first position coordinates of the target customer's hand in the camera coordinate system in the target video stream, and converting the first position coordinates into a second position coordinate in the map coordinate system of the physical store; determining the target shelf location where the target customer's interaction occurred based on the second position coordinates and the shelf layout data in the map coordinate system of the physical store; and determining the corresponding product placed therein as the target product from the product storage database based on the target shelf location.

[0053] In one embodiment, the step of performing customer behavior analysis based on the behavior types of the multiple customers interacting with the products includes: based on the behavior types of the multiple customers interacting with the products, counting at least one of the following data: the number of interactions, duration of interactions, frequency of interactions, results of interactions, and changes in the quantity of products for the same behavior type; and / or, finding interactions that correspond to the behavior type of taking products, and counting customer characteristics and / or taking preferences.

[0054] According to the embodiments of this application, firstly, based on the video stream captured by cameras in physical stores, multiple action sequences generated when customers interact with products are produced. These action sequences consist of multiple atomic actions of a set granularity arranged in chronological order. Then, the behavior type indicated by the action sequence is identified based on the execution order of the atomic actions. Finally, customer behavior analysis is performed based on the behavior types of multiple customers. Compared to traditional solutions that rely solely on a single action (such as only recognizing picking up) for coarse-grained judgment, this solution constructs action sequences containing multiple atomic actions and utilizes the correlation and execution order between these actions to identify customer behavior types. This allows for more accurate capture of customer behavioral intentions and significantly improves the accuracy of behavior analysis. Furthermore, this solution only requires the deployment of conventional cameras and corresponding posture analysis algorithms to complete customer behavior judgment, eliminating the need for high-density, high-resolution camera arrays and high-precision hand tracking algorithms. This significantly reduces equipment and deployment costs, providing stronger compatibility and scalability, and is suitable for low-cost, rapid deployment in a wide range of scenarios such as traditional chain supermarkets and convenience stores. The customer behavior analysis data can provide quantitative and traceable data support for business decisions such as adjusting store merchandise displays, formulating promotional strategies, optimizing customer flow, and managing inventory.

[0055] Especially in scenarios involving damage assessment, traditional solutions often only identify a single customer action of picking up the product. Once contact with the product is detected without a corresponding checkout action, it is considered a potential damage risk. This is prone to misjudgment due to customers simply putting the product back on the shelf, without actually taking it. This application, by analyzing action sequences containing multiple atomic actions, can accurately distinguish whether a customer has actually taken the product, effectively eliminating situations where the product is put back after contact without generating any sales. This significantly improves the accuracy of damage assessment and provides more accurate and comprehensive data support for product display optimization and personalized marketing.

[0056] Furthermore, the action sequences obtained by analyzing the video stream in this embodiment can include multiple sequences. That is, it can continuously detect multiple behaviors of a customer throughout the entire process of entering a physical store, from entering, passing through, stopping, taking goods, to checking out. For example, if the behavior type indicated by a certain action sequence among the multiple action sequences is determined to be taking goods, then it checks whether there is a payment action sequence after the action sequence of taking goods. If there is no payment action sequence after the action sequence of taking goods, an abnormal behavior prompt is generated. Compared to the traditional approach of first identifying customer contact with goods and then manually verifying the same customer's checkout video, this application, through continuous detection of action sequences and analysis of behavior types, can determine in real time and automatically whether there is a corresponding payment behavior after the taking behavior, without the need for manual review of video for comparison afterward. This significantly improves the efficiency of abnormal behavior detection, reduces the false alarm rate, shortens the time window from the occurrence of an anomaly to its discovery, and avoids problems such as long loss recovery cycles and high labor costs caused by low detection efficiency.

[0057] This application embodiment first uses a spatiotemporal graph convolutional network to perform preliminary screening of the video stream, determining whether customer hand-product interaction occurs in each video frame combination, filtering out invalid frame combinations where no interaction occurs. For the remaining frame combinations confirmed to have interaction after screening, a more accurate visual model is then invoked to further identify the specific type of atomic action. Through this cascaded scheme of small-model screening combined with large-model fine-tuning, under the condition of limited edge computing resources, it effectively balances processing timeliness, recall rate, and recognition accuracy, achieving efficient understanding of video behavior, reducing the consumption of computing resources while ensuring the accuracy of key action recognition.

[0058] Furthermore, this application embodiment can also identify the target product corresponding to the abnormal prompt for subsequent continuous detection of the target product. Compared with the traditional method of directly identifying products based on product images, which has the problem of low recognition rate, this application embodiment adopts a combination of spatial positioning and shelf mapping. First, it identifies the first position coordinates of the target customer's hand in the target video stream in the camera coordinate system and converts them into the second position coordinates in the physical store map coordinate system. Then, based on the second position coordinates and the preset shelf layout data in the map coordinate system, it determines the target shelf location where the customer interacts. Finally, it matches the corresponding placed product from the product storage database based on the target shelf location as the target product. By indirectly determining the interacted product through the geometric relationship between the customer's hand position and the shelf space, it avoids recognition errors caused by factors such as product occlusion, light changes, and similar packaging, and is less affected by environmental interference.

[0059] Corresponding to the examples and method embodiments provided in this application, this application also provides a customer behavior analysis device applied to physical stores. For example... Figure 3 The diagram shown is a structural block diagram of a customer behavior analysis device 300 applied to a physical store according to an embodiment of this application. The device 300 may include: an action sequence generation module 301, used to generate multiple action sequences including the interaction behaviors of customers with products in the physical store based on the video stream captured by the camera in the physical store, wherein the action sequence is composed of multiple atomic actions with a set granularity; a behavior type identification module 302, used to identify the behavior type of the customer interaction behavior indicated by the action sequence based on the execution order of the atomic actions in the action sequence; and a behavior analysis module 303, used to perform customer behavior analysis based on the behavior types of the multiple customer interaction behaviors with products.

[0060] In one embodiment, the action sequence generation module includes: a posture recognition submodule, used to identify the hand posture information corresponding to the customer in multiple video frames of the video stream; an atomic action type recognition submodule, used to extract video frame combinations according to a set time window, and identify the atomic action type corresponding to the video frame combination based on the hand posture information corresponding to multiple video frames in the video frame combination; and an action sequence combination submodule, used to sort the atomic action types according to the time order corresponding to the video frame combination to form an action sequence including the interaction behavior between the customer and the product.

[0061] In one embodiment, the posture recognition submodule is specifically used to call a posture detection algorithm to identify hand posture information corresponding to multiple video frames; and to stitch together the hand posture information corresponding to the same customer from multiple cameras to obtain the hand posture information corresponding to the customer in multiple video frames.

[0062] In one embodiment, the atomic action type recognition submodule includes a visual model recognition subunit, which is used to call a visual model to perform action recognition on the hand posture information corresponding to multiple video frames in the video frame combination, and obtain the atomic action type corresponding to the video frame combination.

[0063] In one embodiment, the atomic action type recognition submodule further includes: a convolutional network invocation subunit, used to invoke a spatiotemporal graph convolutional network to perform action recognition on the hand pose information corresponding to multiple video frames in the video frame combination before invoking the visual model to perform action recognition on the hand pose information corresponding to multiple video frames in the video frame combination, and to determine whether there is an action in which the hand interacts with the product; and a combination deletion subunit, used to delete video frame combinations in which there is no action in which the hand interacts with the product.

[0064] In one embodiment, the action sequence generation module further includes: a combination judgment submodule, used to determine whether the atomic action types corresponding to the video frame combination are consistent with those of the preceding and following video frame combinations in terms of time sequence; a combination deletion submodule, used to delete the currently judged video frame combination if the atomic action types of the currently judged video frame combination are inconsistent with those of the preceding and following video frame combinations; and a combination merging submodule, used to merge the currently judged video frame combination with the preceding and / or following video frame combinations that have the same action types as the preceding and following video frame combinations if the atomic action types of the currently judged video frame combination are consistent with those of the preceding and following video frame combinations.

[0065] In one embodiment, the behavior type recognition module includes: a type matching submodule, used to extract the atomic action types corresponding to two adjacent atomic actions in the action sequence and match them with the action types corresponding to preset behavior types; and a behavior type determination submodule, used to determine the behavior type of the customer-product interaction indicated by the action sequence based on the matching result.

[0066] In one embodiment, the behavior analysis module includes: a type lookup submodule, configured to, if the behavior type indicated by a certain action sequence among the plurality of action sequences is determined to be the "taking goods" type, then look for whether there is a payment type action sequence after the "taking goods" type action sequence; and an exception prompting submodule, configured to, if there is no payment type action sequence after the "taking goods" type action sequence, then generate an exception behavior prompt.

[0067] In one embodiment, the device further includes a target product determination module, configured to determine the product corresponding to the abnormal behavior prompt as the target product, and to perform customer behavior analysis for the target product based on the behavior types of the multiple customers interacting with the target product.

[0068] In one embodiment, the target product determination module includes: a video stream search submodule, used to search for the target video stream corresponding to the action sequence of the abnormal behavior prompt; a first coordinate recognition submodule, used to identify the first position coordinates of the target customer's hand in the target video stream in the camera coordinate system; a coordinate transformation submodule, used to convert the first position coordinates into a second position coordinate in the map coordinate system of the physical store; a shelf location determination submodule, used to determine the target shelf location where the target customer's interaction occurred based on the second position coordinates and the shelf layout data in the map coordinate system of the physical store; and a product search submodule, used to determine the corresponding product placed in the product storage database as the target product based on the target shelf location.

[0069] In one embodiment, the behavior analysis module includes: a first statistical submodule and / or a second statistical submodule. The first statistical submodule is used to collect at least one of the following data based on the behavior types of the multiple customers interacting with the products: the number of interactions, duration of interactions, frequency of interactions, results of interactions, and changes in the quantity of products for the same behavior type. The second statistical submodule is used to find interactions that correspond to the type of product taking and to collect customer characteristics and / or taking preferences.

[0070] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0071] Figure 4 This is a block diagram of an electronic device used to implement embodiments of this application. Figure 4 As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the method described in the above embodiments. The number of memories 401 and processors 402 can be one or more.

[0072] The electronic device also includes: Communication interface 403 is used to communicate with external devices and exchange and transmit data.

[0073] If the memory 401, processor 402, and communication interface 403 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0074] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0075] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0076] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any embodiment of this application.

[0077] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0078] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in this application.

[0079] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0080] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0081] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0082] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0084] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0085] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0086] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0088] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A customer behavior analysis method applied to physical stores, comprising: Based on the video stream captured by the cameras in the physical store, generate multiple action sequences including the interaction behaviors of customers with products in the physical store, and the action sequences are composed of multiple atomic actions with a set granularity; Based on the execution order of the atomic actions in the action sequence, identify the type of customer interaction with the product indicated by the action sequence; Customer behavior analysis is performed based on the types of interactions that the various customers have with the products.

2. The method according to claim 1, wherein generating a sequence of actions based on video streams captured by cameras in the physical store, comprising multiple interactions between customers and products in the physical store, includes: Identify the hand gesture information of the customer in multiple video frames of the video stream; Video frames are extracted and combined according to a set time window. Based on the hand posture information corresponding to multiple video frames in the video frame combination, the atomic action type corresponding to the video frame combination is identified. The atomic action types are sorted according to the time sequence corresponding to the video frame combinations to form the action sequence included in the interaction between the customer and the product.

3. The method according to claim 2, wherein identifying the hand gesture information corresponding to the customer in multiple video frames of the video stream includes: The pose detection algorithm is invoked to identify the hand pose information corresponding to multiple video frames; By stitching together the hand gesture information of the same customer from multiple cameras, the hand gesture information of the customer in multiple video frames can be obtained.

4. The method according to claim 2, wherein identifying the atomic action type corresponding to the video frame combination based on the hand gesture information corresponding to multiple video frames in the video frame combination includes: A visual model is invoked to perform motion recognition on the hand posture information corresponding to multiple video frames in the video frame combination, thereby obtaining the atomic motion type corresponding to the video frame combination.

5. The method according to claim 4, before calling the visual model to perform action recognition on the hand pose information corresponding to multiple video frames in the video frame combination, the step of identifying the atomic action type corresponding to the video frame combination based on the hand pose information corresponding to multiple video frames in the video frame combination further includes: The spatiotemporal graph convolutional network is invoked to perform action recognition on the hand pose information corresponding to multiple video frames in the video frame combination, and to determine whether there is an action of hand interaction with the product; Delete video frame combinations that do not show any hand interaction with the product.

6. The method according to claim 2, wherein generating the action sequence of multiple customer interactions with products in the physical store based on the video stream captured by the camera in the physical store further includes: Determine whether the atomic action types corresponding to the video frame combination are consistent with the previous and subsequent video frame combinations in terms of time sequence; If the atomic action types of the currently judged video frame combination are inconsistent with those of the preceding and following video frame combinations, delete the currently judged video frame combination. If the atomic action type of the currently judged video frame combination is consistent with that of the preceding and following video frame combinations, the currently judged video frame combination is merged with the preceding video frame combination and / or the following video frame combination that has the same action type.

7. The method according to claim 1, wherein identifying the behavior type of customer-product interaction indicated by the action sequence based on the execution order of atomic actions in the action sequence includes: Extract the atomic action types corresponding to two adjacent atomic actions in the action sequence and match them with the action types corresponding to preset behavior types. The action sequence indicates the type of customer interaction with the product.

8. The method according to claim 1, wherein the customer corresponds to multiple action sequences in the physical store, and the step of performing customer behavior analysis based on the behavior types of the multiple customers interacting with the products includes: If the behavior type indicated by a certain action sequence among the multiple action sequences is determined to be the "take goods" type, then check whether there is a payment type action sequence after the "take goods" type action sequence; If there is no payment action sequence following the action sequence for retrieving goods, an abnormal behavior prompt will be generated.

9. The method according to claim 8, further comprising: The product corresponding to the abnormal behavior prompt is identified as the target product, and customer behavior analysis is performed on the target product based on the behavior types of the multiple customers interacting with the target product.

10. The method according to claim 9, wherein determining the product corresponding to the abnormal behavior prompt as the target product includes: Locate the target video stream corresponding to the action sequence indicated by the abnormal behavior; Identify the first position coordinates of the target customer's hand in the camera coordinate system in the target video stream, and convert the first position coordinates into the second position coordinates in the map coordinate system of the physical store; Based on the second location coordinates and the shelf layout data in the map coordinate system of the physical store, determine the location of the target shelf where the target customer will interact. Based on the target shelf location, the corresponding product is determined from the product storage database as the target product.

11. The method according to claim 1, wherein performing customer behavior analysis based on the behavior types of the multiple customers interacting with the product comprises: Based on the types of interactions between the multiple customers and the products, at least one of the following data is collected: the number of interactions, duration of interactions, frequency of interactions, results of interactions, and changes in the quantity of products for the same type of interaction among the multiple customers. And / or, search for behavior types that correspond to the type of product retrieval interaction, and statistically analyze customer characteristics and / or retrieval preferences.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-11.

13. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-11.

14. A computer program product, wherein, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

Images