Retail store passenger flow statistical method, device, equipment, medium and product

Abstract

The invention discloses a retail store passenger flow statistical method, device and equipment, a medium and a product, and relates to the technical field of computer vision. According to the method, firstly, a single-stage target detection algorithm is adopted to carry out human body head and shoulder detection processing on video frame images in store import and export video data, and human body head and shoulder detection results are obtained; and according to the corresponding current intra-frame position information / and the historical intra-frame motion trail of the bound target, predicting to obtain a future intra-frame motion trail of the corresponding bound target, tracking the bound target in real time, and then determining a single store passenger flow behavior of the bound target according to a single tracking result of the bound target. And finally, according to the human body head and shoulder features of the binding target, judging whether the target is a known shop assistant and / or a repeated behavior target, and if not, adding 1 to the customer flow volume of the store, so that the customer authenticity of the passenger flow statistical data can be ensured.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a method, device, equipment, medium and product for counting customer flow in a retail store. Background Art

[0002] At present, in the operation and management activities of retail stores such as drug stores, clear and accurate customer flow statistics are needed to formulate appropriate store operation strategies. However, the existing retail store customer flow statistics solutions have the following defects: (1) Usually, one camera is used inside the store to count the customer flow data entering the store, and another camera is used outside the store to count the customer flow data leaving the store and / or the customer flow data passing through the store, which requires a large cost for the camera hardware; (2) Specific groups of people (such as store employees and / or people who perform the same and repeated customer behaviors such as entering, leaving or passing through the store in a short period of time) are not filtered, resulting in the possibility that the customer flow statistics obtained may contain non-real customers; (3) The customer flow statistics task is mainly completed on the cloud server side, which requires the store-side camera to upload surveillance video data in real time, which requires a high bandwidth for the transmission network and also poses a data security risk. Summary of the invention

[0003] The purpose of the present invention is to provide a method, device, computer equipment, computer-readable storage medium and computer program product for counting customer flow in retail stores, so as to solve the problems existing in existing solutions for counting customer flow in retail stores, such as high cost of camera hardware, possible non-real customers in the obtained customer flow statistics data, high bandwidth requirements for transmission networks and potential data security risks.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] In a first aspect, a method for counting customer flow in a retail store is provided, which is executed by an edge computing host located in the retail store, and includes:

[0006] Acquire video data collected in real time by a store camera of an entrance and exit area of ​​the retail store, wherein the store camera is fixedly installed so that the lens field of view covers the entrance and exit area;

[0007] Using a single-stage target detection algorithm to perform human head and shoulder detection processing on the video frame images in the video data to obtain a human head and shoulder detection result;

[0008] For each human head and shoulder mark frame in the human head and shoulder detection result, predict the future intra-frame motion trajectory of the corresponding binding target according to the corresponding current intra-frame position information / and the historical intra-frame motion trajectory of the binding target, wherein the binding target refers to the tracking target bound to the human head and shoulder mark frame;

[0009] According to the future intra-frame motion trajectory of the binding target, combined with the new human head and shoulder detection result obtained based on the new video data, the binding target is tracked in real time;

[0010] Determine a single store customer traffic behavior of the bound target according to a single tracking result of the bound target, wherein the single tracking result refers to a movement trajectory obtained by tracking the bound target from appearance to disappearance in the entrance and exit area, and the single store customer traffic behavior refers to a single store entry behavior, a single store exit behavior, or a single store passing behavior;

[0011] An image within a human head and shoulder mark frame of the binding target is captured from the video data, and a human head and shoulder feature extraction process is performed on the image within the human head and shoulder mark frame to obtain human head and shoulder features of the binding target;

[0012] It is determined whether the bound target is a known store clerk and / or a repeated behavior target based on the human head and shoulder features of the bound target. If not, after determining the single store traffic behavior of the bound target, the store traffic of the retail store is incremented by 1, wherein the repeated behavior target refers to a tracking target that repeats the same store traffic behavior at least twice within a preset time period.

[0013] Based on the above invention content, a new solution for retail store customer flow statistics based on computer vision technology is provided, that is, firstly, a single-stage target detection algorithm is used to perform human head and shoulder detection processing on the video frame images in the store entrance and exit video data to obtain the human head and shoulder detection results, and then for each human head and shoulder mark box in the detection result, according to the corresponding current frame position information / and the historical frame motion trajectory of the bound target, the corresponding future frame motion trajectory of the bound target is predicted, and the bound target is tracked in real time, and then according to the single tracking result of the bound target, the single store customer flow behavior of the bound target is determined, and the human head and shoulder features of the bound target are extracted, and finally, according to the human head and shoulder features of the bound target, it is determined whether it is a known clerk and / or a repeated behavior target, and if not, the store customer flow of the retail store is added by 1. In this way, not only can one camera be used to realize the statistics of in-store, out-store and passing-store customer flow, reducing the cost of camera hardware, but also filtering of specific groups can be realized, ensuring the customer authenticity of the customer flow statistics data, and because the customer flow statistics task is completed on the edge side of the store, it is also unnecessary to upload video data, reducing the bandwidth requirements for the transmission network and potential data security risks, which is convenient for practical application and promotion.

[0014] In a possible design, a single-stage target detection algorithm is used to perform human head and shoulder detection processing on the video frame images in the video data to obtain a human head and shoulder detection result, including:

[0015] The video frame images in the video data are imported into a human head and shoulder recognition model pre-trained based on the YOLO target detection algorithm, and the human head and shoulder detection result is output.

[0016] In a possible design, for each human head and shoulder mark frame in the human head and shoulder detection result, according to the corresponding current frame position information / and the historical frame motion trajectory of the bound target, the future frame motion trajectory of the corresponding bound target is predicted, including:

[0017] For each human head and shoulder marker frame in the human head and shoulder detection result, the corresponding current in-frame position information / and the historical in-frame motion trajectory of the bound target are imported into the motion model, and the future in-frame motion trajectory of the corresponding bound target is output, wherein the bound target refers to the tracking target bound to the human head and shoulder marker frame, and the motion model is a physical motion calculation model based on the principle of uniform motion or the principle of uniform acceleration motion, or a motion trajectory prediction model pre-trained based on a machine learning algorithm.

[0018] In a possible design, according to the future intra-frame motion trajectory of the binding target and in combination with the new human head and shoulder detection result obtained based on the new video data, the binding target is tracked in real time, including:

[0019] According to the future intra-frame motion trajectory of the bound target and the tracking frame of the bound target in the current most recent K frames, the tracking frame of the bound target in the current next frame is predicted by using the Kalman filter method, where K represents a positive integer;

[0020] Obtaining at least one human head and shoulder marking frame in the current next frame according to the new human head and shoulder detection result obtained based on the new video data;

[0021] The Hungarian algorithm and the DIoU index are used to optimally match the tracking frame of the binding target in the current next frame with the at least one human head and shoulder marker frame. If the tracking frame of the binding target in the current next frame matches a certain human head and shoulder marker frame in the at least one human head and shoulder marker frame, the certain human head and shoulder marker frame is bound to the binding target, and according to the tracking frame of the binding target in the current most recent K frames and the certain human head and shoulder marker frame, the Kalman filter method is used to update the tracking frame of the binding target in the current next frame, so as to continue tracking the binding target, otherwise the tracking of the binding target is terminated.

[0022] In a possible design, the image in the human head and shoulder mark frame is subjected to human head and shoulder feature extraction processing to obtain the human head and shoulder features of the binding target, including:

[0023] The image in the human head and shoulder mark frame is imported into a human head and shoulder feature extraction model pre-trained based on a neural network and an unsupervised learning method, and the human head and shoulder features of the binding target are output.

[0024] In a possible design, judging whether the binding target is a known clerk according to the head and shoulder features of the binding target includes:

[0025] reading head and shoulder features of employees of the retail store;

[0026] Calculate the first cosine distance between the head and shoulder features of the human body of the binding target and the head and shoulder features of the clerk;

[0027] Determine whether the first cosine distance is less than or equal to a preset first distance threshold, if so, determine that the binding target is a known clerk, otherwise determine that the binding target is not a known clerk;

[0028] And / or, judging whether the binding target is a repetitive behavior target according to the head and shoulder features of the binding target, including:

[0029] Determine whether there is at least one historical tracking target that has ended tracking within the most recent preset time period. If so, read the head and shoulder features of the at least one historical tracking target. Otherwise, directly determine that the bound target is not a repeated behavior target, wherein the repeated behavior target refers to a tracking target that repeats the same store customer behavior at least twice within the preset time period.

[0030] For each historical tracking target in the at least one historical tracking target, calculating a second cosine distance between a corresponding human head and shoulder feature and a human head and shoulder feature of the binding target;

[0031] It is determined whether all the second cosine distances are greater than a preset second distance threshold; if so, it is determined that the binding target is not a repeated behavior target; otherwise, it is determined that the binding target is a repeated behavior target.

[0032] In a second aspect, a retail store customer flow counting device is provided, which is suitable for being arranged in an edge computing host located in a retail store, and includes a video data acquisition unit, a human head and shoulder detection unit, a motion trajectory prediction unit, a binding target tracking unit, a customer flow behavior determination unit, a head and shoulder feature extraction unit, and a store customer flow deduplication unit;

[0033] The video data acquisition unit is used to acquire video data collected by a store camera in real time from the entrance and exit area of ​​the retail store, wherein the store camera is fixedly installed and the lens field of view covers the entrance and exit area;

[0034] The human head and shoulder detection unit is communicatively connected to the video data acquisition unit and is used to perform human head and shoulder detection processing on the video frame images in the video data using a single-stage target detection algorithm to obtain a human head and shoulder detection result;

[0035] The motion trajectory prediction unit is communicatively connected to the human head and shoulder detection unit, and is used to predict the future intra-frame motion trajectory of the corresponding binding target for each human head and shoulder mark frame in the human head and shoulder detection result according to the corresponding current intra-frame position information / and the historical intra-frame motion trajectory of the binding target, wherein the binding target refers to the tracking target bound to the human head and shoulder mark frame;

[0036] The binding target tracking unit is communicatively connected to the human head and shoulder detection unit and the motion trajectory prediction unit, respectively, and is used to track the binding target in real time according to the future intra-frame motion trajectory of the binding target in combination with the new human head and shoulder detection result obtained based on the new video data;

[0037] The customer behavior determination unit is communicatively connected to the bound target tracking unit, and is used to determine a single store customer behavior of the bound target according to a single tracking result of the bound target, wherein the single tracking result refers to a movement trajectory obtained by tracking the bound target from appearance to disappearance in the entrance and exit area, and the single store customer behavior refers to a single store entry behavior, a single store exit behavior, or a single store passing behavior;

[0038] The head and shoulder feature extraction unit is communicatively connected to the human head and shoulder detection unit, and is used to intercept the image in the human head and shoulder mark frame of the binding target from the video data, and perform human head and shoulder feature extraction processing on the image in the human head and shoulder mark frame to obtain the human head and shoulder features of the binding target;

[0039] The store customer flow deduplication unit is communicatively connected to the head and shoulder feature extraction unit and the customer flow behavior determination unit, respectively, and is used to determine whether the binding target is a known store clerk and / or a repeated behavior target based on the human head and shoulder features of the binding target. If not, after determining the single store customer flow behavior of the binding target, the store customer flow of the retail store is automatically increased by 1, wherein the repeated behavior target refers to a tracking target that repeats the same store customer flow behavior at least twice within a preset time period.

[0040] In a third aspect, the present invention provides a computer device comprising a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the retail store customer flow counting method as described in the first aspect or any possible design of the first aspect.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, executes the method for counting customer flow in a retail store as described in the first aspect or any possible design of the first aspect.

[0042] In a fifth aspect, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the retail store customer flow counting method as described in the first aspect or any possible design of the first aspect.

[0043] Beneficial effects of the above scheme:

[0044] (1) The present invention provides a new solution for counting customer flow in retail stores based on computer vision technology, namely, firstly, a single-stage target detection algorithm is used to perform human head and shoulder detection processing on the video frame images in the store entrance and exit video data to obtain the human head and shoulder detection results, and then, for each human head and shoulder mark frame in the detection result, the corresponding current frame position information / and the historical frame motion trajectory of the bound target are used to predict the corresponding future frame motion trajectory of the bound target, and the bound target is tracked in real time, and then, according to the single tracking result of the bound target, the single store customer flow behavior of the bound target is determined, and the human head and shoulder features of the bound target are extracted, and finally, according to the human head and shoulder features of the bound target, it is determined whether it is a known store clerk and / or a repeated behavior target, and if not, the store customer flow of the retail store is automatically increased by 1. In this way, not only can one camera be used to realize the statistics of customer flow entering, exiting and passing through the store, reducing the cost of camera hardware, but also filtering specific groups of people, ensuring the authenticity of customers in the customer flow statistics data, and because the customer flow statistics task is completed at the edge of the store, there is no need to upload video data, reducing the bandwidth requirements for the transmission network and potential data security risks, which is convenient for practical application and promotion;

[0045] (2) For retail pharmacies, it can focus more on solving problems such as the attire for special positions in pharmacies and the frequency of entering and leaving the store, making it particularly suitable for the pharmaceutical retail field;

[0046] (3) By improving the existing multi-target tracking feature matching method, the impact of the change of the unique identification of the tracking target caused by occlusion on passenger flow statistics can be reduced;

[0047] (4) It can be beneficial to combine container technology to encapsulate pre-processing, model reasoning and post-processing so that they can be easily deployed on edge computing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a method for counting customer flow in a retail store provided in an embodiment of the present application.

[0050] Figure 2 An example diagram of a video frame image in video data is provided for an embodiment of the present application.

[0051] Figure 3 A schematic diagram of the structure of a retail store customer flow counting device provided in an embodiment of the present application.

[0052] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structures of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these embodiments without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0054] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0055] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can indicate three situations: A exists alone, B exists alone, or A and B exist at the same time. For another example, A, B and / or C can indicate the existence of any one of A, B and C or any combination of them. The term " / and" that may appear in this document describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can indicate two situations: A exists alone or A and B exist at the same time. In addition, the character " / " that may appear in this document generally indicates that the previous and next associated objects are in an "or" relationship.

[0056] Example

[0057] like Figure 1 As shown, the retail store customer flow counting method provided in the first aspect of this embodiment can be, but is not limited to, executed by an edge computing host with certain computing resources and located in a retail store, such as a personal computer (PC, a multi-purpose computer with a size, price and performance suitable for personal use; desktops, laptops, small laptops, tablets and ultrabooks are all personal computers), smart phones, personal digital assistants (PDA) or wearable devices and other electronic devices that are connected to the store camera. Figure 1 As shown, the retail store customer flow counting method may include but is not limited to the following steps S1 to S7.

[0058] S1. Obtain video data collected in real time by a store camera at the entrance and exit area of ​​the retail store, wherein the store camera is fixedly installed and the lens field of view covers the entrance and exit area.

[0059] In step S1, the store camera refers to a surveillance camera installed inside or outside the retail store (specifically, it can be installed on the top of the entrance and exit area or the top around it). The specific method of collecting and transmitting the video data to the edge computing host is a prior art method. In addition, the video frame image in the video data is exemplified by Figure 2 shown.

[0060] S2. Use a single-stage target detection algorithm to perform human head and shoulder detection processing on the video frame images in the video data to obtain a human head and shoulder detection result.

[0061] In the step S2, the target detection algorithm is an existing artificial intelligence recognition algorithm for identifying objects in a picture and marking the position of the objects, and specifically includes but is not limited to Faster R-CNN (Faster Regions with Convolutional Neural Networks features, a target detection algorithm proposed by He Kaiming et al. in 2015, which won multiple first places in the 2015 ILSVRV and COCO competitions) target detection algorithm, SSD (Single Shot MultiBox Detector, a target detection algorithm proposed by Wei Liu at ECCV 2016, which is one of the currently popular main detection frameworks) target detection algorithm or YOLO (You only look once, currently the latest version has been developed to V5, and is also widely used in the industry. Its basic principle is: first, the input image is divided into 7×7 grids, 2 borders are predicted for each grid, and then the target windows with relatively low probability are removed according to the threshold, and finally the redundant windows are removed by border merging to obtain the detection result) target detection algorithm, etc. The single-stage target detection algorithm refers to a target detection algorithm that can achieve target detection by extracting features only once and is faster than a multi-stage algorithm (of course, the accuracy is slightly lower), such as the YOLO algorithm. Therefore, the human head and shoulder detection process can be performed based on the single-stage target detection algorithm to obtain the human head and shoulder detection result.

[0062] In the step S2, specifically, a single-stage target detection algorithm is used to perform human head and shoulder detection processing on the video frame images in the video data to obtain human head and shoulder detection results, including but not limited to: importing the video frame images in the video data into a human head and shoulder recognition model pre-trained based on the YOLO target detection algorithm, and outputting the human head and shoulder detection results. The aforementioned YOLO target detection algorithm preferably uses the YOLO V5 network, which is a new type of neural network for building a target detection model, which is used to identify and locate objects in videos or images, that is, using a convolutional neural network to learn the features of objects in the image, and using multi-scale prediction and grid segmentation to detect and locate targets; it consists of three parts, namely, a backbone network backbone, a neck network neck, and a head network head. The backbone network Backbone can use a CSPDarknet53 (CSP stands for Cross Stage Partial) network for feature extraction. The neck network is composed of SPP (Spatial Pyramid Pooling block) blocks and PANet (Path Aggregation Network) networks. The former is used to increase the receptive field and separate the most important features, and the latter is used to ensure that semantic features are received from high-level layers and fine-grained features are received from low-level layers of the lateral backbone network at the same time. The head network head is based on anchor box detection and detects three different sizes of 13×13, 26×26 and 52×52 feature maps, which are used to detect targets from large to small (here, feature maps with large sizes contain more information, so feature maps with a size of 52×52 are used to detect small targets, and vice versa). Therefore, the pre-training process of the human head and shoulders recognition model can conventionally include the following three steps: first, obtain a store customer flow head and shoulders dataset that has been annotated (which specifically consists of original pictures and annotation files, wherein there are several portraits in the original picture, and the annotation file stores the head and shoulders position, size and category information of the portraits in the picture, so that one picture corresponds to one annotation file; an accumulation of more than about 20,000 pictures and corresponding annotation files can constitute the store customer flow head and shoulders dataset); then build a model based on the YOLO V5 network, and modify the head network head (the purpose of the modification is to speed up the calculation speed, and the main modification will be the number of output channels of the fully connected layer and the structure of the detection head, etc.); finally, use the store customer flow head and shoulders dataset to perform conventional model training on the built model (specifically, some evaluation indicators will be calculated, and these evaluation indicators will be used to evaluate the model performance on the verification set during the model training and verification process, etc.), and obtain the human head and shoulders recognition model.

[0063] S3. For each human head and shoulder mark frame in the human head and shoulder detection result, the future intra-frame motion trajectory of the corresponding binding target is predicted according to the corresponding current intra-frame position information / and the historical intra-frame motion trajectory of the binding target, wherein the binding target refers to the tracking target bound to the human head and shoulder mark frame.

[0064] In step S3, since each human head and shoulder mark frame corresponds to a person appearing in the entrance and exit area (such as a person entering the store, a person leaving the store, a person passing through the store, or a clerk, etc.), each human head and shoulder mark frame will be bound to a tracking target; the specific binding method can be but is not limited to giving the human head and shoulder mark frame a unique identifier so that it can be used as a tracking target. The current intra-frame position information refers to the coordinate information of the human head and shoulder mark frame in the current video frame image; the historical intra-frame motion trajectory of the binding target refers to the coordinate information combination of the human head and shoulder mark frame corresponding to the binding target in the historical video frame image (for example, the current multiple most recent historical video frame images) (if the human head and shoulder mark frame in the current video frame image is bound to a tracking target for the first time, there will be no such historical trajectory, so that the future intra-frame motion trajectory of the binding target can only be predicted based on the current intra-frame position information; otherwise, there is such a historical trajectory, so that the future intra-frame motion trajectory of the binding target can be predicted based on the current intra-frame position information and the historical intra-frame motion trajectory of the binding target); the future intra-frame motion trajectory of the binding target refers to the coordinate information combination of the human head and shoulder mark frame corresponding to the binding target in the future video frame image (for example, the current next future video frame image or the current multiple most recent future video frame images).

[0065] In the step S3, specifically, for each human head and shoulder mark frame in the human head and shoulder detection result, according to the corresponding current frame position information / and the historical frame motion trajectory of the bound target, the future frame motion trajectory of the corresponding bound target is predicted, including but not limited to: for each human head and shoulder mark frame in the human head and shoulder detection result, the corresponding current frame position information / and the historical frame motion trajectory of the bound target are imported into the motion model, and the future frame motion trajectory of the corresponding bound target is output, wherein the bound target refers to the tracking target bound to the human head and shoulder mark frame, and the motion model is a physical motion calculation model based on the principle of uniform motion or uniform acceleration motion or a motion trajectory prediction model pre-trained based on a machine learning algorithm. The aforementioned machine learning algorithm is a core artificial intelligence algorithm that specifically studies how computers simulate or realize human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structures to continuously improve their own performance, and is the fundamental way to make computers intelligent; specifically, the machine learning algorithm preferably adopts a linear regression algorithm based on the Python sklearn library to quickly and accurately find the rules in the data. Therefore, based on a certain amount of model input data (i.e., the historical position information of the human head and shoulder mark box in the historical human head and shoulder detection results in the current frame / and the historical motion trajectory of the historical binding target in the frame) and model output data (i.e., the historical motion trajectory of the historical binding target in the frame after the current frame), the conventional calibration verification modeling method (the specific process includes the calibration process and the verification process of the model, that is, the process of first comparing the model simulation results with the measured data, and then adjusting the model parameters according to the comparison results so that the simulation results are consistent with the actual results) is used to pre-train the motion trajectory prediction model. Preferably, in the calibration verification modeling process of the motion trajectory prediction model, the Bayesian optimization algorithm based on the tree structure is used to tune the model parameters. In addition, the machine learning algorithm can also be, but not limited to, artificial intelligence algorithms such as support vector machines, K nearest neighbor method, stochastic gradient descent method, multi-layer perceptron, decision tree, back propagation neural network or radial basis function network.

[0066] S4. According to the future intra-frame motion trajectory of the binding target, combined with the new human head and shoulder detection result obtained based on the new video data, the binding target is tracked in real time.

[0067] In step S4, since there may be multiple binding targets at the same time, the existing deepsort target tracking algorithm can be specifically used to perform multi-target tracking on the multiple binding targets. In order to reduce the problem of early tracking termination caused by occlusion during multi-target tracking, preferably, according to the future intra-frame motion trajectory of the binding target, combined with the new human head and shoulder detection result obtained based on the new video data, the binding target is tracked in real time, including but not limited to the following steps S41 to S43.

[0068] S41. According to the future intra-frame motion trajectory of the bound target and the tracking frame of the bound target in the current most recent K frames, the Kalman filter method is used to predict the tracking frame of the bound target in the current next frame, where K represents a positive integer.

[0069] In the step S41, the Kalman filter method is a commonly used method in existing target tracking algorithms, so the specific prediction process can be derived based on the conventional prior art and will not be repeated here. The current most recent K frames include the current frame and the previous frames (if any). In addition, if the human head and shoulder marker frame in the current video frame image is bound to a tracking target for the first time, the human head and shoulder marker frame can be initially used as the tracking frame of the tracking target in the current frame (i.e., there is no previous frame or previous frames at this time).

[0070] S42. Obtain at least one human head and shoulder marking frame in the current next frame according to the new human head and shoulder detection result obtained based on the new video data.

[0071] S43. The Hungarian algorithm and the DIoU index are used to optimally match the tracking frame of the binding target in the current next frame with the at least one human head and shoulder marker frame. If the tracking frame of the binding target in the current next frame matches a certain human head and shoulder marker frame in the at least one human head and shoulder marker frame, the certain human head and shoulder marker frame is bound to the binding target, and according to the tracking frame of the binding target in the current most recent K frames and the certain human head and shoulder marker frame, the Kalman filter method is used to update the tracking frame of the binding target in the current next frame so as to continue tracking the binding target, otherwise the tracking of the binding target is terminated.

[0072] In the step S43, the Hungarian algorithm is a commonly used method in existing target tracking algorithms, and the DIoU index is a commonly used index in existing target tracking algorithms. Therefore, the specific process of performing the optimal matching can be conventionally derived based on the prior art, and will not be repeated here. In addition, if there is any human head and shoulder marker frame in the at least one human head and shoulder marker frame that has never been matched and bound to a certain tracking target, a new unique identifier can be assigned to the any human head and shoulder marker frame so as to serve as a new tracking target (i.e., a new binding target is obtained).

[0073] S5. Determine a single store customer traffic behavior of the bound target based on a single tracking result of the bound target, wherein the single tracking result refers to a movement trajectory obtained by tracking the bound target from appearance to disappearance in the import and export area, and the single store customer traffic behavior may refer to but is not limited to a single store entry behavior, a single store exit behavior, or a single store passing behavior, etc.

[0074] In step S5, for example, if a tracking target (that is, a binding target) appears in the entrance and exit area at 10:0:21 and does not disappear until 10:0:51, then the single tracking result of the tracking target is the motion trajectory obtained from the tracking from 10:0:21 to 10:0:51, and if the direction of the motion trajectory obtained by tracking is from outside the door to inside the door, it can be determined that the single store customer behavior of the tracking target is a single entry behavior; if the direction of the motion trajectory obtained by tracking is from inside the door to outside the door, it can be determined that the single store customer behavior of the tracking target is a single exit behavior; if the direction of the motion trajectory obtained by tracking is from one side outside the door to the other side outside the door, it can be determined that the single store customer behavior of the tracking target is a single store passing behavior.

[0075] S6. The image within the human head and shoulder mark frame of the binding target is captured from the video data, and a human head and shoulder feature extraction process is performed on the image within the human head and shoulder mark frame to obtain the human head and shoulder features of the binding target.

[0076] In the step S6, considering that the video data is composed of a number of video frame images, multiple human head and shoulder mark frame images of the binding target are obtained during a single tracking process of the binding target. In this way, the clarity of the multiple human head and shoulder mark frame images can be calculated respectively, but not limited to, by using an existing image clarity algorithm, and a human head and shoulder mark frame image with the highest clarity is selected from the multiple human head and shoulder mark frame images as the target image to be feature extracted. The specific method of the aforementioned human head and shoulder feature extraction process can preferably be implemented using a neural network (that is, the neural network can be regarded as a mapping relationship, an image is input, and an output feature corresponding to the input image is obtained through the mapping relationship), that is, preferably, the human head and shoulder feature extraction process is performed on the human head and shoulder mark frame image to obtain the human head and shoulder features of the binding target, including but not limited to: importing the human head and shoulder mark frame image into a human head and shoulder feature extraction model pre-trained based on a neural network and an unsupervised learning method, and outputting the human head and shoulder features of the binding target. The aforementioned neural network specifically but not limited to adopting the existing Resnet50 network, and the aforementioned unsupervised learning method specifically but not limited to adopting the existing MoCo v3 training method, so as to achieve the purpose of reducing computing power requirements and training time by using the self-distillation method to train the model.

[0077] S7. Determine whether the bound target is a known store clerk and / or a repeated behavior target based on the human head and shoulder features of the bound target. If not, after determining the single store traffic behavior of the bound target, the store traffic of the retail store is incremented by 1, wherein the repeated behavior target refers to a tracking target that repeats the same store traffic behavior at least twice within a preset time period.

[0078] In step S7, considering that the clerks are not customers, it is necessary to remove duplicate clerks in this step. Specifically, judging whether the binding target is a known clerk based on the head and shoulder features of the binding target includes but is not limited to the following steps S711 to S713.

[0079] S711. Read the head and shoulder features of the store clerks in the retail store.

[0080] In the step S711, the head and shoulder features of the clerks in the retail store can be extracted in advance based on the head and shoulder images of the clerks in the retail store with reference to the aforementioned step S6, and stored locally, so that they can be directly read and used.

[0081] S712. Calculate and obtain the first cosine distance between the human head and shoulder features of the binding target and the head and shoulder features of the clerk.

[0082] In step S712, the cosine distance is a method for measuring the similarity between two vectors, which is calculated based on the complement of the cosine similarity (its value range is between 0 and 2, the closer to 0, the more similar the two vectors are, and the closer to 2, the less similar they are). Therefore, the human head and shoulder features of the binding target and the head and shoulder features of the clerk can be used as two vectors in one-to-one correspondence to calculate the first cosine distance.

[0083] S713. Determine whether the first cosine distance is less than or equal to a preset first distance threshold; if so, determine that the binding target is a known clerk; otherwise, determine that the binding target is not a known clerk.

[0084] In the step S713, the first distance threshold may be, but is not limited to, 1.

[0085] In the step S7, it is also considered that in the store scenario, there will be people lingering at the door. If they are always in the import and export area, their unique identification (i.e. as a binding target) remains unchanged, and the purpose of customer deduplication can be achieved through the aforementioned step S5. However, if they frequently enter and exit the import and export area (i.e. appear and disappear in a short period of time, and then appear and disappear again), it is impossible to deduplicate customers through the aforementioned step S5. At this time, it will be considered that there are multiple customer flows, resulting in the possibility that the obtained customer flow statistics may contain non-real customers. Therefore, it is necessary to deduplicate repeated behavior targets through this step. Specifically, judging whether the binding target is a repeated behavior target based on the human head and shoulder features of the binding target includes but is not limited to the following steps S721 to S723.

[0086] S721. Determine whether there is at least one historical tracking target that has ended tracking within the most recent preset time period. If so, read the human head and shoulder features of the at least one historical tracking target. Otherwise, directly determine that the bound target is not a repeated behavior target, wherein the repeated behavior target refers to a tracking target that repeats the same store customer behavior at least twice within the preset time period.

[0087] In step S721, the preset time length may be, but is not limited to, 30 minutes or 1 hour. Since each historical tracking target undergoes the aforementioned step S6, the corresponding human head and shoulder features can be obtained and cached locally, so they can be directly read and used.

[0088] S722. For each historical tracking target in the at least one historical tracking target, calculate a second cosine distance between the corresponding human head and shoulder feature and the human head and shoulder feature of the binding target.

[0089] S723. Determine whether all the second cosine distances are greater than a preset second distance threshold; if so, determine that the binding target is not a repeated behavior target; otherwise, determine that the binding target is a repeated behavior target.

[0090] In the step S723, the second distance threshold may be, but is not limited to, 1. In addition, the second distance threshold may be equal to or different from the first distance threshold.

[0091] In step S7, the store customer flow specifically includes store entry customer flow, store exit customer flow, store pass customer flow, etc. Specifically, after determining that the single store customer flow behavior of the binding target is a single store entry behavior, the store entry customer flow of the retail store is incremented by 1; after determining that the single store customer flow behavior of the binding target is a single store exit behavior, the store exit customer flow of the retail store is incremented by 1; after determining that the single store customer flow behavior of the binding target is a single store pass behavior, the store pass customer flow of the retail store is incremented by 1.

[0092] Therefore, based on the retail store customer flow counting method described in the aforementioned steps S1 to S7, a new retail store customer flow counting solution based on computer vision technology is provided, that is, firstly, a single-stage target detection algorithm is used to perform human head and shoulder detection processing on the video frame images in the store entrance and exit video data to obtain the human head and shoulder detection results, and then for each human head and shoulder mark box in the detection result, according to the corresponding current frame position information / and the historical frame motion trajectory of the bound target, the future frame motion trajectory of the corresponding bound target is predicted, and the bound target is tracked in real time, and then according to the single tracking result of the bound target, the bound target is confirmed. The single-time store customer behavior of the binding target is determined, and the human head and shoulder features of the binding target are extracted. Finally, according to the human head and shoulder features of the binding target, it is determined whether it is a known store clerk and / or a repeated behavior target. If not, the store customer flow of the retail store is automatically increased by 1. In this way, not only can one camera be used to realize the statistics of in-store, out-store and passing-store customer flow, reducing the cost of camera hardware, but also specific groups of people can be filtered to ensure the authenticity of the customer flow statistics data. In addition, since the customer flow statistics task is completed at the edge of the store, there is no need to upload video data, reducing the bandwidth requirements for the transmission network and potential data security risks. In addition, the aforementioned retail store customer flow statistics method can be combined with existing container technology, that is, all operations of the retail store customer flow statistics method are first encapsulated to form an image; an edge computing host is installed in the retail store, and the edge computing hosts of all retail stores are connected to the cloud server for communication; the packaged image is sent to the edge computing host of each retail store through the cloud server; the edge computing host of each retail store runs the image, which distributes the computing workload to the edge side of the store, and can greatly reduce the computing workload of the cloud server; the cloud server only needs to subscribe to messages and receive customer flow statistics results.

[0093] like Figure 3 As shown, the second aspect of this embodiment provides a virtual device for implementing the retail store customer flow statistics method described in the first aspect, which is suitable for being arranged in an edge computing host located in a retail store, and includes a video data acquisition unit, a human head and shoulder detection unit, a motion trajectory prediction unit, a binding target tracking unit, a customer flow behavior determination unit, a head and shoulder feature extraction unit, and a store customer flow deduplication unit;

[0094] The video data acquisition unit is used to acquire video data collected by a store camera in real time from the entrance and exit area of ​​the retail store, wherein the store camera is fixedly installed and the lens field of view covers the entrance and exit area;

[0095] The human head and shoulder detection unit is communicatively connected to the video data acquisition unit and is used to perform human head and shoulder detection processing on the video frame images in the video data using a single-stage target detection algorithm to obtain a human head and shoulder detection result;

[0096] The motion trajectory prediction unit is communicatively connected to the human head and shoulder detection unit, and is used to predict the future intra-frame motion trajectory of the corresponding binding target for each human head and shoulder mark frame in the human head and shoulder detection result according to the corresponding current intra-frame position information / and the historical intra-frame motion trajectory of the binding target, wherein the binding target refers to the tracking target bound to the human head and shoulder mark frame;

[0097] The binding target tracking unit is communicatively connected to the human head and shoulder detection unit and the motion trajectory prediction unit, respectively, and is used to track the binding target in real time according to the future intra-frame motion trajectory of the binding target in combination with the new human head and shoulder detection result obtained based on the new video data;

[0098] The customer behavior determination unit is communicatively connected to the bound target tracking unit, and is used to determine a single store customer behavior of the bound target according to a single tracking result of the bound target, wherein the single tracking result refers to a movement trajectory obtained by tracking the bound target from appearance to disappearance in the entrance and exit area, and the single store customer behavior refers to a single store entry behavior, a single store exit behavior, or a single store passing behavior;

[0099] The head and shoulder feature extraction unit is communicatively connected to the human head and shoulder detection unit, and is used to intercept the image in the human head and shoulder mark frame of the binding target from the video data, and perform human head and shoulder feature extraction processing on the image in the human head and shoulder mark frame to obtain the human head and shoulder features of the binding target;

[0100] The store customer flow deduplication unit is communicatively connected to the head and shoulder feature extraction unit and the customer flow behavior determination unit, respectively, and is used to determine whether the binding target is a known store clerk and / or a repeated behavior target based on the human head and shoulder features of the binding target. If not, after determining the single store customer flow behavior of the binding target, the store customer flow of the retail store is automatically increased by 1, wherein the repeated behavior target refers to a tracking target that repeats the same store customer flow behavior at least twice within a preset time period.

[0101] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be referred to the retail store customer flow counting method described in the first aspect, and will not be described in detail here.

[0102] like Figure 4As shown, the third aspect of this embodiment provides a computer device for executing the method for counting customer flow in retail stores as described in the first aspect, including a memory, a processor and a transceiver that are sequentially connected in communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the method for counting customer flow in retail stores as described in the first aspect. For example, the memory may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first-in-first-out memory (FIFO) and / or a first-in-last-out memory (FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen and other necessary components.

[0103] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be found in the retail store customer flow counting method described in the first aspect, and will not be described in detail here.

[0104] In a fourth aspect of this embodiment, there is provided a computer-readable storage medium storing instructions including the method for counting customer flow in retail stores as described in the first aspect, that is, the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the method for counting customer flow in retail stores as described in the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0105] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be referred to the retail store customer flow counting method described in the first aspect, and will not be repeated here.

[0106] In a fifth aspect, the present embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the method for counting customer flow in a retail store as described in the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0107] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for counting customer flow in a retail store, characterized in that: Executed by edge computing hosts located in retail stores, including: Acquire video data collected in real time by a store camera of an entrance and exit area of ​​the retail store, wherein the store camera is fixedly installed so that the lens field of view covers the entrance and exit area; Using a single-stage target detection algorithm to perform human head and shoulder detection processing on the video frame images in the video data to obtain a human head and shoulder detection result; For each human head and shoulder mark frame in the human head and shoulder detection result, predict the future intra-frame motion trajectory of the corresponding binding target according to the corresponding current intra-frame position information / and the historical intra-frame motion trajectory of the binding target, wherein the binding target refers to the tracking target bound to the human head and shoulder mark frame; According to the future intra-frame motion trajectory of the binding target, combined with the new human head and shoulder detection result obtained based on the new video data, the binding target is tracked in real time; Determine a single store customer traffic behavior of the bound target according to a single tracking result of the bound target, wherein the single tracking result refers to a movement trajectory obtained by tracking the bound target from appearance to disappearance in the entrance and exit area, and the single store customer traffic behavior refers to a single store entry behavior, a single store exit behavior, or a single store passing behavior; An image within a human head and shoulder mark frame of the binding target is captured from the video data, and a human head and shoulder feature extraction process is performed on the image within the human head and shoulder mark frame to obtain human head and shoulder features of the binding target; It is determined whether the bound target is a known store clerk and / or a repeated behavior target based on the human head and shoulder features of the bound target. If not, after determining the single store traffic behavior of the bound target, the store traffic of the retail store is incremented by 1, wherein the repeated behavior target refers to a tracking target that repeats the same store traffic behavior at least twice within a preset time period.

2. The method for counting customer flow in a retail store according to claim 1, characterized in that: A single-stage target detection algorithm is used to perform human head and shoulder detection processing on the video frame images in the video data to obtain a human head and shoulder detection result, including: The video frame images in the video data are imported into a human head and shoulder recognition model pre-trained based on the YOLO target detection algorithm, and the human head and shoulder detection result is output.

3. The method for counting customer flow in a retail store according to claim 1, characterized in that: For each human head and shoulder mark frame in the human head and shoulder detection result, according to the corresponding current frame position information / and the historical frame motion trajectory of the binding target, the future frame motion trajectory of the corresponding binding target is predicted, including: For each human head and shoulder marker frame in the human head and shoulder detection result, the corresponding current in-frame position information / and the historical in-frame motion trajectory of the bound target are imported into the motion model, and the future in-frame motion trajectory of the corresponding bound target is output, wherein the bound target refers to the tracking target bound to the human head and shoulder marker frame, and the motion model is a physical motion calculation model based on the principle of uniform motion or the principle of uniform acceleration motion, or a motion trajectory prediction model pre-trained based on a machine learning algorithm.

4. The method for counting customer flow in a retail store according to claim 1, characterized in that: According to the future intra-frame motion trajectory of the binding target, combined with the new human head and shoulder detection result obtained based on the new video data, the binding target is tracked in real time, including: According to the future intra-frame motion trajectory of the bound target and the tracking frame of the bound target in the current most recent K frames, the tracking frame of the bound target in the current next frame is predicted by using the Kalman filter method, where K represents a positive integer; Obtaining at least one human head and shoulder marking frame in the current next frame according to the new human head and shoulder detection result obtained based on the new video data; The Hungarian algorithm and the DIoU index are used to optimally match the tracking frame of the binding target in the current next frame with the at least one human head and shoulder marker frame. If the tracking frame of the binding target in the current next frame matches a certain human head and shoulder marker frame in the at least one human head and shoulder marker frame, the certain human head and shoulder marker frame is bound to the binding target, and according to the tracking frame of the binding target in the current most recent K frames and the certain human head and shoulder marker frame, the Kalman filter method is used to update the tracking frame of the binding target in the current next frame, so as to continue tracking the binding target, otherwise the tracking of the binding target is terminated.

5. The method for counting customer flow in a retail store according to claim 1, characterized in that: Performing human head and shoulder feature extraction processing on the image in the human head and shoulder mark frame to obtain the human head and shoulder features of the binding target includes: The image in the human head and shoulder mark frame is imported into a human head and shoulder feature extraction model pre-trained based on a neural network and an unsupervised learning method, and the human head and shoulder features of the binding target are output.

6. The method for counting customer flow in a retail store according to claim 1, characterized in that: Judging whether the binding target is a known store clerk according to the head and shoulder features of the binding target includes: reading head and shoulder features of employees of the retail store; Calculate the first cosine distance between the head and shoulder features of the human body of the binding target and the head and shoulder features of the clerk; Determine whether the first cosine distance is less than or equal to a preset first distance threshold, if so, determine that the binding target is a known clerk, otherwise determine that the binding target is not a known clerk; And / or, judging whether the binding target is a repetitive behavior target according to the head and shoulder features of the binding target, including: Determine whether there is at least one historical tracking target that has ended tracking within the most recent preset time period. If so, read the head and shoulder features of the at least one historical tracking target. Otherwise, directly determine that the bound target is not a repeated behavior target, wherein the repeated behavior target refers to a tracking target that repeats the same store customer behavior at least twice within the preset time period. For each historical tracking target in the at least one historical tracking target, calculating a second cosine distance between a corresponding human head and shoulder feature and a human head and shoulder feature of the binding target; It is determined whether all the second cosine distances are greater than a preset second distance threshold; if so, it is determined that the binding target is not a repeated behavior target; otherwise, it is determined that the binding target is a repeated behavior target.

7. A retail store customer flow counting device, characterized in that: It is suitable for being arranged in an edge computing host located in a retail store, and includes a video data acquisition unit, a human head and shoulder detection unit, a motion trajectory prediction unit, a binding target tracking unit, a customer flow behavior determination unit, a head and shoulder feature extraction unit, and a store customer flow deduplication unit; The video data acquisition unit is used to acquire video data collected by a store camera in real time from the entrance and exit area of ​​the retail store, wherein the store camera is fixedly installed and the lens field of view covers the entrance and exit area; The human head and shoulder detection unit is communicatively connected to the video data acquisition unit and is used to perform human head and shoulder detection processing on the video frame images in the video data using a single-stage target detection algorithm to obtain a human head and shoulder detection result; The motion trajectory prediction unit is communicatively connected to the human head and shoulder detection unit, and is used to predict the future intra-frame motion trajectory of the corresponding binding target for each human head and shoulder mark frame in the human head and shoulder detection result according to the corresponding current intra-frame position information / and the historical intra-frame motion trajectory of the binding target, wherein the binding target refers to the tracking target bound to the human head and shoulder mark frame; The binding target tracking unit is communicatively connected to the human head and shoulder detection unit and the motion trajectory prediction unit, respectively, and is used to track the binding target in real time according to the future intra-frame motion trajectory of the binding target in combination with the new human head and shoulder detection result obtained based on the new video data; The customer behavior determination unit is communicatively connected to the bound target tracking unit, and is used to determine a single store customer behavior of the bound target according to a single tracking result of the bound target, wherein the single tracking result refers to a movement trajectory obtained by tracking the bound target from appearance to disappearance in the entrance and exit area, and the single store customer behavior refers to a single store entry behavior, a single store exit behavior, or a single store passing behavior; The head and shoulder feature extraction unit is communicatively connected to the human head and shoulder detection unit, and is used to intercept the image in the human head and shoulder mark frame of the binding target from the video data, and perform human head and shoulder feature extraction processing on the image in the human head and shoulder mark frame to obtain the human head and shoulder features of the binding target; The store customer flow deduplication unit is communicatively connected to the head and shoulder feature extraction unit and the customer flow behavior determination unit, respectively, and is used to determine whether the binding target is a known store clerk and / or a repeated behavior target based on the human head and shoulder features of the binding target. If not, after determining the single store customer flow behavior of the binding target, the store customer flow of the retail store is automatically increased by 1, wherein the repeated behavior target refers to a tracking target that repeats the same store customer flow behavior at least twice within a preset time period.

8. A computer device, characterized in that: It includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the retail store customer flow counting method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the computer, the retail store customer flow counting method as described in any one of claims 1 to 6 is executed.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the method for counting customer flow in a retail store as claimed in any one of claims 1 to 6 is implemented.

Images