Judgment method for grouping customers entering store
Through deep learning technology and REID model, combined with trajectory similarity algorithm, the deviation problem of traditional customer flow counting methods in customer group identification is solved, accurate grouping and accurate statistics of customers entering the store are achieved, and the accuracy of customer flow analysis is improved. .
Patent Information
- Application Number
- CN202411171600.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-13
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The traditional passenger flow counting method has deviations when processing customer groups, and cannot accurately identify and group, especially when the image size of the detection target changes, resulting in calculation errors and affecting the accuracy of passenger flow analysis.
Using deep learning technology, combined with portrait recognition, REID pedestrian re-identification and trajectory similarity algorithm, customers are grouped through portrait extraction, de-repetition numbering, trajectory positioning and trajectory comparison, and feature extraction and de-extraction using convolutional neural network and REID model. Heavy, dynamic time regularization algorithm calculates trajectory similarity, and self-learning adjusts the threshold to improve judgment accuracy.
Accurate grouping of customers entering the store, improve the accuracy of customer statistics and group judgment, and accurately divide customer groups through camera equipment without extracting biological characteristics. The model continues to be used as the model is used. Improve judgment accuracy.
Smart Images
Figure CN119130514A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of customer grouping methods, and is a method for dividing customers entering a store into customer groups in a commercial scenario, and performing accurate judgment and counting, and is specifically a judgment method for grouping customers entering a store. Background Art
[0002] In actual commercial or public scenarios, customers often travel as a family or with a consultant to carry out consumption activities, forming a purchasing power. However, traditional customer flow counting solutions only consider individual counting, but ignore the situation of customer group consumption. Using traditional counting methods, the purchasing power of a customer group is magnified to every customer entering the store, which distorts the purchasing power data.
[0003] At present, the technology for customer flow and passenger flow grouping mainly uses video surveillance to count people, and mainly uses deep learning neural network for feature recognition + line collision detection.
[0004] The main advantages of line collision detection are: by calculating whether the identification frame point touches the detection line and the number of times it crosses, the direction of the target crossing the detection line can be quickly determined. The calculation speed is fast and the application is wide. The detection is based on real-time video stream and can count people in real time. According to the synchronization of pedestrian entry, the passenger flow grouping is calculated.
[0005] The main drawbacks are: the calculation method of the customer group will produce a large deviation, which is of little practical significance. The size change of the detection target in the image will affect the size of the recognition frame. If the image size of the detection target changes drastically, it will cause the target to hit the line multiple times, resulting in calculation errors. Therefore, there are high requirements for the installation angle and point of the camera.
[0006] In general, video surveillance is a relatively mature method for realizing passenger flow analysis, but it has major defects in customer group analysis. Summary of the invention
[0007] The purpose of the present invention is to provide a method for determining groups of customers entering a store, so as to solve the problems raised in the above-mentioned background technology.
[0008] To achieve the above object, the present invention provides the following technical solution: a method for determining groups of customers entering a store, comprising the following steps:
[0009] S1: Portrait recognition and extraction: Based on deep learning technology, portraits in the original video are recognized and extracted;
[0010] S2: De-duplicate portrait ID: Use REID person re-identification technology to deduplicate the extracted portraits and assign each person a unique ID.
[0011] S3: scene position coordinate positioning, coordinate positioning of each position in the scene according to the initial image without pedestrians;
[0012] S4: Portrait ID positioning, extract the original video frame by frame, and locate the coordinates of the positioning point of each portrait ID in each frame;
[0013] S5: ID trajectory drawing, the positioning points of each portrait ID are gathered and drawn into lines to form a trajectory;
[0014] S6: trajectory comparison, comparing the trajectory of each portrait ID with the portrait IDs entering and exiting the scene in the same time period for trajectory similarity;
[0015] S7: Customer grouping: customers are grouped according to the similarity of the compared trajectories, and customers whose trajectory similarity reaches a set threshold are grouped together.
[0016] Preferably, in S1, a pedestrian target detection algorithm based on a convolutional neural network is used, yolov8Nano is used as the model basis for target detection, and training is performed in the COCO detection dataset and the actual application scenario dataset to obtain better usability. Using the trained model, the video is detected frame by frame, and the detected pedestrian frame point data and the picture in the detection frame are extracted and saved as a pedestrian detection dataset.
[0017] Preferably, in S1, the convolutional neural network includes a feature extraction layer and a feature mapping layer.
[0018] Preferably, in the feature extraction layer, the input of each neuron is connected to the local receptive field of the previous layer, and the local feature is extracted. Once the local feature is extracted, its positional relationship with other features is also determined.
[0019] Preferably, in the feature map layer, each computing layer of the network is composed of multiple feature maps, each feature map is a plane, and the weights of all neurons on the plane are equal. The feature map structure uses an activation function with a small influence function kernel, so that the feature map has displacement invariance.
[0020] Preferably, each convolutional layer in the convolutional neural network is followed by a computational layer for local averaging and secondary extraction.
[0021] Preferably, in S2, a convolutional neural network-based pedestrian re-identification algorithm is adopted. Without using facial information, the REID neural network model is used to extract feature values of all portrait features, extract features such as the color and style of the pedestrian's clothing, and perform a binary classification problem to determine whether two frames of pedestrian images belong to the same pedestrian. The cosine similarity between the portrait feature values is calculated, the detected portraits of the previous and next frames are deduplicated, and separate IDs are assigned to the same customer individuals.
[0022] Preferably, FastReID is used as the model basis for pedestrian re-identification, and is trained on the Market-1501 dataset and actual application scenario datasets to better match the actual application scenarios. The trained model is used to perform re-identification detection on the pedestrian dataset after target detection, and all pedestrians identified as the same are assigned independent IDs. The pedestrian's ID, detection box coordinates, continuous frames, and re-identification confidence are recorded and saved.
[0023] Preferably, in S6, a trajectory sequence similarity algorithm based on dynamic time warping is adopted, comprising the following steps:
[0024] Step 1: Based on the consistency of the starting point, start the similarity calculation model;
[0025] Step 2: Compare each ID in pairs and calculate the similarity and dissimilarity between segments, where the segment is the segment between two IDs; by defining segment time, similar segments, and similar routes, calculate the similarity and dissimilarity between trajectories, thereby obtaining trajectory similarity;
[0026] Step 3: Based on the judgment of endpoint consistency, supplement the samples that may be missed by the pairwise comparison model;
[0027] Step 4: Similarity consistency judgment, based on the empirical threshold, continuous iterative learning during the recognition process.
[0028] Preferably, in S6, a self-learning algorithm is used to annotate the judgment results by manual annotation, and the results are fed back to the learning algorithm. The algorithm corrects the judgment threshold according to the manual annotation results.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] The present invention can use cameras and other equipment to extract feature values of all portrait features using the REID neural network model without extracting the biological features of customers entering the store, compare the trajectory of each portrait ID with the portrait ID of people entering and leaving the scene in the same time period, and accurately divide the customers entering the store into groups. In addition, the model has a self-learning function, and the accuracy of customer group judgment will continue to improve with the continuous use of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 The present invention is a flow chart of a method for determining grouping customers entering a store. DETAILED DESCRIPTION
[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0033] See also Figure 1 The present invention provides a technical solution: a method for determining the groups of customers entering a store, comprising the following steps:
[0034] S1: Portrait recognition and extraction, based on deep learning technology, recognize and extract portraits in the original video; this step uses deep learning technology, especially models such as convolutional neural networks (CNN), to automatically recognize and extract portraits from the original video stream; this is the basis of the entire process and provides necessary input data for subsequent steps;
[0035] S2: De-duplicate portrait ID: Use REID (Re-Identification) technology to deduplicate extracted portraits and assign a unique ID to each person. Use REID (Re-Identification) technology to deduplicate extracted portraits and assign a unique ID to each person. REID technology can identify the same person across cameras and time. It effectively solves the problem of repeated counting of the same person in the video at different times and locations, ensuring that each customer is counted only once, improving the accuracy of customer statistics.
[0036] S3: Scene position coordinate positioning: coordinate positioning of each position in the scene based on the initial image without pedestrians; construction of scene map to provide a reference for subsequent portrait position positioning; enabling the system to accurately understand the customer's movement path in the store;
[0037] S4: Portrait ID positioning: extract the original video frame by frame, and locate the coordinates of each portrait ID in each frame. Through the position information in consecutive frames, the movement trajectory of each customer in the scene can be constructed, providing key data for subsequent customer behavior analysis.
[0038] S5: ID trajectory drawing, which gathers the positioning points of each portrait ID and draws a line to form a trajectory; it intuitively displays the customer's movement path in the store and provides a visualization tool for understanding customer behavior patterns;
[0039] S6: Trajectory comparison: compare the trajectory of each portrait ID with the portrait IDs that enter and exit the scene in the same time period for trajectory similarity. By comparing trajectories, customer groups with similar behavior patterns can be identified, providing a basis for subsequent customer grouping.
[0040] S7: Customer grouping: customers are grouped according to the similarity of their trajectories after comparison. Customers whose trajectories have reached a set threshold are grouped together. Customer grouping helps merchants better understand the behavioral characteristics of different customer groups, thereby formulating more targeted marketing strategies and service plans. At the same time, it also provides data support for further analyzing customer behavior and optimizing store layout.
[0041] The threshold setting of customer trajectory similarity depends on multiple factors, including application scenarios, data characteristics, analysis purposes, and the balance between false positive rate and false negative rate. This example provides a reference:
[0042] Low similarity threshold (20%-40%): When the similarity between two trajectories is lower than this threshold range, it can be considered that the difference between them is large and has no significant similarity.
[0043] Medium similarity threshold (40%-70%): Within this range, two trajectories may have certain similarities, but may need to be further judged in combination with other factors.
[0044] High similarity threshold (70%-100%): When the similarity between two trajectories is within this threshold range, it can be considered that the similarity between them is high and they may belong to the same customer group or have similar behavior patterns.
[0045] In S1, a pedestrian target detection algorithm based on a convolutional neural network is used, and yolov8Nano is used as the model basis for target detection. It is trained in the COCO detection dataset and the actual application scenario dataset to obtain better usability. Using the trained model, the video is detected frame by frame, and the detected pedestrian frame point data and the pictures in the detection frame are extracted and saved as a pedestrian detection dataset.
[0046] The COCO detection dataset is a large, general object detection dataset that contains multiple object categories, including pedestrians. You can download labeled data suitable for pedestrian detection from the COCO official website. Collect video or image data for specific application scenarios and manually annotate the pedestrians in them. This data should cover as many situations as possible in actual applications to improve the generalization ability of the model;
[0047] Download the pre-trained weights or source code of YOLOv8Nano and configure it appropriately, setting the training parameters such as learning rate, batch size, training rounds, etc. Use the COCO dataset and the actual application scenario dataset for training. You can use the transfer learning method to first use the COCO dataset for pre-training, and then use the actual application scenario dataset for fine-tuning.
[0048] In S1, the convolutional neural network includes a feature extraction layer and a feature mapping layer.
[0049] In the feature extraction layer, the input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local feature is extracted, its positional relationship with other features is also determined.
[0050] In the feature map layer, each computational layer of the network consists of multiple feature maps, each feature map is a plane, and the weights of all neurons on the plane are equal. The feature map structure uses an activation function with a small influence function kernel, so that the feature map has displacement invariance.
[0051] Each convolutional layer in the convolutional neural network is followed by a computational layer for local averaging and secondary extraction. This unique two-time feature extraction structure reduces the feature resolution.
[0052] In a convolutional neural network (CNN), feature extraction layers and feature mapping layers (sometimes also called convolution layers and pooling layers) are the core parts of building a network. They work together to extract useful features from the input data. In the feature extraction layer (convolution layer), the input of each neuron (or convolution kernel, filter) is only connected to the local area of the previous layer (called the local receptive field or receptive field). This local connection allows the network to focus on the local features of the input data, such as edges, corners, etc. When the convolution kernel slides on the input data, it extracts the same features at different locations. The positional relationship of these features is determined by the sliding step size of the convolution kernel and the size of the input data.
[0053] Although each neuron in the convolutional layer is connected to the local area of the previous layer, all neurons in the same feature map share the same weights and biases. This weight sharing mechanism greatly reduces the number of network parameters, reduces the risk of overfitting, and enables the network to learn the spatial hierarchical structure of the input data;
[0054] The feature mapping layer (usually called the pooling layer) follows the convolutional layer and is used to downsample the feature map output by the convolutional layer. Pooling operations usually include maximum pooling (taking the maximum value in a local area) and average pooling (taking the average value in a local area). These operations not only reduce the size of the feature map and reduce the computational complexity, but also enhance the translation invariance and robustness of the network. Through the pooling layer, the network can further extract more abstract and robust features.
[0055] The feature map structure in the pooling layer achieves shift invariance by using a small activation function kernel (i.e., pooling window). This means that even if the features in the input data are offset in position, the pooling operation can still capture these features because the pooling operation is performed in a local area and is not sensitive to the specific location of the feature.
[0056] In convolutional neural networks, convolutional layers and pooling layers usually appear alternately, forming a unique two-time feature extraction structure. This structure first extracts local features of the input data through the convolutional layer, and then downsamples and re-extracts the features through the pooling layer, thereby reducing the feature resolution and enhancing the robustness of the network. This structure can extract feature representations of different levels of input data layer by layer, providing strong support for subsequent classification or regression tasks.
[0057] In S2, a convolutional neural network-based pedestrian re-identification algorithm is adopted. Without using facial information, the REID neural network model is used to extract feature values of all portrait features, extract features such as the color and style of pedestrians' clothing, and perform binary classification to determine whether two frames of pedestrian images belong to the same pedestrian. The cosine similarity between the portrait feature values is calculated, the detected portraits of the previous and next frames are deduplicated, and separate IDs are assigned to the same customer individuals.
[0058] FastReID is used as the model basis for pedestrian re-identification. It is trained on the Market-1501 dataset and actual application scenario datasets to better match the actual application scenarios. The trained model is used to perform re-identification detection on the pedestrian dataset after target detection. All pedestrians identified as the same are given independent IDs, and the pedestrian's ID, detection box coordinates, continuous frames, and re-identification confidence are recorded and saved.
[0059] FastReID is a library designed for person re-identification. It integrates many advanced algorithms and technologies to efficiently extract and match pedestrian features. Download the Market-1501 dataset from the official website or public resources and organize it into the format required by FastReID. Verify the annotation quality of the dataset to ensure that each pedestrian has the correct ID and corresponding image.
[0060] In S6, a trajectory sequence similarity algorithm based on dynamic time warping is adopted, which includes the following steps:
[0061] Step 1: Based on the consistency of the starting point, start the similarity calculation model; before comparing two trajectories, first confirm whether their starting points are consistent or close enough. This is because although the DTW algorithm can handle time expansion, the consistency of the starting point can be used as an effective filtering condition to reduce unnecessary calculations;
[0062] Step 2: Compare each ID in pairs and calculate the similarity and dissimilarity between segments, where the segment is the segment between two IDs; by defining segment time, similar segments, and similar routes, calculate the similarity and dissimilarity between trajectories, thereby obtaining trajectory similarity;
[0063] Step 3: Based on the judgment of endpoint consistency, supplement the samples that may be missed by the pairwise comparison model; although starting point consistency is an effective filtering condition, endpoint consistency can also be used as a supplementary condition to ensure that trajectories with different starting points but the same or close endpoints are not missed; for those trajectory pairs with inconsistent starting points but consistent endpoints, re-evaluate their similarity and possibly add them to the set of similar trajectories;
[0064] Step 4: Similarity consistency judgment: Based on the empirical threshold, continuous iterative learning is performed during the recognition process; an initial similarity threshold is set, and the effect of the current threshold is evaluated using a manually annotated data set. If the misjudgment rate is high, the threshold is adjusted based on the results of manual annotation, and the process is iterated until a satisfactory accuracy rate is achieved or the predetermined number of iterations is reached;
[0065] Using a self-learning algorithm, we use manual labeling to label the judgment results and feed them back to the learning algorithm. The algorithm corrects the judgment threshold based on the manual labeling results. In each iteration, we collect misjudged samples and analyze why they are misjudged. We adjust the parameters or thresholds of similarity calculation based on the analysis results. Repeat the above process until the model performance is stable or other stop conditions are met.
[0066] The present invention can use cameras and other equipment to extract feature values of all portrait features using the REID neural network model without extracting the biological features of customers entering the store, compare the trajectory of each portrait ID with the portrait ID of people entering and leaving the scene in the same time period, and accurately divide the customers entering the store into groups. In addition, the model has a self-learning function, and the accuracy of customer group judgment will continue to improve with the continuous use of the model.
[0067] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.
Claims
1. A method for grouping customers entering a store, characterized in that: The following steps are involved: S1: Portrait recognition and extraction: Based on deep learning technology, portraits in the original video are recognized and extracted; S2: De-duplicate portrait ID: Use REID person re-identification technology to deduplicate the extracted portraits and assign each person a unique ID. S3: scene position coordinate positioning, coordinate positioning of each position in the scene according to the initial image without pedestrians; S4: Portrait ID positioning, extract the original video frame by frame, and locate the coordinates of the positioning point of each portrait ID in each frame; S5: ID trajectory drawing, the positioning points of each portrait ID are gathered and drawn into lines to form a trajectory; S6: trajectory comparison, comparing the trajectory of each portrait ID with the portrait IDs entering and exiting the scene in the same time period for trajectory similarity; S7: Customer grouping: customers are grouped according to the similarity of the compared trajectories, and customers whose trajectory similarity reaches a set threshold are grouped together.
2. A method for grouping customers entering a store according to claim 1, characterized in that: In S1, a pedestrian target detection algorithm based on a convolutional neural network is used, and yolov8Nano is used as the model basis for target detection. It is trained in the COCO detection dataset and the actual application scenario dataset to obtain better usability. Using the trained model, the video is detected frame by frame, and the detected pedestrian frame point data and the pictures in the detection frame are extracted and saved as a pedestrian detection dataset.
3. A method for determining the grouping of customers entering a store according to claim 2, characterized in that: In S1, the convolutional neural network includes a feature extraction layer and a feature mapping layer.
4. A method for grouping customers entering a store according to claim 3, characterized in that: In the feature extraction layer, the input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local feature is extracted, its positional relationship with other features is also determined.
5. A method for determining the grouping of customers entering a store according to claim 3, characterized in that: In the feature map layer, each computational layer of the network consists of multiple feature maps, each feature map is a plane, and the weights of all neurons on the plane are equal. The feature map structure uses an activation function with a small influence function kernel, so that the feature map has displacement invariance.
6. A method for determining the groups of customers entering a store according to claim 3, characterized in that: Each convolutional layer in a convolutional neural network is followed by a computational layer for local averaging and secondary extraction.
7. A method for grouping customers entering a store according to claim 1, characterized in that: In S2, a convolutional neural network-based pedestrian re-identification algorithm is adopted. Without using facial information, the REID neural network model is used to extract feature values of all portrait features, extract features such as the color and style of pedestrians' clothing, and perform binary classification to determine whether two frames of pedestrian images belong to the same pedestrian. The cosine similarity between the portrait feature values is calculated, the detected portraits of the previous and next frames are deduplicated, and separate IDs are assigned to the same customer individuals.
8. A method for determining the groups of customers entering a store according to claim 7, characterized in that: FastReID is used as the model basis for pedestrian re-identification. It is trained on the Market-1501 dataset and actual application scenario datasets to better match the actual application scenarios. The trained model is used to perform re-identification detection on the pedestrian dataset after target detection. All pedestrians identified as the same are given independent IDs, and the pedestrian's ID, detection box coordinates, continuous frames, and re-identification confidence are recorded and saved.
9. A method for grouping customers entering a store according to claim 1, characterized in that: In S6, a trajectory sequence similarity algorithm based on dynamic time warping is adopted, which includes the following steps: Step 1: Based on the consistency of the starting point, start the similarity calculation model; Step 2: Compare each ID in pairs and calculate the similarity and dissimilarity between segments, where the segment is the segment between two IDs; by defining segment time, similar segments, and similar routes, calculate the similarity and dissimilarity between trajectories, thereby obtaining trajectory similarity; Step 3: Based on the judgment of endpoint consistency, supplement the samples that may be missed by the pairwise comparison model; Step 4: Similarity consistency judgment, based on the empirical threshold, continuous iterative learning during the recognition process.
10. A method for grouping customers entering a store according to claim 9, characterized in that: In S6, a self-learning algorithm is used to annotate the judgment results by manual labeling, and the results are fed back to the learning algorithm. The algorithm modifies the judgment threshold according to the manual labeling results.