Association method and association system for facilitating customer movement line and shelf display of store

Abstract

The embodiment of the invention provides an association method and an association system for facilitating customer movement lines and shelf display of stores, and belongs to the field of intelligent data processing. Comprising the following steps: performing distortion correction and scale transformation on each frame of monitoring image in a monitoring video according to pre-stored camera parameters and distortion parameters to obtain a target image corresponding to each frame of monitoring image; identifying pedestrian information of any pedestrian in the target image based on a multi-target detection model; determining moving line information of the pedestrian in the monitoring video according to the pedestrian information of the pedestrian in each frame of monitoring image of the monitoring video; and associating the moving line information of each pedestrian in the monitoring video with the shelf display information of the convenience store. Compared with an existing technical method, the method has the advantages that the customer moving line in the monitoring video can be accurately analyzed in an automatic mode and is associated with shelf display, manual observation and manual judgment are not needed, and labor cost and subjective deviation in the decision making process are avoided.

Description

Technical Field

[0001] The present invention relates to the field of intelligent data processing technology, and in particular to a method and a system for associating customer traffic lines and shelf displays in a convenience store. Background Art

[0002] In the retail industry, customer movement information is of great significance for optimizing shelf placement and product display, which directly affects sales, customer satisfaction and the overall operating efficiency of the store. Based on customer movement information, merchants can formulate more targeted marketing strategies. For example, for "hot spots" that are frequently visited, merchants can increase advertising and promotional activities to obtain higher sales and profits. By continuously collecting and analyzing customer movement data, merchants can regularly monitor the effects of shelf placement and product display, and make timely adjustments based on marketing data feedback.

[0003] With the development of artificial intelligence, especially deep learning technology, existing technologies can use cameras to capture video data in stores and use intelligent video analysis algorithms to obtain some information about customers entering the store. Although existing technologies can obtain the location and trajectory of pedestrians based on convenience store surveillance videos or images, the camera will project the three-dimensional world onto a two-dimensional image during the shooting process. The error caused by this projection distortion will have a certain impact on the accuracy of customer movement line analysis. Moreover, the analysis results of existing technologies are usually expressed in the form of pixel coordinates in the image, which cannot be associated with the actual location in the store and the specific shelf.

[0004] Therefore, the existing technology still requires merchants to manually observe heat maps and other methods to manually judge the correlation between customer traffic lines in the image and the actual shelf displays in the store. It cannot directly provide merchants with decision-making suggestions on shelf placement and product display, which affects the analysis efficiency. At the same time, the data source of its analysis data is often distorted images, which will further cause analysis errors. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method and system for associating customer movement lines and shelf displays in a convenience store, thereby aiming to solve the problems of low accuracy in existing customer movement line analysis technologies and the need for manual observation and judgment of analysis results.

[0006] In order to achieve the above-mentioned purpose, an embodiment of the present invention provides a method for associating customer movement lines and shelf displays in a convenience store, the association method comprising: performing distortion correction and scale transformation on each frame of surveillance image in a surveillance video according to pre-stored camera parameters and distortion parameters to obtain a target image corresponding to each frame of surveillance image; identifying pedestrian information of any pedestrian in the target image based on a multi-target detection model; determining the movement line information of the pedestrian in the surveillance video according to the pedestrian information of the pedestrian in each frame of surveillance image of the surveillance video; and associating the movement line information of each pedestrian in the surveillance video with the shelf display information of the convenience store.

[0007] Optionally, the camera parameters include a principal point and a focal length, and the scaling of each surveillance image frame in the surveillance video includes: according to the principal point [px, py], the focal length [fx, fy] and any pixel point p (x, y) of each surveillance image frame in the surveillance video, determining a first pixel point p' (x', y') corresponding to the pixel point by the following formula:

[0008]

[0009] Wherein, [cx, cy] is the coordinate of the center point of the monitoring image after transformation.

[0010] Optionally, the distortion parameters include radial distortion parameters and tangential distortion parameters, and the distortion correction of each surveillance image frame in the surveillance video includes: according to the radial distortion parameters [k2, k4], the tangential distortion parameters [tx, ty] and the first pixel point p'(x', y'), the following formula is used to determine the second pixel point p" (x", y") after radial distortion and the third pixel point p"' (x"', y"') after tangential distortion:

[0011] x″=x′(1+k2×r 2 +k4×r 4 ),

[0012] y″=y′(1+k2×r 2 +k4×r 4 ),

[0013] x″′=x″+tx×y″,

[0014] y″′=y″+ty×x″,

[0015] Where r is the distance from point p(x,y) to the center point of the monitoring image, which is

[0016] Optionally, the identifying the pedestrian information of any pedestrian in the target image based on the multi-target detection model includes: based on the multi-target detection model, identifying the first pedestrian information of the pedestrian in the target image and identifying the second pedestrian information of all pedestrians in the previous frame image of the target image; determining whether the pedestrian is an existing pedestrian or a newly appeared pedestrian based on the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information; in the case that the pedestrian is the existing pedestrian, associating the pedestrian position information of the pedestrian to the end of the pedestrian's historical trajectory; and in the case that the pedestrian is the newly appeared pedestrian, creating a new trajectory for the pedestrian and using the pedestrian position information of the pedestrian as the starting point of the new trajectory.

[0017] Optionally, the pedestrian information includes pedestrian appearance information, pedestrian position information and trajectory prediction information, and the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information includes: a first similarity in pedestrian appearance information, a second similarity in pedestrian position information and a third similarity in trajectory prediction information, wherein the first similarity is determined by the Euclidean distance measurement between vectors, and the second similarity and the third similarity are determined by the Euclidean distance measurement between coordinates.

[0018] Optionally, the determining of the pedestrian as an existing pedestrian or a newly appeared pedestrian based on the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information includes: when the first similarity, the second similarity and the third similarity between a pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian all meet the set conditions, the pedestrian is determined as the existing pedestrian; and when the first similarity, the second similarity and the third similarity between any pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian do not simultaneously meet the set conditions, the pedestrian is determined as the newly appeared pedestrian.

[0019] Optionally, after identifying the pedestrian information of any pedestrian in the target image based on the multi-target detection model, the association method further includes: identifying the employee of the convenience store from the target image based on the employee classification model and the pedestrian appearance information in the pedestrian information; after determining the movement information of the pedestrian in the surveillance video based on the pedestrian information in each frame of the surveillance video, the association method further includes: deleting the movement information of the employee in the surveillance video.

[0020] Optionally, the identifying the employees of the convenience store from the target image based on the employee classification model and the pedestrian appearance information in the pedestrian information includes: obtaining a training set, wherein the training set is a pedestrian classification feature data set based on historical surveillance videos; using the training set to train the employee classification model so that the accuracy of the output result of the employee classification model and the actual result is greater than a set ratio; and inputting the image block where each pedestrian target in the target image is located into the employee classification model to identify the pedestrian target as an employee or a customer.

[0021] Optionally, associating the movement information of each pedestrian in the surveillance video with the shelf display information of the convenience store includes: mapping the pedestrian position information of any pedestrian in the target image to the ground coordinates of the convenience store based on the homography matrix between the ground in the surveillance video and the ground of the convenience store, so as to obtain the mapping coordinates of the pedestrian on the ground of the convenience store; according to preset shelf display information, determining the shelf that is closest to the mapping coordinates of the pedestrian and the distance between the two is less than a set distance as the target shelf; and associating the pedestrian's number, mapping coordinates, and time information of the target image with the target shelf.

[0022] On the other hand, the present invention provides a system for associating customer traffic flow and shelf displays in a convenience store, the association system comprising: an image processing device for performing distortion correction and scale transformation on each frame of surveillance image in a surveillance video according to pre-stored camera parameters and distortion parameter information, so as to obtain a target image corresponding to each frame of surveillance image; a pedestrian recognition device for identifying pedestrian information of any pedestrian in the target image based on a multi-target detection model; a traffic flow determination device for determining the traffic flow information of the pedestrian in the surveillance video according to the pedestrian information of the pedestrian in each frame of surveillance image of the surveillance video; and an association device for associating the traffic flow information of each pedestrian in the surveillance video with the shelf display information of the convenience store.

[0023] Through the above technical solution, the improvements and beneficial effects of the present invention are as follows:

[0024] Compared with existing technical methods, the present invention can accurately analyze customer movement lines in surveillance videos and associate them with shelf displays in an automated manner, without the need for manual observation and judgment, thus avoiding labor costs and subjective biases in the decision-making process. In terms of accuracy improvement, the present invention achieves coordinate mapping between video images and real stores through distortion correction and position mapping, avoiding errors caused by image distortion and camera projection, thereby solving the problem of low accuracy in existing customer movement line analysis technologies. In terms of automated analysis, the present invention utilizes a multi-target detection model that can process and analyze surveillance video data in real time, achieve accurate identification of customer movement lines within stores, and automatically associate the identified customer movement line trajectory information with shelf display information at specific locations.

[0025] Through the information on the association between customer traffic and shelf display provided by the present invention, the popularity or attention of the shelf can be automatically analyzed and obtained later, thereby providing a decision-making reference for the shelf display of the merchant. Therefore, the present invention can effectively improve the operating efficiency and intelligence level of the store, thereby improving the customer satisfaction and sales revenue of the store.

[0026] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0028] Figure 1 It is a flowchart of a method for associating customer traffic flow and shelf display in a convenience store provided according to an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of location mapping from surveillance video to store map provided according to an embodiment of the present invention;

[0030] Figure 3 is a logical schematic diagram of a video acquisition and analysis process provided according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a system for associating customer traffic flow and shelf display in a convenience store provided according to an embodiment of the present invention;

[0032] Figure 5 It is a schematic diagram of the execution logic of each functional module provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0034] The present invention first provides a method 100 for associating customer movement lines and shelf displays in a convenience store. The shelf display information may include the shelf number, the actual location of the shelf on the store digital map, the commodity information displayed on the shelf, etc. The customer movement line information may include the pixel coordinates of the customer's trajectory, the corresponding shelf number, the passing time, the customer ID, etc.

[0035] Specific as Figure 1 As shown, the association method 100 of the present invention may include steps S110 - S140 .

[0036] Step S110 , performing distortion correction and scale transformation on each surveillance image frame in the surveillance video according to pre-stored camera parameters and distortion parameters, so as to obtain a target image corresponding to each surveillance image frame.

[0037] In one embodiment, the surveillance video data can be collected by a camera, and then the video stream data can be obtained in real time through the RTSP protocol and / or the FFMpeg library to obtain each frame of the surveillance image. Then, the video stream data can be stored, for example, in a data storage module (such as a MySQL database). In addition, the shelf display information and the detected customer information (customer information) can also be stored in the data storage module.

[0038] In order to improve the accuracy of subsequent pedestrian detection and tracking, it is necessary to use the pre-stored camera parameters and distortion parameter information to perform distortion correction and scale transformation on the collected video images, such as Figure 2 As shown, it meets the input requirements of subsequent modules. Among them, the camera parameters and distortion parameter information can be obtained through the camera's internal parameter calibration link. For example, a camera can be used to shoot 10 black and white chessboard images at different angles, and then the camera calibration toolbox of Matlab software is used to complete the camera calibration. Among them, the camera parameters can include the principal point and focal length, and the distortion parameters can include radial distortion and tangential distortion parameters.

[0039] In one embodiment, the required camera parameters may include the principal point [px, py] and the focal length [fx, fy]. Then, the scaling of each surveillance image frame in the surveillance video in step S110 may be specifically performed as follows: a point p(x, y) in the image is transformed by the camera parameters to form a new point p'(x', y'). Specifically, according to the principal point [px, py], the focal length [fx, fy] and any pixel point p(x, y) of each surveillance image frame in the surveillance video, the first pixel point p'(x', y') corresponding to the pixel point may be determined by the following formula:

[0040]

[0041] Among them, [cx, cy] is the coordinate of the center point of the monitoring image after transformation.

[0042] In addition, the distortion parameters may include radial distortion [k2, k4] and tangential distortion parameters [tx, ty]. Assuming that the second pixel point in the radially distorted image is p" (x", y), the distortion correction of each frame of the surveillance image in the surveillance video in step S110 may be specifically as follows:

[0043] According to the radial distortion parameters [k2, k4], the tangential distortion parameters [tx, ty] and the first pixel point p'(x', y'), the second pixel point p'(x", y") after radial distortion is determined by the following formula:

[0044] x″=x′(1+k2×r 2 +k4×r 4 ),

[0045] y″=y′(1+k2×r 2 +k4×r 4 ),

[0046] Among them, r is the distance from point p(x,y) to the center point of the monitoring image, which is Assuming that the third pixel in the tangentially distorted image is p″′(x″′,y″′), then:

[0047] x″′=x″+tx×y″,

[0048] y″′=y″+ty×x″.

[0049] According to the above formula, the target image after distortion correction and scale transformation is obtained. That is, after the required parameters are obtained through intrinsic parameter calibration, a frame of video image can be transformed into a normal image without distortion by using the inverse transformation of the above formula, that is, the straight lines and planes in the real world are also presented as straight lines and planes in the video image. Finally, according to the input requirements of the subsequent pedestrian detection model, the image is scaled to the specified size.

[0050] In summary, in order to solve the problem of low accuracy in existing customer movement line analysis technology, the present invention proposes a fully automatic high-precision customer movement line analysis method. Compared with directly calculating the relationship between customers and shelves in the surveillance video, this method uses a mapping matrix method to avoid the accuracy error caused by camera projection and bring better association and analysis accuracy.

[0051] Step S120, based on the multi-target detection model, identify the pedestrian information of any pedestrian in the target image. For example, pedestrian detection can be achieved through a deep neural network. Among them, the pedestrian detection model (multi-target detection model) refers to a deep neural network model trained using a special training set. The built-in multi-target tracking algorithm in the model can comprehensively consider the appearance similarity, position similarity and trajectory prediction information to achieve the association between pedestrian trajectories and detection targets.

[0052] In one embodiment, step S120 may include steps S121 - S123 .

[0053] Step S121, based on the multi-target detection model, first pedestrian information of the pedestrian in the target image is identified and second pedestrian information of all pedestrians in the previous frame image of the target image is identified. The pedestrian information may include pedestrian appearance information, pedestrian location information and trajectory prediction information.

[0054] Specifically, the multi-target detection model can be a pre-trained yolov8 target detection model. For a frame of image in the video, the trained yolov8 pedestrian detection model is used to obtain information about all pedestrians in the image, which may include ID, position rectangle, confidence, etc. For example, (pedestrian 1, [200, 200, 250, 300], 0.95) means that a pedestrian is detected at the position of the rectangle [200, 200, 250, 300] on the image, and its ID number is pedestrian 1, and the confidence is 0.95. If the detection confidence is less than 0.4, the result will be filtered out to avoid interference caused by false detection.

[0055] The pedestrian detection model can be trained by constructing a special training set. The training set includes a public pedestrian detection data set and a proprietary data set made from store monitoring data. The multi-target tracking algorithm in the multi-target detection model adopted by the present invention can also detect the position of pedestrians in the image, and track each pedestrian using the multi-target tracking algorithm to obtain a complete pedestrian trajectory represented by a set of pixel point coordinates. For each detected pedestrian target, the image block within the rectangular frame can be cropped and a 512-dimensional feature vector can be extracted using a ResNet network as the appearance feature of the pedestrian.

[0056] Step S122: Determine whether the pedestrian is an existing pedestrian or a newly appeared pedestrian based on the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information.

[0057] Among them, the similarity between each pedestrian information in the first pedestrian information and the second pedestrian information may include: a first similarity in pedestrian appearance information, a second similarity in pedestrian position information, and a third similarity in trajectory prediction information. Among them, the pedestrian position information can be expressed as the coordinates of the bottom midpoint of the pedestrian rectangular box, and the trajectory prediction information refers to the coordinate position of the pedestrian that may appear in the current frame predicted by the pedestrian's historical trajectory. In one embodiment, the first similarity can be determined by the Euclidean distance measurement between vectors, that is, the similarity of the appearance features of two pedestrians can be measured by the Euclidean distance between vectors. The second similarity and the third similarity are determined by the Euclidean distance measurement between coordinates, that is, the similarity of the position information can be measured by the Euclidean distance between coordinates, and the similarity of the prediction information can also be measured by the Euclidean distance between coordinates.

[0058] Among them, the similarity of appearance features, the similarity of location information and the similarity of prediction information may be comprehensively considered to determine whether the pedestrian is an existing pedestrian or a pedestrian newly appearing in the current frame. That is, step S122 may include:

[0059] 1) if the first similarity, the second similarity and the third similarity between the pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian all meet the set conditions, determine the pedestrian as an existing pedestrian; and

[0060] 2) When the first similarity, the second similarity and the third similarity between any pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian cannot simultaneously satisfy the set conditions, the pedestrian is determined as a newly appeared pedestrian.

[0061] Through the above steps, if all three similarities meet the threshold conditions, the current target is considered to be an existing pedestrian. If any similarity does not meet the threshold conditions, the current target is considered to be a newly appeared pedestrian.

[0062] Step S123, when the pedestrian is an existing pedestrian, the pedestrian position information of the pedestrian is associated with the end of the historical track of the pedestrian.

[0063] Step S124: when the pedestrian is a new pedestrian, a new track is created for the pedestrian and the pedestrian position information of the pedestrian is used as the starting point of the new track.

[0064] In summary, the multi-target tracking algorithm can be used to identify the appearance information and position information of any pedestrian in the target image, and determine whether the pedestrian is an existing pedestrian or a new pedestrian in the current frame based on the appearance similarity, position similarity and trajectory prediction information of a pedestrian in adjacent frames. If it is an existing pedestrian, the pedestrian's pedestrian position information is associated with the end of the pedestrian's historical trajectory, that is, the pedestrian information of the current frame is added to the end of the original pedestrian trajectory; if it is a new pedestrian, a new id and a new trajectory are created, a new trajectory is created for the pedestrian, and the pedestrian position information of the pedestrian is used as the starting point of the new trajectory, and the appearance information and position information of the pedestrian are saved at the same time.

[0065] Step S130, determining the movement line information of the pedestrian in the surveillance video according to the pedestrian information in each surveillance image frame of the surveillance video.

[0066] In this step, the multi-target tracking algorithm in the multi-target detection model can be used to obtain the pedestrian trajectory, thereby obtaining the pedestrian's movement line information in the surveillance video. That is, by identifying the pedestrian information in each frame, confirming the position of the same pedestrian between different frames, and then tracking each pedestrian to obtain a complete pedestrian trajectory represented by a set of pixel point coordinates.

[0067] Step S140, associating the movement information of each pedestrian in the surveillance video with the shelf display information of the convenience store.

[0068] Among them, the customer position detected in the image can be mapped to the store floor in the real world through a pre-calculated homography matrix, and then the coordinates are compared with the shelf floor coordinates recorded in the data storage module. If the distance threshold condition is met, the customer information is recorded in the data storage module.

[0069] In one embodiment, step S140 may include steps S141 - S142 .

[0070] Step S141, based on the homography matrix between the ground in the surveillance video and the ground of the convenience store, mapping the pedestrian position information of any pedestrian in the target image to the ground coordinates of the convenience store to obtain the mapping coordinates of the pedestrian on the ground of the convenience store;

[0071] Among them, the homography matrix describes the correspondence between two planes and can be obtained through the external parameter calibration of the camera. Specifically, the customer position detected in the image can be mapped to the store floor in the real world by using the position mapping module through the pre-calculated homography matrix between the ground in the surveillance video and the ground of the convenience store. For example, it can be assumed that the ground in the store is a plane, and the homography matrix is ​​calculated by establishing the correspondence between the ground in the video image and the ground in the store map. Obviously, the point x1 on the ground in the image corresponds one-to-one to the point x2 on the actual ground, that is:

[0072]

[0073] Where H represents the homography matrix. It can be seen that the matrix contains 9 unknowns in total. If the scale factor is not considered, it contains 8 unknowns. Since each set of corresponding points contains 2 constraints, solving H requires at least 4 sets of corresponding points.

[0074] In this regard, the homography matrix H can be solved through the external parameter calibration of the camera. First, a digital map is created for the store, and the location of the shelves on the map is marked. Then, no less than 4 groups of corresponding points are selected on the corrected video image and the digital map, and the image pixel coordinate points and the world coordinate points are recorded. Finally, based on the coordinates of the corresponding points, the singular value decomposition (SVD) method is used to calculate the homography matrix H.

[0075] Specifically, the image coordinates of the customer's location can be transformed into the ground coordinates of the store based on the homography matrix H. Let z1 = 1, then:

[0076]

[0077] Thus, the mapping coordinates (x2, y2) of the pedestrian on the ground of the convenience store are obtained.

[0078] Therefore, in this embodiment, ground coordinates are used when calculating the distance between the customer and the shelf, thereby avoiding the error caused by camera projection when using image coordinates, so the result of the association analysis is more accurate.

[0079] Step S142: According to the preset shelf display information, the shelf that is closest to the mapped coordinates of the pedestrian and whose distance is less than a set distance is determined as the target shelf.

[0080] Among them, the shelf display information may include the shelf number, the actual location of the shelf, the information of the goods displayed on the shelf, etc. The actual location of the shelf refers to the location of the shelf in the store map, which can be actually measured using a measuring tool such as a ruler. For example, (Shelf 1, [100, 100], [Goods 1, Goods 2]) means that the shelf numbered 1 is located at the coordinates [100, 100] in the store map, and the goods displayed are Goods 1 and Goods 2. A table can be further established for the information of the goods displayed on the shelf to record information such as the shelf to which the goods belong, the location of the goods display, and store it in the data storage module. The customer information (customer information) may include the customer ID, the pixel coordinates of the customer's trajectory, the corresponding shelf number, the elapsed time, etc., which can be obtained through the above step S120 and automatically written after the corresponding position mapping module completes the calculation.

[0081] Since the display information of each shelf is known, the mapping coordinates (x2, y2) of the pedestrian on the ground of the convenience store obtained in step S141 are compared with the ground coordinates of all shelves. If the nearest shelf meets the distance threshold condition, the shelf is considered to be the shelf where the customer is currently located, and the information of the customer passing the shelf is recorded in the data storage module.

[0082] Step S143, the pedestrian number, mapping coordinates, time information of the target image are associated with the target shelf. The number of customers near a shelf within a period of time can be calculated by querying the customer information in the data storage module, as the popularity or attention of the shelf, and provided to the merchant as a decision reference for shelf product display.

[0083] For example, customer 1 appears at position [301,236] in the video image at 10:30:55 on January 5, 2024. After position mapping, it is found that the customer is near shelf 1 of the store. Then a record (customer 1, [301,236], shelf 1, 2024-1-5-10-30-55) is inserted into the customer information table of the data storage module. If the user records the product information displayed on each shelf in the data storage module, the system can calculate and store the correspondence between customer locations and product displays in the same way.

[0084] In addition, the number of customers near different shelves represents the popularity or attention of the shelf. Therefore, the customer information corresponding to the shelf area within a period of time can be counted. For example, all shelves can be sorted according to the number of customers who stayed during the period of time to obtain the popularity or attention of each shelf.

[0085] Specifically, the above steps can be executed by a shelf statistics and decision-making module. By querying the customer information table, the popularity or attention of each shelf can be statistically obtained. For example, if a user needs to query the popularity information of the shelf with ID 1 during the time period (t1, t2), they only need to call the SQL statement "SELECT Count(*) FROM customer_info WHERE shelf_id = 1 AND t > t1 AND t < t2". Here, customer_info is the name of the customer information table. Sorting all the shelves according to the popularity value can obtain the relative popularity information of the shelves. If the user records the product information displayed on each shelf in the data storage module, the customer attention of various products can be obtained in the same way, providing a decision-making reference for the merchant's shelf product display.

[0086] In summary, the present invention proposes a fully automatic high-precision customer movement analysis method. According to the preset mapping matrix and shelf information, this method can automatically analyze the customer movement in the monitored video and associate it with the shelf display, and give the popularity or attention information of the in-store shelves. Since the popularity information of the shelves and products is automatically calculated, there is no need to manually observe the heat map, so the analysis result is more objective and effective.

[0087] In addition, the applicant also found that the final analysis result of the existing algorithm model is several pedestrian trajectories that appear in the video. However, this model does not distinguish whether it is the trajectory of an employee or a customer. That is to say, when performing subsequent heat map analysis, the trajectory information of all pedestrians including customers and employees is considered at the same time, and this employee information will cause certain interference to the analysis result, affecting the accuracy of the final statistical result. In this regard, after step S120, the association method 100 may further include:

[0088] Step S150, based on the employee classification model and the pedestrian appearance information in the pedestrian information, identify the employees of the convenience store from the target image.

[0089] Specifically, a personnel classification module based on a deep neural network can be used to perform the above discrimination function. Among them, the deep neural network means using a dedicated trajectory classification training set to classify each pedestrian target in the trajectory in turn, and classifying the trajectory as an employee trajectory or a customer trajectory according to the majority voting principle. Therefore, pedestrians can be classified into employees and customers through the deep neural network, and then the personnel trajectories can be classified into employee trajectories and customer trajectories through trajectory classification.

[0090] The applicant found that convenience store employees usually wear special colored uniforms, which are obviously different from ordinary customers in appearance. Therefore, a special training set can be constructed to train the personnel classification model. The training set contains a public pedestrian detection dataset and a proprietary dataset made from employee and customer data in store surveillance videos. Based on the employee classification model and the appearance information of pedestrians in pedestrian information, employees wearing special colored uniforms can be distinguished from ordinary customers, and convenience store employees can be identified from the target image.

[0091] Specifically, step S150 may include steps S151 - S153 .

[0092] Step S151, obtaining a training set, wherein the training set is a pedestrian classification feature data set based on historical surveillance videos;

[0093] Step S152, using the training set to train the employee classification model, so that the accuracy of the output result of the employee classification model and the actual result is greater than a set ratio; and

[0094] Step S153: input the image block where each pedestrian target in the target image is located into the employee classification model to identify the pedestrian target as an employee or a customer.

[0095] Specifically, since convenience store employees usually wear special colored uniforms, they look significantly different from ordinary customers. Therefore, a special training set can be constructed to train a dedicated personnel classification model, such as the yolov8_cls image classification model. The training set contains a public pedestrian detection dataset and a proprietary dataset made from employee and customer data in store surveillance videos. For each detected pedestrian target, the image block in the rectangular frame is cropped and the personnel classification model is called to determine whether the pedestrian target is a customer or an employee, expressed as:

[0096] cls(p)→{0,1},

[0097] Where cls() represents the employee classification model, and the classification result 0 represents an employee and 1 represents a customer. In addition, the classification method based on the complete trajectory can also improve the accuracy of employee classification and the robustness to occlusion and illumination changes. Specifically, suppose a complete pedestrian trajectory P contains the target in N frames of images, P = (p1,…,p N ). Classify the N pedestrian targets in the trajectory one by one, and count the number of customers and employees in the result. If the number of customers is greater than the number of employees, classify the trajectory as customers. Otherwise, classify the trajectory as employees. The above classification method based on the complete trajectory can be expressed as:

[0098]

[0099] In addition, after step S130, the association method may further include: step S160, deleting the employee's movement information in the surveillance video. That is, in order to make the analysis result of the customer's movement more reliable, the trajectory information of the customer category is retained, and the trajectory information of the employee category is directly discarded.

[0100] In summary, in order to make the results of customer movement line analysis more accurate, the present invention adopts a classification method based on complete trajectories to improve the accuracy of employee classification and the robustness to occlusion and illumination changes. After the pedestrian trajectory tracking is completed, each detection box corresponding to the current pedestrian trajectory is input into the personnel classification model for classification, and the pedestrian is classified as an employee or customer according to the majority voting principle. The trajectory information classified as a customer is retained, and the trajectory classified as an employee is directly discarded. That is, the present invention filters out the employee information of the store during pedestrian detection and tracking, and only considers the customer's trajectory information during thermal analysis, so that the analysis results are more reliable, and thus more accurate movement line and heat information can be obtained.

[0101] It is worth noting that the various steps in the present invention are an organic whole, and together constitute a complete customer flow association and analysis system. Figure 3 As shown in the figure, first, it is necessary to use a camera to capture video data to collect images of each frame; secondly, it is necessary to perform image preprocessing, and the accuracy of the subsequent position mapping module can be improved by distortion correction, scale transformation, etc., which serves as the basis for subsequent analysis; then, it is necessary to use a pedestrian detection model to detect pedestrians in the image based on the original data in the image, and determine whether it is an existing trajectory by matching the position features with the appearance features, and create a new trajectory or add it to the end of the existing trajectory based on the determination result; wherein, it is also possible to use a personnel classification model to identify whether it is an employee based on the appearance features of the pedestrian, and discard the employee information, and calculate the final customer movement line data; finally, based on the shelf display data input by the user, the refined pedestrian detection and recognition data is mapped to the actual store floor, and the customer movement line information and the shelf display information are associated according to the mapped position information, and the final customer movement line information and shelf display information association analysis results are obtained and stored in the database.

[0102] Through the above technical solution, the improvements and beneficial effects of the present invention are as follows:

[0103] Compared with existing technical methods, the present invention can accurately analyze customer movement lines in surveillance videos and associate them with shelf displays in an automated manner, without the need for manual observation and judgment, thus avoiding labor costs and subjective biases in the decision-making process. In terms of accuracy improvement, the present invention achieves coordinate mapping between video images and real stores through distortion correction and position mapping, avoiding errors caused by image distortion and camera projection, thereby solving the problem of low accuracy in existing customer movement line analysis technologies. In terms of automated analysis, the present invention utilizes a multi-target detection model that can process and analyze surveillance video data in real time, achieve accurate identification of customer movement lines within stores, and automatically associate the identified customer movement line trajectory information with shelf display information at specific locations.

[0104] Through the information on the association between customer traffic and shelf display provided by the present invention, the popularity or attention of the shelf can be automatically analyzed and obtained later, thereby providing a decision-making reference for the shelf display of the merchant. Therefore, the present invention can effectively improve the operating efficiency and intelligence level of the store, thereby improving the customer satisfaction and sales revenue of the store.

[0105] In addition, the technical solution provided by the present invention only requires the user to input a small amount of data in the early stage, and the popularity of the shelf can be automatically analyzed based on the customer movement lines in the video during the subsequent system operation, without the need for manual observation and manual judgment. In actual operation, if the merchant updates the shelf layout in the store, it only needs to modify the location information of the shelf on the digital map in the data storage module, and after updating the product display, it is possible to quickly obtain new shelf heat analysis results without modifying any input, which greatly facilitates user use and reduces the difficulty of system promotion and maintenance.

[0106] On the other hand, the present invention also provides a system 200 for associating customer traffic flow and shelf display in a convenience store, such as Figure 4 As shown, the association system 200 may include:

[0107] The image processing device 210 is used to perform distortion correction and scale transformation on each frame of the monitoring image in the monitoring video according to the pre-stored camera parameter and distortion parameter information, so as to obtain a target image corresponding to each frame of the monitoring image;

[0108] A pedestrian recognition device 220, used to recognize pedestrian information of any pedestrian in the target image based on a multi-target detection model;

[0109] A movement line determination device 230, used to determine the movement line information of the pedestrian in the surveillance video according to the pedestrian information in each surveillance image frame of the surveillance video; and

[0110] The associating device 240 is used to associate the movement information of each pedestrian in the surveillance video with the shelf display information of the convenience store.

[0111] In one embodiment, the camera parameters include a principal point and a focal length, and the image processing device 210 is specifically used to perform the following functions: according to the principal point [px, py], the focal length [fx, fy] and any pixel point p (x, y) of each frame of the surveillance image in the surveillance video, the first pixel point p' (x', y') corresponding to the pixel point is determined by the following formula:

[0112]

[0113]

[0114] Wherein, [cx, cy] is the center point of the monitoring image after transformation.

[0115] In one embodiment, the distortion parameters include radial distortion parameters and tangential distortion parameters, and the image processing device 210 is specifically used to perform the following functions: according to the radial distortion parameters [k2, k4], the tangential distortion parameters [tx, ty] and the first pixel point p'(x', y'), the following formula is used to determine the second pixel point p"(x", y") after radial distortion and the third pixel point p"'(x"', y"') after tangential distortion:

[0116] x″=x′(1+k2×r 2 +k4×r 4 ),

[0117] y″=y′(1+k2×r 2 +k4×r 4 ),

[0118] x″′=x″+tx×y″,

[0119] y″′=y″+ty×x″,

[0120] Where r is the distance from point p to the center point of the monitoring image, which is

[0121] In one embodiment, the pedestrian recognition device 220 is specifically used to perform the following functions: based on the multi-target detection model, identify the first pedestrian information of the pedestrian in the target image and identify the second pedestrian information of all pedestrians in the previous frame image of the target image; determine whether the pedestrian is an existing pedestrian or a newly appeared pedestrian based on the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information; when the pedestrian is the existing pedestrian, associate the pedestrian position information of the pedestrian to the end of the pedestrian's historical trajectory; and when the pedestrian is the newly appeared pedestrian, create a new trajectory for the pedestrian and use the pedestrian position information of the pedestrian as the starting point of the new trajectory.

[0122] In one embodiment, the pedestrian information includes pedestrian appearance information, pedestrian position information and trajectory prediction information, and the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information includes: a first similarity in pedestrian appearance information, a second similarity in pedestrian position information and a third similarity in trajectory prediction information, wherein the first similarity is determined by the Euclidean distance measurement between vectors, and the second similarity and the third similarity are determined by the Euclidean distance measurement between coordinates.

[0123] In one embodiment, the determining of the pedestrian as an existing pedestrian or a newly appeared pedestrian based on the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information includes: when the first similarity, the second similarity and the third similarity between a pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian all meet the set conditions, the pedestrian is determined as the existing pedestrian; and when the first similarity, the second similarity and the third similarity between any pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian do not simultaneously meet the set conditions, the pedestrian is determined as the newly appeared pedestrian.

[0124] In one embodiment, the association system also includes: a personnel classification device, which is used to identify the employees of the convenience store from the target image based on an employee classification model and the pedestrian appearance information in the pedestrian information after the pedestrian information of any pedestrian in the target image is identified based on the multi-target detection model; and after determining the movement line information of the pedestrian in the surveillance video based on the pedestrian information in each frame of the surveillance video, the movement line information of the employee in the surveillance video is deleted.

[0125] In one embodiment, the personnel classification device is specifically used to perform the following functions: obtain a training set, wherein the training set is a pedestrian classification feature data set based on historical surveillance videos; use the training set to train the employee classification model so that the accuracy of the output result of the employee classification model and the actual result is greater than a set ratio; and input the image block where each pedestrian target in the target image is located into the employee classification model to identify the pedestrian target as an employee or a customer.

[0126] In one embodiment, the association device 240 is specifically used to perform the following functions: based on the homography matrix between the ground in the surveillance video and the ground of the convenience store, the pedestrian position information of any pedestrian in the target image is mapped to the ground coordinates of the convenience store to obtain the mapping coordinates of the pedestrian on the ground of the convenience store; according to preset shelf display information, the shelf that is closest to the mapping coordinates of the pedestrian and the distance between the two is less than a set distance is determined as the target shelf; and the pedestrian's number, mapping coordinates, and time information of the target image are associated with the target shelf.

[0127] It is worth noting that the various modules in the present invention are an organic whole, which together constitute a complete customer movement line analysis system. Among them, the customer movement line analysis system may include: a data storage module, a data collection and processing module, a pedestrian detection and tracking module, a personnel classification module, a location mapping module, and a shelf statistics and decision-making module. Specifically, the data storage module is used to store shelf information and customer information. The data collection and processing module is used to collect and process monitoring video data. The pedestrian detection and tracking module is used to detect pedestrians and track pedestrian trajectories from the video to obtain pedestrian trajectories represented by a set of pixel point coordinates. The personnel classification module is used to classify pedestrian trajectories into employee trajectories and customer trajectories. The location mapping module is used to map the pixel point coordinates of the customer trajectory to the store ground coordinates in the real world. The shelf statistics and decision-making module is used to calculate the popularity or attention of each shelf.

[0128] The execution logic between the above functional modules can refer to Figure 5 As shown: The data storage module contains user input data and is the prerequisite for the operation of the entire system. The data collection and processing module collects video data as the basis for subsequent module analysis, and improves the accuracy of the subsequent position mapping module through distortion correction. The pedestrian detection and tracking module obtains the original data of video analysis, the personnel classification module further refines the data, and finally the position mapping module calculates the final customer movement data. The shelf statistics and decision-making module responds to customer needs and obtains the final correlation analysis results between customer movement and shelf display.

[0129] Through the above technical solution, the improvements and beneficial effects of the present invention are as follows:

[0130] Compared with existing technical methods, the present invention can accurately analyze customer movement lines in surveillance videos and associate them with shelf displays in an automated manner, without the need for manual observation and judgment, thus avoiding labor costs and subjective biases in the decision-making process. In terms of accuracy improvement, the present invention achieves coordinate mapping between video images and real stores through distortion correction and position mapping, avoiding errors caused by image distortion and camera projection, thereby solving the problem of low accuracy in existing customer movement line analysis technologies. In terms of automated analysis, the present invention utilizes a multi-target detection model that can process and analyze surveillance video data in real time, achieve accurate identification of customer movement lines within stores, and automatically associate the identified customer movement line trajectory information with shelf display information at specific locations.

[0131] Through the information on the association between customer traffic and shelf display provided by the present invention, the popularity or attention of the shelf can be automatically analyzed and obtained later, thereby providing a decision-making reference for the shelf display of the merchant. Therefore, the present invention can effectively improve the operating efficiency and intelligence level of the store, thereby improving the customer satisfaction and sales revenue of the store.

[0132] In addition, the technical solution provided by the present invention only requires the user to input a small amount of data in the early stage, and the popularity of the shelf can be automatically analyzed based on the customer movement lines in the video during the subsequent system operation, without the need for manual observation and manual judgment. In actual operation, if the merchant updates the shelf layout in the store, it only needs to modify the location information of the shelf on the digital map in the data storage module, and after updating the product display, it is possible to quickly obtain new shelf heat analysis results without modifying any input, which greatly facilitates user use and reduces the difficulty of system promotion and maintenance.

[0133] In addition, in one embodiment, the present invention may further include a processor and a memory, and each of the above-mentioned devices 210, 220, 230, 240, etc. is stored in the memory as a program unit, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions.

[0134] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the purpose of the present invention is achieved by adjusting kernel parameters.

[0135] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0136] An embodiment of the present invention provides a storage medium having a program stored thereon, which, when executed by a processor, implements a method for associating customer traffic flow and shelf display in the convenience store.

[0137] An embodiment of the present invention provides a processor, which is used to run a program, wherein when the program is run, a method for associating customer traffic flow and shelf display in the convenience store is executed.

[0138] An embodiment of the present invention provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, each step of the method for associating customer traffic lines and shelf displays in the convenience store is implemented. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0139] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing the various steps of the method for associating customer traffic flow and shelf display in the convenience store as described above.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0146] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0148] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for associating customer traffic flow and shelf display in a convenience store, characterized in that: The association method comprises: According to the pre-stored camera parameters and distortion parameters, each frame of the monitoring image in the monitoring video is subjected to distortion correction and scale transformation to obtain a target image corresponding to each frame of the monitoring image; Based on the multi-target detection model, identifying pedestrian information of any pedestrian in the target image; Determining the movement line information of the pedestrian in the surveillance video according to the pedestrian information of the pedestrian in each surveillance image frame of the surveillance video; and The movement line information of each pedestrian in the surveillance video is associated with the shelf display information of the convenience store.

2. The association method according to claim 1, characterized in that: The camera parameters include a principal point and a focal length, and the scaling of each frame of the surveillance image in the surveillance video includes: According to the principal point [px,py], the focal length [fx,fy] and any pixel point p(x,y) of each frame of the surveillance image in the surveillance video, the first pixel point p'(x',y') corresponding to the pixel point is determined by the following formula: Wherein, [cx, cy] is the coordinate of the center point of the monitoring image after transformation.

3. The association method according to claim 2, characterized in that: The distortion parameters include radial distortion parameters and tangential distortion parameters, and the distortion correction of each surveillance image frame in the surveillance video includes: According to the radial distortion parameter [k2, k4], the tangential distortion parameter [tx, ty] and the first pixel point p'(x', y'), the second pixel point p" (x", y") after radial distortion and the third pixel point p"' (x"', y"') after tangential distortion are determined by the following formula: x″=x′(1+k2×r 2 +k4×r 4 ), y″=y′(1+k2×r 2 +k4×r 4 ), x″′=x″+tx×y″, y″′=y″+ty×x″, Where r is the distance from point p(x,y) to the center point of the monitoring image, which is 4. The association method according to claim 1, characterized in that: The identifying pedestrian information of any pedestrian in the target image based on the multi-target detection model includes: Based on the multi-target detection model, first pedestrian information of the pedestrian in the target image is identified, and second pedestrian information of all pedestrians in a frame image before the target image is identified; Determining whether the pedestrian is an existing pedestrian or a newly appeared pedestrian based on the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information; In the case where the pedestrian is the existing pedestrian, associating the pedestrian position information of the pedestrian to the end of the pedestrian's historical track; and In the case that the pedestrian is the newly appeared pedestrian, a new track is created for the pedestrian and the pedestrian position information of the pedestrian is used as the starting point of the new track.

5. The association method according to claim 4, characterized in that: The pedestrian information includes pedestrian appearance information, pedestrian position information and trajectory prediction information, and the similarity between the first pedestrian information and each pedestrian information in the second pedestrian information includes: a first similarity in pedestrian appearance information, a second similarity in pedestrian position information and a third similarity in trajectory prediction information, The first similarity is determined by using the Euclidean distance measurement between vectors, and the second similarity and the third similarity are determined by using the Euclidean distance measurement between coordinates.

6. The association method according to claim 5, characterized in that: The determining, based on the similarity between each pedestrian information in the first pedestrian information and the second pedestrian information, whether the pedestrian is an existing pedestrian or a newly appeared pedestrian includes: If the first similarity, the second similarity, and the third similarity between the pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian all meet set conditions, determining the pedestrian as the existing pedestrian; and If the first similarity, the second similarity, and the third similarity between any pedestrian information in the second pedestrian information and the first pedestrian information of the pedestrian cannot simultaneously satisfy the set conditions, the pedestrian is determined as the newly appeared pedestrian.

7. The association method according to claim 1, characterized in that: After identifying pedestrian information of any pedestrian in the target image based on the multi-target detection model, the association method further includes: Based on the employee classification model and the pedestrian appearance information in the pedestrian information, the employee of the convenience store is identified from the target image, After determining the pedestrian's movement line information in the surveillance video according to the pedestrian information in each surveillance image frame of the surveillance video, the association method further includes: The movement information of the employee in the surveillance video is deleted.

8. The association method according to claim 7, characterized in that: The identifying the employee of the convenience store from the target image based on the employee classification model and the pedestrian appearance information in the pedestrian information includes: Obtaining a training set, wherein the training set is a pedestrian classification feature dataset based on historical surveillance videos; Using the training set to train the employee classification model so that the accuracy of the output result of the employee classification model and the actual result is greater than a set ratio; and The image block where each pedestrian target in the target image is located is input into the employee classification model to identify the pedestrian target as an employee or a customer.

9. The association method according to claim 1, characterized in that: The associating the movement line information of each pedestrian in the surveillance video with the shelf display information of the convenience store includes: Based on the homography matrix between the ground in the surveillance video and the ground of the convenience store, the pedestrian position information of any pedestrian in the target image is mapped to the ground coordinates of the convenience store to obtain the mapping coordinates of the pedestrian on the ground of the convenience store; According to the preset shelf display information, the shelf that is closest to the mapped coordinates of the pedestrian and the distance between the two is less than a set distance is determined as the target shelf; and The pedestrian number, mapping coordinates, and time information of the target image are associated with the target shelf.

10. A system for associating customer traffic flow and shelf display in a convenience store, characterized in that: The association system comprises: An image processing device, used to perform distortion correction and scale transformation on each frame of monitoring image in the monitoring video according to pre-stored camera parameters and distortion parameter information, so as to obtain a target image corresponding to each frame of monitoring image; A pedestrian recognition device, used to recognize pedestrian information of any pedestrian in the target image based on a multi-target detection model; A movement line determination device, used to determine the movement line information of the pedestrian in the monitoring video according to the pedestrian information in each monitoring image frame of the monitoring video; and The associating device is used to associate the movement line information of each pedestrian in the monitoring video with the shelf display information of the convenience store.

Images