A moving target tracking and analysis method and system based on video surveillance
The video surveillance system obtains consumer location and shopping operation information, combines multi-objective tracking and key point detection algorithms to analyze consumer behavior, solves the problem that existing technology cannot track and analyze consumer behavior, and realizes the attraction assessment of the product and marketing strategy adjustments have improved the marketing efficiency of the mall.
Patent Information
- Application Number
- CN202411216837.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing video surveillance system cannot effectively track and analyze consumers' behavior in the mall, cannot make timely marketing strategies based on the consumer's behavior in the video surveillance screen, and cannot understand consumers' preferences and interests.
By obtaining consumer location information and connectivity domains in each video surveillance screen, using DeepSORT multi-target tracking algorithm to identify consumers, combining human key point detection model, calculate consumer shopping operations and purchase coefficients, and count the number of people browsing products and Sales, adjust marketing strategy.
It realizes tracking and analysis of consumer behavior, understands consumers' preferences and interests for products, adjusts marketing strategies based on the attractiveness of products, and improves the marketing efficiency of the mall.
Smart Images

Figure CN118761795B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a moving target tracking and analysis method and system based on video monitoring. Background Art
[0002] With the development of technology, video surveillance has been applied to various industries. Tracking and analyzing moving targets in video surveillance images can not only improve security management efficiency, but also provide data support and intelligent solutions for business decision-making, public management and other aspects.
[0003] The application of video surveillance in shopping malls can monitor commodities and consumers, but the consumption situations of consumers in shopping malls vary, and the information displayed during the shopping process is complex and diverse. The existing video surveillance only monitors simple behaviors of consumers, and cannot track and analyze the preferences and interests of consumers based on the behaviors of consumers in the video surveillance images, and thus cannot make timely adjustments to the marketing strategies of commodities based on the behaviors of consumers in the video surveillance images. Summary of the invention
[0004] The present invention provides a moving target tracking and analysis method and system based on video monitoring to solve the existing problems.
[0005] The present invention provides a moving target tracking and analysis method and system based on video surveillance, which adopts the following technical solutions:
[0006] An embodiment of the present invention provides a moving target tracking and analysis method based on video surveillance, the method comprising the following steps:
[0007] Obtain each frame of video surveillance images from each camera, and obtain the location information and connected domain of each consumer in each frame of video surveillance images, as well as the location information of each commodity and the daily sales of each commodity according to each frame of video surveillance images from each camera;
[0008] Based on the change distance of the connected domain of the same consumer in different frames of video surveillance images, all the consumer's purchase start and end times are obtained;
[0009] Obtaining the number of shopping operations of the consumer according to the change distance of the key points of the hand in the connected domain between the start time and the end time of the shopping of the consumer;
[0010] Obtaining the purchase coefficient of the consumer for purchasing the product according to the product corresponding to the location information of the consumer at the start of the purchase and the consumer's purpose of purchasing the product;
[0011] Based on the number of people who browse the same product in one day and the purchase coefficient of the same product by different consumers, the attractiveness of the product to consumers on that day is obtained. According to the attractiveness of all products to consumers on that day, the marketing strategy of the product is adjusted.
[0012] Furthermore, the specific steps of obtaining the location information and connected domain of each consumer in each frame of video surveillance picture, as well as the location information of each commodity and the daily sales of each commodity in each frame of video surveillance picture of each camera are as follows:
[0013] The DeepSORT multi-target tracking algorithm is used to identify the area where consumers are located, and the location information and connected domain of each consumer in each frame of video surveillance is obtained; the location of each product in the mall is obtained through manual marking and collection.
[0014] Furthermore, the obtaining of all shopping start and end times of consumers includes the following specific steps:
[0015] A centroid change threshold is preset. For the i-th frame of video surveillance, if the centroid change distance between the consumer's i-th frame and the i+1-th frame is less than the centroid change threshold, the moment corresponding to the i-th frame of surveillance video is recorded as the consumer's purchase start time, and the next moment is judged until there is a constant c that makes the centroid change distance between the i+c-1-th frame and the i+c-th frame greater than or equal to the centroid change threshold, then the moment corresponding to the i+c-th frame of surveillance video is recorded as the consumer's purchase end time.
[0016] Furthermore, the specific method for obtaining the centroid change distance is as follows:
[0017] Construct a coordinate system for each frame of video surveillance image captured by the camera, taking the pixel point in the lower left corner of the surveillance video image as the origin of the coordinate system, and obtain the coordinate system of the first frame. The centroid of the connected domain of the consumer in the i+1th frame and the i+1th frame of video surveillance images is calculated according to the The distance between the centroid of the connected domain of the consumer in the i-th frame and the i+1-th frame of video surveillance screen is obtained to obtain the centroid change distance of the consumer in the i-th frame and the i+1-th frame.
[0018] Furthermore, the obtaining of the number of shopping operations of the consumer includes the following specific steps:
[0019] Between the start time and the end time of shopping, the coordinates of the consumer's hand key points are obtained through the human body key point detection model; the consumer's hand change distance is obtained, and a hand change threshold is preset. When the consumer's hand change distance is greater than the hand change threshold, the consumer is recorded as having performed a shopping operation; the number of shopping operations performed by the consumer between the start time and the end time of shopping is obtained.
[0020] Furthermore, the step of obtaining the purchase coefficient of the consumer's selection of the commodity includes the following specific steps:
[0021] For For each consumer, between the time when the purchase starts and the time when the purchase ends, the The location information of each consumer is matched with the location information of the product to obtain the The commodity object when a consumer performs a shopping operation is recorded as A consumer chooses the nth product;
[0022] The first The time corresponding to the surveillance video screen where a consumer first appears in all surveillance video screens is recorded as The initial moment of a consumer , No. Consumers choose The specific method of obtaining the purchase coefficient of a product is as follows:
[0023]
[0024] In the formula, Indicates Consumers choose The purchase coefficient of a product; Indicates The initial moment of a consumer; Indicates The time when a consumer starts to purchase the nth product; Indicates The moving distance of a consumer from the initial moment to the moment when he starts to purchase the nth product; It means that before the mth consumer starts to purchase the nth product, the The number of times a consumer chooses other products, but the number of shopping operations for other products is 0 between the start time and the end time of the purchase; It represents the difference between the start time and the end time of the purchase of the nth product by the mth consumer. represents a hyperparameter; represents the linear normalization function.
[0025] Furthermore, the specific method of obtaining the moving distance is as follows:
[0026] The sum of the distances of the centroid changes of all adjacent frames of video surveillance images of the mth consumer from the initial moment to the start of purchasing is recorded as the moving distance of the mth consumer from the initial moment to the start of purchasing the nth product.
[0027] Furthermore, the obtaining of the attractiveness of the product to consumers on that day includes the following specific steps:
[0028]
[0029] In the formula, Indicates the attractiveness of the nth commodity on the rth day; It represents the average purchase coefficient of all consumers who purchase the nth product on the rth day; represents the total number of consumers on day r; Indicates the number of Number of visitors to a product; represents the sum of sales of all commodities on day r; Indicates the number of Sales of products.
[0030] Furthermore, the specific method of obtaining the number of visitors is as follows:
[0031] Statistics The number of consumers who have a purchase start time for a product in a day and whose shopping operations from the purchase start time to the purchase end time are odd is recorded as The number of people who viewed the product.
[0032] The present invention also proposes a motion target tracking and analysis system based on video surveillance, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program stored in the memory to implement the steps of the aforementioned motion target tracking and analysis method based on video surveillance.
[0033] The beneficial effects of the technical solution of the present invention are: by obtaining the position information and connected domain of each consumer in each frame of video surveillance screen and the position information of each product in each frame of video surveillance screen of each camera, the same consumer can be tracked and analyzed; based on the distance change of the same consumer in different frames of video surveillance screen, several purchase start times and purchase end times of the consumer are obtained; according to the change distance of the consumer's hand when shopping, the number of shopping operations of the consumer is obtained, which shows the specific situation of the consumer's purchase of goods, and combined with the consumer's purposeful expression of the purchase of goods, the consumer's purchase coefficient for the goods is obtained, indicating each consumer's preference demand for the goods; according to the number of people who browse the goods and the purchase coefficient, the attractiveness of the goods to consumers on that day is obtained, and the marketing strategy of the goods is adjusted. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 The present invention is a flow chart of the steps of a moving target tracking and analysis method based on video monitoring. DETAILED DESCRIPTION
[0036] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a moving target tracking and analysis method and system based on video surveillance proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0037] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0038] The following is a detailed description of a method and system for tracking and analyzing a moving target based on video surveillance provided by the present invention in conjunction with the accompanying drawings.
[0039] See also Figure 1 , which shows a flow chart of a method for tracking and analyzing a moving target based on video surveillance provided by an embodiment of the present invention, the method comprising the following steps:
[0040] Step S001, obtain each frame of video surveillance screen from each camera, and obtain the location information and connected domain of each consumer in each frame of video surveillance screen, as well as the location information of each product according to each frame of video surveillance screen from each camera; match the same consumer according to the moving distance of each consumer in the video surveillance screen of consecutive frames.
[0041] The purpose of this embodiment is to formulate a mall marketing strategy based on tracking and analyzing the behavioral performance of consumers in the mall surveillance video; therefore, it is first necessary to continuously install multiple surveillance cameras on each row of shelves inside the mall, covering the main product display areas and crowded areas, and obtain each frame of video surveillance screen in the surveillance video of each camera in the mall every day, and identify the area where the consumer is located based on the DeepSORT multi-target tracking algorithm, obtain the connected domain and location information of each consumer in each frame of video surveillance screen; obtain the daily sales of each product; it should be noted that the DeepSORT multi-target tracking algorithm is an existing public known technology and will not be repeated in this embodiment, and this embodiment does not specifically limit the specific installation angle and number of cameras, and the installation method is determined according to the actual size of the mall.
[0042] Furthermore, since the positions of commodities in the mall are fixed, the position of each commodity in the mall is obtained through manual marking and collection.
[0043] It should be noted that the movement trajectories of consumers in shopping malls are complex and changeable, and there is a lot of uncertainty. By analyzing the continuous multi-frame video surveillance images, we can understand the location of consumers' preferred products. Therefore, it is first necessary to lock each consumer through the moving distance of each consumer in the continuous frame video surveillance images to determine the movement trajectory of each consumer.
[0044] Specifically, after each consumer enters the mall, the initial position information and initial time of each consumer in the surveillance video are obtained; in the process from the initial position information to the moment when the consumer disappears from all frames of video surveillance screens, the moving distance and disappearance time of the position information of each consumer in each frame of video surveillance screen are obtained through the DeepSORT multi-target tracking algorithm; it should be noted that the disappearance of the consumer from all frames of video surveillance screens described in this embodiment means disappearing from the video surveillance screens of all surveillance cameras. For the same consumer in different surveillance videos, this embodiment obtains all corner points of the connected domain of each consumer in the last frame of video surveillance screen in the surveillance video of the current camera through a corner detection algorithm, performs corner point matching in the surveillance videos of other surveillance cameras to obtain the video surveillance screen of the same consumer as the continuation video of the surveillance video of the current camera, until the consumer disappears from all surveillance cameras, and obtains the moving distance and disappearance time of the consumer's position information. The corner point detection and corner point matching algorithms are both existing public known technologies and are not described in detail in this embodiment.
[0045] Step S002, obtaining the consumer's purchase of goods according to the change of the consumer's position in the continuous frames; obtaining the consumer's purchase of goods according to the magnitude of the change of the consumer's body parts and hands.
[0046] It should be noted that the consumer's movement trajectory information can be displayed based on the matched video surveillance images of the same consumer; when the consumer stays in front of the product, the position of the consumer in the continuous frames of the video surveillance images does not change; there are two situations in which the consumer stays in front of the product: need to buy and not need to buy, so it is necessary to judge whether the consumer buys the product in front of him based on the performance of the consumer's hand movements. When the consumer stays in front of the product, the corner points on the consumer's body have a smaller offset than the corner points on the hands in the continuous frames of the video surveillance images.
[0047] Specifically, for any consumer, a coordinate system is constructed for each frame of the video surveillance screen captured by the camera, and the pixel point in the lower left corner of the surveillance video screen is used as the origin of the coordinate system to obtain the first The centroid of the connected domain of the consumer in the i+1th frame and the i+1th frame of video surveillance images is calculated according to the The distance between the centroid of the connected domain of the consumer in the frame and the i+1th frame of the video surveillance screen is obtained to obtain the centroid change distance of the consumer in the i-th frame and the i+1th frame. In this embodiment, the Euclidean distance between the centroid of the consumer in the i-th frame and the i+1th frame is used as the centroid change distance. Other embodiments may be described in other ways, which are not specifically limited in this embodiment.
[0048] A centroid change threshold is preset. For the i-th frame of video surveillance, if the centroid change distance between the i-th frame and the i+1-th frame of the consumer is less than the centroid change threshold, the moment corresponding to the i-th frame of surveillance video is recorded as the consumer's purchase start time, and the next moment is determined until there is a constant c such that the centroid change distance between the i+c-1-th frame and the i+c-th frame is greater than or equal to the centroid change threshold, then the moment corresponding to the i+c-th frame of surveillance video is recorded as the consumer's purchase end time; it should be noted that the preset centroid change threshold of 30 in this embodiment is described by taking 5 frames as an example, other embodiments may set other values, which are not specifically limited in this embodiment;
[0049] Furthermore, between the start time and the end time of the purchase, the coordinates of the consumer's hand key points are obtained through the Human Keypoints Detection model. The consumer's hand change distance is obtained according to the method for obtaining the centroid change distance, and a hand change threshold is preset. When the consumer's hand change distance is greater than the hand change threshold, it means that the consumer may have made a shopping action, and the consumer is recorded as having performed a shopping operation at this time; the number of shopping operations of the consumer between the start time and the end time of the purchase is obtained. When the number of shopping operations of the consumer is an odd number, it means that the consumer has taken the goods from the shelf and has not put them back. When the number of shopping operations of the consumer is an even number, it means that the consumer has taken the goods from the shelf and put them back. It should be noted that if the number of shopping operations of the consumer between the start time and the end time of the purchase is 0, it means that the consumer has just stayed and has not performed a shopping operation.
[0050] Step S003: Obtain the consumer's purchase coefficient for the commodity based on the commodity corresponding to the location information of the consumer at the start of shopping and the consumer's purpose expression for the commodity.
[0051] It should be noted that when consumers are shopping, those who have pre-order target products walk faster and stop less often in front of other products before reaching the pre-order target products; consumers without pre-order targets walk slower and stop more often in front of other products because they shop aimlessly; when consumers take a product, it indicates that they have a desire to buy the product.
[0052] Specifically, for For each consumer, between the time when the purchase starts and the time when the purchase ends, the The location information of each consumer is matched with the location information of the product to obtain the The commodity object when a consumer performs a shopping operation is recorded as A consumer chooses the nth product;
[0053] The first The time corresponding to the first surveillance video screen in which a consumer appears among all surveillance video screens is recorded as The initial moment of a consumer, Consumers choose The specific method of obtaining the purchase coefficient of a product is as follows:
[0054]
[0055] In the formula, Indicates Consumers choose The purchase coefficient of a product; Indicates The initial moment of a consumer; Indicates Consumers choose The time when the purchase of a product starts; Indicates The moving distance of a consumer from the initial moment to the purchase start moment of the nth commodity, wherein the moving distance is the sum of the distances of the centroid changes of all adjacent frames of video surveillance images from the initial moment to the purchase start moment; Indicates Consumers choose Before the purchase of the product begins, The number of times a consumer chooses other products, but the number of shopping operations for other products is 0 between the start time and the end time of the purchase; Indicates Consumers choose The difference between the purchase start time and the purchase end time of a product, represents a hyperparameter; represents the linear normalization function.
[0056] Indicates The speed at which a consumer moves from the initial position to the position where the product is purchased. It means that when consumers walk faster, stop less frequently in front of other products and stay longer in front of the purchased product, it means that consumers buy this product with a purpose and have a higher purchase coefficient for this product.
[0057] Similarly, obtain the consumer's purchase coefficient for each purchased commodity.
[0058] Step S004: According to the purchase coefficient of each consumer for the same product and the sales situation of the product, as well as the flow of people in the mall, the attractiveness of each product is obtained every day, and the marketing strategy of the product is adjusted.
[0059] It should be noted that when a certain product is more attractive to consumers, the number of people who buy the product will account for a large proportion of the total number of consumers on that day, and the sales volume will be large. Regardless of whether the consumer has taken the product during the time period when the product is currently in the store, the more times each consumer stays in front of the same product, the more attractive the product is; at the same time, the sales of various products on that day and the purchases of consumers are also important manifestations of the attractiveness of the product.
[0060] Specifically, statistics The number of consumers who have a purchase start time for a product in a day and whose shopping operations from the purchase start time to the purchase end time are odd is recorded as Number of visitors to products, The attractiveness of a product to consumers on that day:
[0061]
[0062] In the formula, Indicates the attractiveness of the nth commodity on the rth day; It represents the average purchase coefficient of all consumers who purchase the nth product on the rth day; represents the total number of consumers on day r; Indicates the number of The number of people who viewed the product. represents the sum of sales of all commodities on day r; Indicates the number of Similarly, we can obtain the attractiveness of each product every day.
[0063] The mall adjusts its marketing strategy based on the degree of appeal of different commodities to consumers, by increasing the purchase quantity or adjusting the position of highly attractive commodities, and reducing the purchase quantity of less attractive commodities or selling them in conjunction with more attractive commodities.
[0064] So far, the present invention is completed.
[0065] The present invention also provides a motion target tracking and analysis system based on video surveillance, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program stored in the memory to implement the steps of the aforementioned motion target tracking and analysis method based on video surveillance.
[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A moving target tracking and analysis method based on video surveillance, characterized in that: The method comprises the following steps: Obtain each frame of video surveillance images from each camera, and obtain the location information and connected domain of each consumer in each frame of video surveillance images, as well as the location information of each commodity and the daily sales of each commodity according to each frame of video surveillance images from each camera; Based on the change distance of the connected domain of the same consumer in different frames of video surveillance images, all the consumer's purchase start and end times are obtained; Obtaining the number of shopping operations of the consumer according to the change distance of the key points of the hand in the connected domain between the start time and the end time of the shopping of the consumer; Obtaining the purchase coefficient of the consumer for purchasing the product according to the product corresponding to the location information of the consumer at the start of the purchase and the consumer's purpose of purchasing the product; Based on the number of people who browse the same product in a day and the purchase coefficient of different consumers for the same product, the attractiveness of the product to consumers on that day is obtained, and the marketing strategy of the product is adjusted according to the attractiveness of all products to consumers on that day; The specific steps of obtaining the location information and connected domain of each consumer in each frame of video surveillance picture, as well as the location information of each commodity and the daily sales of each commodity in each frame of video surveillance picture of each camera are as follows: The DeepSORT multi-target tracking algorithm is used to identify the area where consumers are located, and obtain the location information and connected domain of each consumer in each frame of video surveillance. The location of each product in the mall is obtained through manual marking and collection. The specific steps of obtaining all the shopping start and end times of the consumer are as follows: A centroid change threshold is preset. For the i-th frame of video surveillance, if the centroid change distance between the i-th frame and the i+1-th frame of the consumer is less than the centroid change threshold, the moment corresponding to the i-th frame of surveillance video is recorded as the consumer's purchase start time, and the next moment is determined until there is a constant c such that the centroid change distance between the i+c-1-th frame and the i+c-th frame is greater than or equal to the centroid change threshold, then the moment corresponding to the i+c-th frame of surveillance video is recorded as the consumer's purchase end time; The specific method for obtaining the centroid change distance is as follows: Construct a coordinate system for each frame of video surveillance image captured by the camera, taking the pixel point in the lower left corner of the surveillance video image as the origin of the coordinate system, and obtain the coordinate system of the first frame. The centroid of the connected domain of the consumer in the i+1th frame and the i+1th frame of video surveillance images is calculated according to the The distance between the centroid of the connected domain of the consumer in the i-th frame and the i+1-th frame of the video surveillance screen is obtained to obtain the centroid change distance of the consumer in the i-th frame and the i+1-th frame; The specific steps of obtaining the number of shopping operations of the consumer are as follows: Between the start time and the end time of shopping, the coordinates of the key points of the consumer's hand are obtained through the human key point detection model; the change distance of the consumer's hand is obtained, and a hand change threshold is preset. When the change distance of the consumer's hand is greater than the hand change threshold, the consumer is recorded as having performed a shopping operation; the number of shopping operations of the consumer between the start time and the end time of shopping is obtained; The specific steps of obtaining the purchase coefficient of the consumer's purchase of the commodity are as follows: For For each consumer, between the time when the purchase starts and the time when the purchase ends, the The location information of each consumer is matched with the location information of the product to obtain the The commodity object when a consumer performs a shopping operation is recorded as A consumer chooses the nth product; The first The time corresponding to the surveillance video screen where a consumer first appears in all surveillance video screens is recorded as The initial moment of a consumer , No. Consumers choose The specific method of obtaining the purchase coefficient of a product is as follows: In the formula, Indicates Consumers choose The purchase coefficient of a product; Indicates The initial moment of a consumer; Indicates The time when a consumer starts to purchase the nth product; Indicates The moving distance of a consumer from the initial moment to the moment when he starts to purchase the nth product; It means that before the mth consumer starts to purchase the nth product, the The number of times a consumer chooses other products, but the number of shopping operations for other products is 0 between the start time and the end time of the purchase; It represents the difference between the start time and the end time of the purchase of the nth product by the mth consumer. represents a hyperparameter; represents the linear normalization function; The specific method of obtaining the moving distance is as follows: The sum of the distances of the centroid changes of all adjacent frames of the video surveillance images of the mth consumer from the initial moment to the start of the purchase is recorded as the moving distance of the mth consumer from the initial moment to the start of the purchase of the nth product; The specific steps of obtaining the attractiveness of the product to consumers on that day are as follows: In the formula, Indicates the attractiveness of the nth commodity on the rth day; It represents the average purchase coefficient of all consumers who purchase the nth product on the rth day; represents the total number of consumers on day r; Indicates the number of Number of visitors to a product; represents the sum of sales of all commodities on day r; Indicates the number of Sales volume of products; The specific method for obtaining the number of visitors is as follows: Statistics The number of consumers who have a purchase start time for a product in one day and whose shopping operations from the purchase start time to the purchase end time are odd numbers is recorded as The number of people who viewed the product.
2. A moving target tracking and analysis system based on video surveillance, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of a moving target tracking and analysis method based on video surveillance as claimed in claim 1 are implemented.
