A method and system for generating a comprehensive heat map of a mall

Abstract

A method and system for generating a comprehensive heat map of a shopping mall. In the method, a consumer index of each region of the shopping mall is determined by factors such as bag rate, emotional happiness degree, and the like, so as to generate a comprehensive heat map of the shopping mall. The technical solution provided in the application can improve the accuracy of data analysis on the heat points of the shopping mall, so as to assist the shopping mall managers to guide the customers to the comfortable areas, and help the ordinary merchants to attract more customers.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a method and system for generating a comprehensive heat map of a shopping mall. BACKGROUND

[0002] Modern large shopping malls gather a large number of stores and consumers. Consumers want to buy their favorite goods and enjoy higher quality services, while store owners want more consumers to visit their stores to generate more consumption and create profits for themselves. In addition, as the manager of the shopping mall, the optimal choice is to balance between consumers and merchants, making both customers and merchants satisfied, which maximizes their own interests. However, how to achieve a win-win situation cannot be separated from scientific data analysis.

[0003] In related technologies, in large shopping malls, many electronic devices are used to assist in user tracking, consumer flow diversion, store management, parking guidance, and other data analysis of shopping malls. For example, a hot spot flow system divides the site into multiple areas and estimates the number of people in the area at a certain time through a camera to measure the consumer potential in the area.

[0004] However, these shopping mall data analysis and statistics methods are not scientific and cannot provide enough accurate and valuable information for shopping mall managers to make management decisions. For example, if a certain area of the shopping mall is congested, based on the number density information obtained, the management personnel cannot determine whether it is a negative energy hot zone caused by channel congestion, toilet queuing, or a positive energy hot zone caused by merchant activities attracting customers. The data obtained by such analysis cannot assist in decision-making, and a win-win situation for shopping mall managers, consumers, and merchants cannot be achieved. SUMMARY

[0005] The present application provides a method and system for generating a comprehensive heat map of a shopping mall to improve the accuracy of data analysis of the hot spots of the shopping mall, so as to assist the shopping mall manager to guide the customers to the comfortable area and help the ordinary merchants to attract more customers.

[0006] In a first aspect, the application provides a method for generating a comprehensive heat map of a shopping mall, comprising: at a first time, obtaining video pictures captured by video cameras in each region of the shopping mall and region pixel temperatures collected by thermal imaging cameras; determining, based on the video pictures of each region, a mood happiness degree, a number of people crowded degree and a bag rate of each region; determining, based on the region pixel temperatures of each region, a region heat value and a heat crowded degree of each region; determining, according to the mood happiness degree, the number of people crowded degree, the bag rate, the region heat value and the heat crowded degree of each region, a consumption index of each region at the first time; displaying, in a region distribution map of the shopping mall displayed on a display screen, the consumption index of each region at the first time as a comprehensive heat map of the shopping mall at the first time by brightness of color, and the higher the consumption index of a region, the brighter the color in the region; wherein the mood happiness degree is used to represent the happiness degree of the mood of the people in the region; the number of people crowded degree is used to represent the crowded condition of the people in the region; the bag rate is used to represent the number of goods purchased by the people in the region; the region heat value is used to represent the average pixel temperature of the people in the region; and the heat crowded degree is used to represent the average pixel temperature in the region.

[0007] In the above embodiment, the mood happiness degree, the number of people crowded degree, the bag rate, the region heat value and the heat crowded degree of each region are obtained by analyzing the video pictures and the region pixel temperatures, and are comprehensively analyzed and determined as the consumption index, which is displayed in the region distribution map of the shopping mall as the comprehensive heat map of the shopping mall by brightness of color, thereby greatly improving the accuracy of the data analysis of the hot spot of the people flow in the shopping mall, and improving the efficiency of the accurate information obtained by the shopping mall manager, so as to assist the shopping mall manager to guide the customers to the comfortable region and help the ordinary merchants to attract more customers.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the mood happiness degree, the number of people crowded degree and the bag rate of each region are determined based on the video pictures of each region, specifically comprising: obtaining, based on the video pictures of each region and a human body detection model, the bag rate and the number of people crowded degree of each region; and obtaining, based on the video pictures of each region and a facial emotion model, the mood happiness degree of each region.

[0009] In the above embodiment, the bag rate, the number of people crowded degree and the mood happiness degree are obtained by the human body detection model and the facial emotion model, thereby improving the accuracy of the data analysis.

[0010] In some embodiments of the first aspect, in some embodiments, the method further comprises: using a plurality of sets of labeled video pictures belonging to different temperature ranges collected in the shopping mall as human model training data to train a plurality of sets of human detection models adapted to different temperature ranges; the human model training data uses video pictures collected in the shopping mall and annotated with human bodies as positive samples, and video pictures collected in the shopping mall and not containing human bodies as negative samples; using annotated data extracted from video pictures in the shopping mall as face emotion training data to train the face emotion model; the face emotion training data uses human pictures collected in the shopping mall and annotated with face positions and face emotions as positive samples, and video pictures collected in the shopping mall and not containing faces as negative samples.

[0011] In the above embodiments, the human detection model is trained in different temperature ranges for use, further improving the accuracy and reliability of the artificial intelligence model in identifying human bodies.

[0012] In some embodiments of the first aspect, in some embodiments, the video pictures and human detection model based on each region are used to obtain the bag rate and the number of crowdedness of each region, specifically including: using the human detection model to detect human pictures and the number of people from the video pictures of each region; using the shopping bag recognition model on the human pictures to identify the number of shopping bags of each region in the current season; based on the number of people and the number of shopping bags of each region in the current season, calculating the bag rate of each region, the bag rate being in a positive proportional relationship with the number of shopping bags of the region in the current season and in an inverse proportional relationship with the number of people; based on the area of each region and the number of people, calculating the number of crowdedness of each region, the number of crowdedness being in a positive proportional relationship with the number of people and in an inverse proportional relationship with the area.

[0013] In the above embodiments, the bag rate of consumers is calculated by using the shopping bag recognition model to identify the shopping bags of each region in the current season, without using a unified cash register system, so that the performance of each merchant can be effectively evaluated.

[0014] In some embodiments of the first aspect, in some embodiments, the method further comprises: using labeled pictures including shopping bags of different shopping seasons of merchants in each region of the shopping mall as shopping bag training data to train a plurality of sets of shopping bag recognition models adapted to different shopping seasons; when training the shopping bag recognition model of the current season, the shopping bag training data uses shopping bags of merchants in each region of the shopping mall in the current season as positive sample data, and shopping bags that have been withdrawn from the market or shopping bags of other shopping malls as negative sample data.

[0015] In the above embodiment, the shopping bag recognition model for the merchants in each region in the season can be trained separately, and the accuracy of the bag taking rate analysis is further improved.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the emotion happiness degree of each region is obtained based on the video pictures of each region and the face emotion model, and specifically includes: determining the number and category of emotion faces in each region from the human body pictures detected by the human body detection model using the face emotion model; the category includes happy, calm, and uncomfortable; and calculating the emotion happiness degree of each region based on the number and category of emotion faces, the emotion happiness degree being inversely proportional to the number of emotion faces and being proportional to the number of emotion faces in the happy category and the number of emotion faces in the uncomfortable category.

[0017] In the above embodiment, the data of emotion faces of different types in each region can be accurately recognized, and the emotion happiness degree of the region is determined based on the difference between the number of happy emotion faces and the number of uncomfortable emotion faces, so that the emotion analysis of consumers in the region is more accurate.

[0018] In combination with some embodiments of the first aspect, in some embodiments, the region thermal value and the thermal congestion degree of each region are determined based on the region pixel temperature of each region, and specifically includes: determining the region thermal value of each region based on the region pixel temperature of each region, the region thermal value being related to the number of human body pixel points in each region and the sum of the temperatures of the human body pixel points; and calculating the thermal congestion degree based on the total number of region pixel points obtained by the thermal imaging camera and the sum of the temperatures of the human body pixel points.

[0019] In the above embodiment, the pixel points in the temperature range close to the human body are determined as effective pixel points, and the region thermal value and the thermal congestion degree related thereto are determined, so that the shopping environment and the congestion degree of each region can be more accurately represented.

[0020] In a second aspect, the application provides a comprehensive heat map generation system for a shopping mall, comprising: a data acquisition module configured to acquire, at a first time, video pictures captured by video cameras in each region of the shopping mall and region pixel temperatures collected by thermal imaging cameras; a first data determination module configured to determine, based on the video pictures of each region, a mood happiness degree, a number of people crowded degree and a bag carrying rate of each region; a second data determination module configured to determine, based on the region pixel temperatures of each region, a region heat value and a heat crowded degree of each region; a consumption index determination module configured to determine, according to the mood happiness degree, the number of people crowded degree, the bag carrying rate, the region heat value and the heat crowded degree of each region, a consumption index of each region at the first time; and a heat map display module configured to display, in a region distribution map of the shopping mall displayed on a display screen, the consumption index of each region at the first time as a comprehensive heat map of the shopping mall at the first time, with a higher consumption index of a region being displayed as a brighter color in the region.

[0021] In a third aspect, the application provides a comprehensive heat map generation system for a shopping mall, comprising a plurality of video cameras and a plurality of thermal imaging cameras distributed in each region, a display screen and a data processing center, wherein the data processing center comprises one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the comprehensive heat map generation system for the shopping mall to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fourth aspect, the application provides a computer program product comprising instructions, which, when executed on a data processing center in a comprehensive heat map generation system for a shopping mall, enable the comprehensive heat map generation system for the shopping mall to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] In a fifth aspect, the application provides a computer-readable storage medium comprising instructions, which, when executed on a data processing center in a comprehensive heat map generation system for a shopping mall, enable the comprehensive heat map generation system for the shopping mall to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0024] It can be understood that the mall comprehensive heat map generation system provided by the second aspect and the third aspect, the computer program product provided by the fourth aspect, and the computer storage medium provided by the fifth aspect are all used to execute the method provided by the first aspect and any possible implementation manner in the first aspect. Therefore, the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is an exemplary people flow heat map in the related art;

[0026] Figure 2 is an exemplary mall comprehensive heat map generated in the embodiments of the present application;

[0027] Figure 3 is an exemplary mall comprehensive heat map generation system in the embodiments of the present application;

[0028] Figure 4 is a flowchart of a mall comprehensive heat map generation method in the embodiments of the present application;

[0029] Figure 5 is an exemplary data flow diagram for determining analysis data using an AI model in the embodiments of the present application;

[0030] Figure 6A is an exemplary diagram for training and using a human body detection model in the embodiments of the present application;

[0031] Figure 6B is an exemplary diagram for training and using a human face emotion model in the embodiments of the present application;

[0032] Figure 6C is an exemplary diagram for training and using a shopping bag recognition model in the embodiments of the present application;

[0033] Figure 7 is another flowchart of a mall comprehensive heat map generation method in the embodiments of the present application;

[0034] Figure 8 is an exemplary diagram of a region pixel temperature collected by a thermal imaging camera in the embodiments of the present application;

[0035] Figure 9 is an exemplary scene diagram for displaying a mall comprehensive heat map in the embodiments of the present application;

[0036] Figure 10 is another exemplary scene diagram for displaying a mall comprehensive heat map in the embodiments of the present application;

[0037] Figure 11This is a schematic diagram of a hardware module of the shopping mall comprehensive heat map generation system 1100 in this application embodiment. Detailed Implementation

[0038] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0039] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0040] like Figure 1 The diagram shown is an exemplary heatmap of pedestrian flow obtained after analyzing pedestrian flow data in a shopping mall using relevant technologies. The shopping mall includes four areas: Area 1, Area 2, Area 3, and Area 4. The current number of pedestrians in each area is marked with the analyzed data. For example, the data analysis results show: 200 people in Area 1, 500 people in Area 2, 300 people in Area 3, and 100 people in Area 4.

[0041] However, data analysis that only provides pedestrian traffic in each area is insufficient to support the comprehensive management needs of a shopping mall. For example, if a certain area has a high pedestrian traffic volume, it's impossible to determine whether it's due to a popular event attracting a large number of customers, or whether the crowd is concentrated due to damage to a public facility or a disturbance. Therefore, the pedestrian traffic heat map obtained through such data analysis cannot assist in analyzing and adjusting the distribution of the actual shopping crowd in the mall.

[0042] Furthermore, due to the development of electronic payments, individual shops in shopping malls have established their own POS systems, eliminating the need for the mall's own POS system. Therefore, the mall cannot accurately obtain individual merchants' sales figures through the POS system, making it impossible to effectively evaluate each merchant's performance.

[0043] The comprehensive heat map of the shopping mall generated by the data analysis method in the embodiment of the present application can not only improve the accuracy of the data analysis of the heat spot of the people flow in the shopping mall, so as to help the shopping mall manager to guide the customers to the comfortable area, and help the ordinary business to attract more customers. Moreover, the performance of each business can be effectively evaluated without using the cash register system of the shopping mall, so as to assist the shopping mall to adjust and assist the business.

[0044] As shown in Figure 2 , it is an exemplary schematic diagram of the comprehensive heat map of the shopping mall generated in the embodiment of the present application. In the process of generating the comprehensive heat map of the shopping mall shown in Figure 2 , not only the people flow data of each area is analyzed, but also the mood and shopping condition of each person in each area are intelligently analyzed, the consumption index of each area is comprehensively analyzed, and finally the consumption index of each area is intuitively displayed in the area distribution diagram by the brightness of the color.

[0045] As shown in Figure 2 , the area 4 has the least number of people, but the mood of most of the people is very happy, and the bag rate is very high (that is, most of the people have purchased the goods in this area), so the color of this area is the brightest, indicating that the consumption index of this area is the highest.

[0046] As shown in Figure 2 , the area 2 has the most number of people, but the bag rate is not particularly high, and the mood of some people is relatively calm or even angry. Therefore, the color of this area is not as bright as that of the area 4, indicating that the consumption index of this area is lower than that of the area 4.

[0047] As shown in Figure 2 , the area 3 has the second largest number of people, but the bag rate is too low, and the mood of the people is generally not high, so the color of this area is the darkest, indicating that the consumption index of this area is the lowest.

[0048] Since the comprehensive heat map of the shopping mall in the embodiment of the present application uses the consumption index which can better reflect the actual consumption capacity and consumption experience of each area as the display index, and the consumption index not only considers the people flow factor, but also comprehensively analyzes the mood, bag rate and other factors of the people in each area. Therefore, the accuracy of the data analysis of the heat spot of the people flow in the shopping mall is greatly improved. In addition, since the bag rate is considered as a factor, even the shopping bags of each business can be identified, so the performance of each business can be effectively evaluated even if the shopping mall does not use a unified cash register system.

[0049] It can be understood that, for the convenience of description, the embodiment of the present application is described by taking four areas in the shopping mall as an example. In the actual scene, there can be more or fewer areas in the shopping mall, which is not limited here.

[0050] As Figure 3 shown, it is an exemplary schematic diagram of a mall comprehensive heat map generation system in the embodiments of the present application. According to the mall map, video cameras 301 and thermal imaging cameras 302 can be respectively arranged at each elevator, stairwell, segmented area, washroom waiting area, etc. in each area.

[0051] The data obtained by the video cameras 301 and the thermal imaging cameras 302 arranged in each area can be transmitted to a data processing center 303 of the mall, and the data processing center 303 can perform human body detection, facial emotion detection, shopping bag detection, etc. on the data, and finally comprehensively analyze and generate a mall comprehensive heat map, which is displayed on a display screen 304.

[0052] The mall comprehensive heat map generation system can be based on a mall vitality model of deep learning, combined with an evaluation model of multiple elements such as emotion, bag rate, crowd density, time, temperature, etc., and use the existing video equipment in the mall to analyze and judge through video data, and real-time display the data to the manager. The manager can dynamically apply means such as flow guiding and environment control according to the data to guide customers to the comfortable area. At the same time, the managers of ordinary shops can also obtain such information in real time, and purchase the flow guiding service or activities of the mall through the value-added service to attract more customers.

[0053] The following describes a mall comprehensive heat map generation method provided in the embodiments of the present application in combination with the mall comprehensive heat map shown in Figure 2 and the exemplary mall comprehensive heat map generation system shown in Figure 3

[0054] Please refer to Figure 4 , which is a flowchart of the mall comprehensive heat map generation method in the embodiments of the present application.

[0055] S401、At a first time, video pictures captured by video cameras in each area of the mall and area pixel temperatures collected by thermal imaging cameras are obtained.

[0056] The mall comprehensive heat map generation system can obtain video pictures captured by video cameras 301 installed in each area of the mall and area pixel temperatures collected by thermal imaging cameras 302 in real time.

[0057] It should be noted that the number of pixel points in the area that can be collected by each thermal imaging camera 302 can be different based on the size of the sensor in the thermal imaging camera 302.

[0058] ​For example, if the sensor in a thermal imaging camera 302 is W pixels long and H pixels wide, the number of pixels in the area it can capture can be W*H. The pixel temperature of the area captured by a thermal imaging camera 302 represents the temperature of the W*H pixels in the area captured by the thermal imaging camera 302. For example, some pixels have a temperature of 10°C, some pixels have a temperature of 38°C, and so on, which is not limited here.

[0059] S402, based on the video pictures of each area, determine the emotional happiness degree, the number of people crowdedness Ynt1 and the bag rate of each area;

[0060] After the mall comprehensive heat map generation system obtains the video pictures of each area, it can determine the emotional happiness degree, the number of people crowdedness Ynt1 and the bag rate of each area. The emotional happiness degree is used to represent the degree of happiness of the emotions of the personnel in the area; the number of people crowdedness Ynt1 is used to represent the crowdedness of the personnel in the area; and the bag rate is used to represent the number of personnel purchasing the goods in the area.

[0061] In actual application, the specific implementation of determining the emotional happiness degree, the number of people crowdedness Ynt1 and the bag rate of each area can be realized by a preset recognition algorithm, or an AI model can be used to realize it, or other video picture analysis methods can be used to realize it, which is not limited here.

[0062] It can be understood that in an area, the higher the emotional happiness degree, the more comfortable the consumers are in the area; the higher the number of people crowdedness Ynt1, the more people and the more lively the area is; and the higher the bag rate, the better the sales performance of the merchant in the area.

[0063] S403, based on the area pixel temperature of each area, determine the area heat value and heat crowdedness Ynt2 of each area;

[0064] After the mall comprehensive heat map generation system obtains the area pixel temperature of each area, it can determine the area heat value and heat crowdedness Ynt2 of each area. The area heat value is used to represent the average body pixel temperature in the area; and the heat crowdedness Ynt2 is used to represent the average pixel temperature in the area.

[0065] In actual application, the area heat value and heat crowdedness Ynt2 of each area can be directly realized by a preset formula, or an AI model can be used to realize it, or other pixel temperature analysis methods can be used to realize it, which is not limited here.

[0066] It can be understood that, as the average temperature of people is also high when they are excited and happy or when they are in a crowded and difficult situation, in a region, if the regional thermal value is higher, it indicates that the emotional fluctuation of the consumers in the region is greater, and the proportion of people with high temperature is greater; if the thermal congestion degree Ynt2 is higher, it indicates that the emotional fluctuation of the consumers in the region is greater, and the temperature is higher.

[0067] In some embodiments, in the process of calculating and determining the emotional happiness degree, the number of people congestion degree Ynt1, the bag taking rate, the regional thermal value, and the thermal congestion degree Ynt2, different weighting coefficients can also be multiplied for different parameters based on the actual management needs of the mall to represent different degrees of attention.

[0068] S404, according to the emotional happiness degree, the number of people congestion degree Ynt1, the bag taking rate, the regional thermal value, and the thermal congestion degree Ynt2 of each region, determine the consumer index of each region at the first time;

[0069] The higher the consumer index is, the higher the comfort degree, the satisfaction degree, and the consumption ability of the consumers in the region are.

[0070] The consumer index is in a positive proportional relationship with the emotional happiness degree and the bag taking rate, and the higher the emotional happiness degree or the bag taking rate is, the higher the consumer index is. Based on different actual needs, the influence of the emotional happiness degree and the bag taking rate on the consumer index can also be different, which is not limited here.

[0071] S405, in the mall region distribution map displayed on the display screen, the consumer index of each region at the first time is displayed by the brightness of the color, as the mall comprehensive thermal map at the first time.

[0072] After the mall comprehensive thermal map generation system determines the consumer index of each region at the first time, the mall comprehensive thermal map at the first time can be displayed. The mall comprehensive thermal map at the first time is that the consumer index of each region at the first time is displayed by the brightness of the color in the mall region distribution map displayed on the display screen. Among them, the higher the consumer index is, the brighter the color is.

[0073] The generation process of the mall comprehensive thermal map is described in the above embodiment taking the first time as an example. It can be understood that, the video camera 301 and the thermal imaging camera 302 are both collecting video pictures and regional pixel temperatures in real time, so in some embodiments, the mall comprehensive thermal map generation system can also obtain the mall comprehensive thermal map at the current time in real time. In some other embodiments, in order to save computing resources, it can also be set to only need to analyze and obtain the mall comprehensive thermal map once in a preset period (for example, every hour); in some other embodiments, the mall comprehensive thermal map can also be obtained at other set time intervals, which is not limited here.

[0074] In the embodiments of the present application, the mood happiness degree, the number of crowded degree Ynt1, the bag carrying rate, the regional heat value and the heat crowded degree Ynt2 of each region obtained by analyzing the pixel temperature of the video picture and the region, and the comprehensive analysis and determination of the consumption index are displayed as the comprehensive heat map of the shopping mall in the brightness distribution of the color in the shopping mall region map. Not only the accuracy of the data analysis of the hot spot of the shopping mall is greatly improved, but also the efficiency of the accurate information obtained by the shopping mall manager is improved, so that the shopping mall manager can guide the customers to the comfortable area, and help the ordinary merchants to attract more customers.

[0075] In the above embodiments, the shopping mall comprehensive heat map generation system can determine the mood happiness degree, the number of crowded degree Ynt1 and the bag carrying rate of each region based on the video picture of each region. It can be understood that in the process of determining these parameters, the number of human bodies in each region, the number of faces belonging to each emotion classification and the number of bags carried are determined as the basic analysis data. In order to improve the accuracy of the determined basic analysis data, multiple artificial intelligence (AI) models based on deep learning can be used to analyze the data.

[0076] Please refer to Figure 5 An exemplary data flow diagram for determining these analysis data using AI models.

[0077] In the embodiments of the present application, in order to facilitate understanding, three AI models are included in the data processing. It can be understood that in actual application, multiple AI models can be split or combined into one or more without affecting their function, which is not limited here.

[0078] As Figure 5 shown, the video pictures captured by multiple video cameras 301 are input into a human body detection model 501. After analysis and processing by the human body detection model 501, the human body pictures and the number of human bodies detected from the video pictures can be output.

[0079] The detected human body pictures are then input into a face emotion model 502. After analysis and processing by the face emotion model 502, the recognized emotion faces and the number of emotion faces belonging to each emotion classification can be output. For example, the emotion classification can be set as happy, calm and uncomfortable. Through the face emotion model 502, the number of recognized faces belonging to the three categories can be determined.

[0080] In addition, the detected human body pictures can be input into a shopping bag recognition model 503. After analysis and processing by the face emotion model 502, it can be determined whether the human body has a shopping bag, and the shopping bag picture and the number of bags carried can be output.

[0081] The video footage captured by the video cameras 301 in each area is analyzed by three types of AI models: human detection model 501, facial emotion model 502, and shopping bag recognition model 503. This analysis yields basic data to determine the emotional happiness level, crowding level Ynt1, and bag carrying rate of each area: the number of people in each area, the number of faces belonging to each emotion category, and the number of bags.

[0082] To improve the accuracy of various AI models in analyzing shopping mall video footage, in this embodiment, each AI model can be trained using the following methods:

[0083] like Figure 6A The diagram shown is an exemplary schematic of training and using the human detection model 501 in this embodiment of the application. To improve the accuracy of the trained human detection model 501, clothing factors need to be considered. Therefore, according to temperature differences, the model can be divided into five regions: below zero, 0℃~10℃, 10℃~20℃, 20℃~30℃, and above 30℃, to train five different types of human recognition models.

[0084] When training the human detection model 501, actual video footage captured by multiple cameras in the mall at a preset outdoor temperature can be used as positive and negative samples of training data to train each human recognition model separately, so that the trained model can better meet the needs of the mall.

[0085] For example, for "Human Detection Model 1 - Temperature Below Zero", video footage of people labeled with human bodies collected in the shopping mall when the outdoor temperature is below 0℃ and the number of people can be used as positive samples, and video footage of people not containing human bodies collected in the shopping mall when the outdoor temperature is below 0℃ can be used as negative samples to train "Human Detection Model 1 - Temperature Below Zero".

[0086] For example, for "Human Detection Model 2 - Temperature 0℃~10℃", video footage of people labeled with human bodies collected in the mall when the outdoor temperature is 0℃~10℃ and the number of people can be used as positive samples, and video footage of people not containing human bodies collected in the mall when the outdoor temperature is 0℃~10℃ can be used as negative samples to train "Human Detection Model 2 - Temperature 0℃~10℃".

[0087] For example, for the "Human Detection Model 3 - Temperature 10℃~20℃", video footage of people labeled with human bodies and the number of people collected in the mall when the outdoor temperature is 10℃~20℃ can be used as positive samples, and video footage of people not included in human bodies collected in the mall when the outdoor temperature is 10℃~20℃ can be used as negative samples to train the "Human Detection Model 3 - Temperature 10℃~20℃".

[0088] For example, for the "human body detection model 4-temperature 20-30°C", the video pictures and the number of human bodies collected in the mall at an outdoor temperature of 20-30°C and labeled with human bodies can be used as positive samples, and the video pictures collected in the mall at an outdoor temperature of 20-30°C and not containing human bodies can be used as negative samples to train the "human body detection model 4-temperature 20-30°C".

[0089] For example, for the "human body detection model 5-temperature above 30°C", the video pictures and the number of human bodies collected in the mall at an outdoor temperature above 30°C and labeled with human bodies can be used as positive samples, and the video pictures collected in the mall at an outdoor temperature above 30°C and not containing human bodies can be used as negative samples to train the "human body detection model 5-temperature above 30°C".

[0090] After the training is completed, when the human body detection model 501 is used, for example, the "human body detection model 5-temperature above 30°C" trained is used: after the video pictures of the area 1 in the mall taken by the multiple cameras at an outdoor temperature above 30°C are input into the "human body detection model 5-temperature above 30°C", the human body pictures extracted from the area 1 in the mall and the number of human bodies in the area 1 in the mall can be output after analysis.

[0091] It can be understood that the above temperature division is only an example, and in actual application, based on the local daily temperature range and the dressing habit, another temperature interval division manner can be used to train different human body recognition models, which is not limited here.

[0092] As shown in FIG. 5, it is an exemplary schematic diagram for training and using the face emotion model 502 in the embodiment of the present application. Figure 6B

[0093] When the face emotion model 502 is trained, the human body pictures with labeled faces extracted from the video pictures of the mall, the face emotion classification

[0094] The human body pictures collected in the mall and labeled with face positions and face emotions are used as positive samples, and the video pictures collected in the mall and not containing faces are used as negative samples to train the face emotion model.

[0095] When the face emotion model trained is used, after the extracted human body pictures are input into the face emotion model, the face emotion model can first extract the face pictures therein, and then perform emotion recognition on the extracted face pictures (for example, the face emotion model can output the face pictures and the emotions of the face pictures). Figure 6B ​As shown in the middle, the skeleton point recognition mode can be adopted, or other recognition modes can be adopted, which are not limited here), and finally the recognized facial emotions are classified, such as happy, calm, or uncomfortable, etc. The number of faces belonging to different emotion categories can be counted.

[0096] It can be understood that the above emotion classification is only an example. In actual application, based on the needs of subsequent data analysis, more emotion classifications can be used to train the facial emotion model, which is not limited here.

[0097] As shown in the middle, the skeleton point recognition mode can be adopted, or other recognition modes can be adopted, which are not limited here), and finally the recognized facial emotions are classified, such as happy, calm, or uncomfortable, etc. The number of faces belonging to different emotion categories can be counted. Figure 6C As shown in the middle, the skeleton point recognition mode can be adopted, or other recognition modes can be adopted, which are not limited here), and finally the recognized facial emotions are classified, such as happy, calm, or uncomfortable, etc. The number of faces belonging to different emotion categories can be counted.

[0098] To improve the accuracy of the trained shopping bag recognition model 503, the time factor needs to be considered. Only the shopping bags of the current season of the merchant are positive samples, and other seasonal shopping bags that have been withdrawn from the market and some common shopping bags on the market are negative sample data. The trained model can be used with the current season.

[0099] For example, four different shopping bag recognition models for Spring Festival, summer, autumn, and winter can be trained in different seasons. It can be understood that the seasonal division of the shopping bag recognition model here is only an example for easy understanding and description. In actual application, many different time zones can be used as the current season to train a unique shopping bag recognition model, such as Christmas season, Spring Festival season, etc. which are not limited here.

[0100] For example, for "shopping bag recognition model 1-spring", the human body pictures and the number of shopping bags labeled as the spring shopping bags of the merchants in this region extracted from the video pictures of this shopping mall in spring can be used as positive samples, and the video pictures of other seasonal shopping bags that have been withdrawn from the market and the video pictures of common shopping bags on the market can be used as negative samples to train the "shopping bag recognition model 1-spring".

[0101] For example, for "shopping bag recognition model 2-summer", the human body pictures and the number of shopping bags labeled as the summer shopping bags of the merchants in this region extracted from the video pictures of this shopping mall in summer can be used as positive samples, and the video pictures of other seasonal shopping bags that have been withdrawn from the market and the video pictures of common shopping bags on the market can be used as negative samples to train the "shopping bag recognition model 2-summer".

[0102] For example, for the "shopping bag recognition model 3-autumn", the human body pictures and the number of shopping bags that are labeled as belonging to the autumn shopping bags of the local area merchants and are extracted from the video pictures of the video camera in the autumn can be used as positive samples, and the video pictures that are labeled as the shopping bags that have been withdrawn from the market in other seasons and the video pictures that are labeled as the common shopping bags on the market can be used as negative samples to train the "shopping bag recognition model 3-autumn".

[0103] For example, for the "shopping bag recognition model 4-winter", the human body pictures and the number of shopping bags that are labeled as belonging to the winter shopping bags of the local area merchants and are extracted from the video pictures of the video camera in the winter can be used as positive samples, and the video pictures that are labeled as the shopping bags that have been withdrawn from the market in other seasons and the video pictures that are labeled as the common shopping bags on the market can be used as negative samples to train the "shopping bag recognition model 4-winter".

[0104] After the training is completed, when the human body detection model 501 is used, for example, the "shopping bag recognition model 4-winter" that is trained is used: after the extracted human body pictures are input into the "shopping bag recognition model 4-winter", the analysis can be performed to output the pictures of the consumers carrying the winter shopping bags of the local area merchants and the number of the shopping bags.

[0105] In the above Figure 6A - Figure 6C In the exemplary AI training mode shown in the above

[0106] The generation method of the comprehensive heat map of the shopping mall in the embodiments of the present application will be described in detail below in combination with the above embodiments:

[0107] As Figure 7 shown, another flowchart of the generation method of the comprehensive heat map of the shopping mall in the embodiments of the present application is shown.

[0108] S701, continuously acquiring video pictures of each region of the shopping mall captured by a video camera;

[0109] The video cameras 301 distributed at various positions in each region of the shopping mall can capture the video pictures of each region of the shopping mall, and the comprehensive heat map generation system of the shopping mall can continuously acquire the captured video pictures of each region of the shopping mall.

[0110] S702, continuously acquiring the pixel temperature of each region collected by a thermal imaging camera;

[0111] Similarly, the thermal imaging camera 302 distributed in each position of each area of the mall can collect the area pixel temperature of each area of the mall, and the mall comprehensive thermal map generation system can continuously obtain the collected area pixel temperature of each area of the mall.

[0112] It can be understood that the mall comprehensive thermal map generation system can analyze and obtain the mall comprehensive thermal map at the current time in real time, but in order to save computing resources and based on actual needs, it can also be set to only analyze and obtain the mall comprehensive thermal map once in a preset period. In the embodiment of the application, the mall comprehensive thermal map generation system analyzes and obtains the mall comprehensive thermal map once at the first time of each hour of each working day is described as an example:

[0113] For the convenience of recording, the mall comprehensive thermal map generation system can number each day of each week as D1, D2, D3,..., D7 in a weekly cycle, and further number each business hour of each working day as D1a, D1b, D1c,..., etc. according to hours, so as to distinguish the time of each hour and facilitate identification of the mall comprehensive thermal map generated in the hour.

[0114] If only one mall comprehensive thermal map needs to be generated per hour, the first time of each hour can be a preset fixed time in each hour, or a random time in each hour, or a time determined based on other preset rules, which is not limited here.

[0115] For the first time of each hour of each working day, the video picture and the area pixel temperature of each area obtained can be processed according to the following steps S703-S713:

[0116] Among them, steps S703-S706 are the process of analyzing the bag carrying rate and the number of crowded people based on the video picture and the human body detection model 501:

[0117] S703, using the human body detection model, detecting the human body picture and the number of people from the video picture of the area;

[0118] The human body detection model 501 trained by the human body detection model training method as shown in FIG. Figure 6A The human body detection model 501 trained by the human body detection model training method as shown in FIG.

[0119] S704, using the shopping bag recognition model on the human body picture to identify the number of shopping bags of the season of the merchants in the area;

[0120] For the human body picture detected by the human body detection model 501, the shopping mall comprehensive heat map generation system can analyze it by using the shopping bag recognition model 503 trained in the manner as shown in Figure 6C The shopping bag recognition model 503 can identify the number of the seasonal shopping bags of the consumers in the region 1 carrying the seasonal shopping bags of the merchants in the region 1.

[0121] For example, assuming that the region 1 includes three merchants A, B and C, and the shopping bag recognition model 503 identifies that the number of the seasonal shopping bags of the merchants ABC carried by the consumers in the region 1 is 30 in total at the first time, the shopping mall comprehensive heat map generation system can determine that the number of the seasonal shopping bags of the merchants in the region 1 is Bt1=30.

[0122] S705, based on the number of people in the region and the number of the seasonal shopping bags of the merchants in the region, calculating the bag carrying rate of the region;

[0123] The shopping mall comprehensive heat map generation system can calculate the bag carrying rate Ttn of the region based on the number of people in the region and the number of the seasonal shopping bags of the merchants in the region.

[0124] For example, for the region 1, if the number of people Ht1 is 200 and the number of the seasonal shopping bags of the merchants Bt1=30, the bag carrying rate Tt1 of the region 1 can be Bt1 / Ht1*td=15%*td, where td is the influence deviation of the preset bag carrying rate coefficient on the total coefficient, used to control the influence degree of the bag carrying rate on the total consumption index. If td is 0.4, the bag carrying rate Tt1 of the region 1 is 0.15*0.4=0.06. The greater the value of the bag carrying rate Ttn, the better the sales performance of the merchants in the region 1.

[0125] It should be noted that the calculation formula in the example is only an example for representing the relationship between the number of people Htn in the region, the number of the seasonal shopping bags Btn in the region and the bag carrying rate Ttn of the region. In actual use, the bag carrying rate Ttn of the region can be determined by using a more complex algorithm based on the number of people Htn in the region and the number of the seasonal shopping bags Btn in the region, so as to meet the accuracy of the bag carrying rate Ttn under specific circumstances. Regardless of what kind of algorithm is used, only the Ttn and Btn are in direct proportion and the Htn is in inverse proportion, which is not limited here.

[0126] S706, based on the number of people in the region and the area of the region, calculating the number of people crowdedness of the region;

[0127] The area Sn of each region can be preset in the shopping mall comprehensive heat map generation system, and the number of people crowdedness Ynt1 of each region can be calculated based on the number of people Htn in the region and the area Sn of the region.

[0128] For example, for the region 1, if the number of people Ht1 is 200 and the area S1 is 1000 square meters, the number of people crowdedness Yt1 of the region 1 can be Ht1 / S1*mj, where the crowdedness coefficient mj is a preset influence deviation of the number of people crowdedness Ynt1 on the total coefficient, used to control the influence degree of the number of people crowdedness Ynt1 on the total consumption index. If mj is 0.1, the number of people crowdedness Yt1 of the region 1 is 200 / 1000*0.1=0.02. The greater the number of people crowdedness Ynt1 value, the more crowded the region 1 is.

[0129] It should be noted that the calculation formula in this example is only an example for representing the relationship between the number of people crowdedness Ynt1, the number of people Htn in the region and the area Sn of the region. In actual use, the number of people crowdedness Ynt1 of the region can be determined based on the number of people Htn in the region and the area Sn of the region by using a more complex algorithm to meet the accuracy of the number of people crowdedness Ynt1 in a specific situation. Regardless of what algorithm is used, only the positive proportional relationship between Ynt1 and Htn and the inverse proportional relationship between Ynt1 and Sn need to be maintained, which is not limited here.

[0130] In step S707-S708, the process of analyzing the emotional happiness degree based on the video picture and the face emotion model 502 is as follows:

[0131] S707, using the face emotion model, determining the number and category of emotional faces in the region from the detected human body pictures;

[0132] The face emotion model 502 trained by using the face emotion model training method as shown in Figure 6B The face emotion model 502 trained by using the face emotion model training method as shown in

[0133] For example, for the region 1 at the first time, the mall comprehensive heat map generation system can determine that a total of 100 emotional faces Ft1 are collected, Ft1h=30 happy, Ft1p=60 calm, and Ft1a=10 uncomfortable.

[0134] S708, based on the number and category of emotional faces, calculating the emotional happiness degree of the region;

[0135] The mall comprehensive heat map generation system can calculate the emotional happiness degree Qnt of the region based on the determined number and category of emotional faces (Ftnh, Ftnp, Ftna).

[0136] For example, for the first time zone 1, the number of calm people Ftlp is removed, and the emotional happiness degree Qlt of the zone 1 at this moment is (Ftlh-Ftl a) / Ftl *qx, wherein the emotional coefficient qx is a preset emotional coefficient deviation value of the total coefficient, used to control the influence degree of the emotional happiness degree Qnt on the total consumption index. If qx is 0.3, the emotional happiness degree Qlt of the zone 1 is (30-10) / 100*0.3=0.06. The greater the value of the emotional happiness degree Qlt, the more excited the consumers in the zone 1, and the positive and negative deviation relationship is.

[0137] It should be noted that the calculation formula in this example is only an example for representing the relationship among the emotional happiness degree Qnt, the number of emotional faces Ftn, the number of happy emotional faces Ftnh, and the number of uncomfortable emotional faces Ftna. In actual use, a more complex algorithm can be used to determine the emotional happiness degree Qnt of the zone based on the number of emotional faces Ftn, the number of happy emotional faces Ftnh, and the number of uncomfortable emotional faces Ftna, to meet the accuracy of the emotional happiness degree Qnt in a specific situation. Regardless of what algorithm is used, only the inverse proportional relationship between Qnt and Ftn and the positive proportional relationship between the difference between Ftnh and Ftna are required, which is not limited here.

[0138] In step S709-S710, the zone thermal value and the thermal congestion degree are analyzed based on the zone pixel temperature:

[0139] In step S709, the zone thermal value of the zone is determined based on the zone pixel temperature, and the zone thermal value is related to the number of human pixel points in the zone and the sum of the temperatures of the human pixel points.

[0140] For better understanding and description, please refer to Figure 8 is an exemplary schematic diagram of the zone pixel temperature collected by the thermal imaging camera 302 in the embodiment of the present application.

[0141] Taking the zone 1 with two thermal imaging cameras 302, and the number of pixel points that can be collected by each thermal imaging camera 302 being 20*20 as an example, the two thermal imaging cameras 302 can collect the temperatures of 2*20*20=800 pixel points in the zone 1. It can be understood that the number of pixel points that can be collected by the actual thermal imaging camera 302 is much higher than the example, which is not limited here.

[0142] As Figure 8As shown, due to different situations in different positions, the temperature of the collected pixel points will be different. In the figure, the gray depth of the pixel points is used to represent that: some pixel points correspond to the empty land, and the temperature of the collected pixel points is low, so the gray is shallow; some pixel points correspond to the human body, and the temperature is generally between 33~38℃ which is the surface temperature of the human body; some pixel points correspond to the heating equipment, and the temperature is high, so the gray is deep.

[0143] After obtaining the regional pixel temperature of a region, the shopping mall comprehensive thermal map generation system can obtain the thermal value Rnt in the region based on the temperature of the human body region pixel points in the regional pixel temperature.

[0144] For example, the human body region pixel points in the pixel points can be selected as effective pixel points, for example Figure 8 As shown in (a), there are 20 effective pixel points with a temperature within 33~38℃ which is the surface temperature of the human body, and for example Figure 8 As shown in (b), there are 8 effective pixel points with a temperature within 33~38℃ which is the surface temperature of the human body, and the number of effective pixel points Ptn in the region is 20+8=28.

[0145] Based on the specific temperature of these effective pixel points collected by the two thermal imaging cameras 302, the temperature sum Wtn of these effective pixel points can be calculated as 1036℃.

[0146] Then, the thermal value Rnt in the region is Rnt=the temperature sum of all effective pixel points Wtn / the number of effective pixel points Ptn*wd, wherein the temperature coefficient wd is a preset average temperature influence deviation of the total coefficient, which is used to control the influence degree of the thermal value Rnt in the region on the total consumption index. If wd is 0.1, the thermal value Rnt in the region is 740 / 28*0.1=3.7. The larger the thermal value Rnt is, the higher the temperature of the consumers in the region is.

[0147] S710, based on the sum of the total number of regional pixel points and the temperature of the human body pixel points obtained by the thermal imaging camera, calculate the thermal congestion degree;

[0148] For example, as shown in Figure 8 When there is no one, the pixel values of the 800 pixel points collected by the two thermal imaging cameras 302 in the region are all out of the 33~38℃ interval, so the total number of regional pixel points Pwn in the region can be determined as 800.

[0149] The thermal crowding degree Ynt2 in the region is Wtn / Pwn*yj, where yj is a thermal crowding degree coefficient, and is a preset average temperature to total coefficient influence deviation, used to control the influence degree of the thermal crowding degree Ynt2 in the region on the total consumption index. If yj is 0.1, the thermal crowding degree Ynt2 in the region is 1036 / 800*0.1=0.1259. The greater the value of the thermal crowding degree Ynt2, the more crowded the region is.

[0150] In some embodiments, the sum of the bagging rate coefficient td, the density coefficient mj, the emotion coefficient qx, the temperature coefficient wd, and the thermal crowding degree coefficient yj can be 1; in some embodiments, each of the above coefficients can be freely adjusted between 0 and 1; in some embodiments, the above system can only focus on the relative size between each other to control the importance, and not on the absolute value, which is not limited here.

[0151] In some embodiments, in addition to using the bagging rate, the number of people crowding, the emotional happiness degree, the regional thermal value, and the thermal crowding degree to determine the consumption index, the external environment temperature can also be added to the influence parameter:

[0152] S711, determining an external environment temperature coefficient based on the external environment temperature;

[0153] The value of the external environment grading coefficient hj can be -1, 0, and 1, respectively representing that the environment temperature is cold, moderate, and hot. People want to gather in a warm place when it is cold, and gather in a cool place when it is hot. When the value of hj is 1, it means that the external temperature is high and it is more comfortable when it is cold. When the value of hj is -1, it means that the external temperature is low and it is more comfortable when it is hot.

[0154] S712, determining the consumption index of the region at the first time based on the bagging rate, the number of people crowding, the emotional happiness degree, the regional thermal value, the thermal crowding degree, and the external environment grading coefficient;

[0155] For the consumption index at the first time, the bagging rate Tnt and the emotional happiness degree Qtn are positively correlated, and the bagging rate Tnt and the emotional happiness degree Qtn are affected by both the merchant activity and the environment. In the case of an uncomfortable environment, a high bagging rate proves that the activity is effective, and at the same time, the discomfort of the environment will also reduce the purchase willingness of the user. Therefore, the number of people crowding Ynt1, the regional thermal value Rnt, the thermal crowding degree Ynt2, and the external environment grading coefficient hj can be used to represent the influence of the environment on the bagging rate Tnt and the emotional happiness degree Qtn.

[0156] For example, the consumption index Xnt of a region at the first moment can be calculated using the following formula: Xnt = Tnt * (Ynt1 + Ynt2 + Rnt * hj) * qz + Qtn * (Ynt1 + Ynt2 + Rnt * hj) * (1 - qz). The preset bag-taking rate weighting coefficient qz is used to set the degree of influence of the bag-taking rate. For example, in some cases, qz can be 70%, and in others, it can have other values, which are not limited here.

[0157] S713. Display the shopping mall's comprehensive heat map at the first moment of each hour of the current workday in a cycle according to the preset time period.

[0158] After the shopping mall comprehensive heat map generation system obtains multiple shopping mall comprehensive heat maps for the day, it can cyclically display the shopping mall comprehensive heat map at the first moment of each hour of the current workday according to a preset time cycle. This shopping mall comprehensive heat map is a distribution map of shopping mall areas, displaying the consumption index of each area using color brightness; where the higher the consumption index, the brighter the color.

[0159] Specifically, for the consumption index Xnt in each area, its displayed color value can be represented using YUV, where Y is the area's brightness value and UV is its chromaticity value. UV uses fixed, varying values; for example, the UV values ​​for D1a and D1b are set to constants. Y is an integer between 0 and 255. Since Xnt varies within a certain area, mapping it to the corresponding value within the 0-255 range allows calculation of the Y value for the current area at the current moment, which is then displayed on the shopping mall's area distribution map. This allows for a direct reflection of the heat (brightness) of each area at each moment based on its brightness.

[0160] like Figure 9 The diagram shown is an exemplary scenario illustrating a comprehensive heat map of a shopping mall in an embodiment of this application. Assume that 3 hours have passed since the mall opened today (Monday), and the consumption index values ​​Xnt for each area during these 3 hours are shown in Figure 9(a).

[0161] Each hour corresponds to a preset fixed UV value. For example, the preset UV value for D1a corresponds to yellow, the preset UV value for D1b corresponds to orange, and the preset UV value for D1c corresponds to aquamarine.

[0162] like Figure 9 As shown in (b), assuming that the consumption index is controlled within the range of 0 to 100 based on pre-set parameters, a mapping relationship between the consumption index value and the brightness Y value can be established. For example, the Y value can be Xnt / 100*255.

[0163] After mapping the consumption index of each region and time period to the brightness Y value, the comprehensive heat map of the shopping mall for each hour can be displayed in a cyclical manner every 10 minutes:

[0164] like Figure 9 (c) shows the overall heat map of the shopping mall during the D1a period. Areas Q1, Q2, Q3, and Q4 are all displayed in yellow. Since area Q2 has the highest consumption index, its yellow color is the brightest. This displays the overall heat map of the shopping mall during the D1a period. Figure 10 Minutes later, the mall's overall heat map for time period D1b will be displayed again.

[0165] During the D1a period, businesses in the Q4 area experienced poor business and purchased mall promotion and customer acquisition services. For example... Figure 9 Figure (b) shows the overall heat map of the shopping mall during the D1b time period. Areas Q1, Q2, Q3, and Q4 are all displayed in orange. Clearly, after purchasing the shopping mall promotion and traffic-driving service, the consumption index of area Q4 rises to the highest level; therefore, area Q4 has the brightest color brightness during the D1b time period. Area Q1, due to its better location, naturally experiences increased traffic, and its consumption index rises to second place; therefore, area Q1's orange brightness increases to second place during the D1b time period. This displays the overall heat map of the shopping mall during the D1b time period. Figure 10 Minutes later, the mall's overall heat map for the D1c time period will be displayed again.

[0166] During the D1b time period, merchants in region Q3 saw the effects of region Q4 and also purchased mall promotion and traffic generation services. Meanwhile, merchants in region Q4 continued to purchase mall promotion and traffic generation services. For example... Figure 9 Figure (b) shows the overall heat map of the shopping mall during time period D1c. Areas Q1, Q2, Q3, and Q4 are all displayed in a light green hue. After purchasing the shopping mall promotion and customer acquisition service, the consumption index of area Q3 significantly increased, as evidenced by a marked increase in its color intensity compared to the overall heat map of the shopping mall during time period D1b. Meanwhile, area Q4, which continued to purchase the shopping mall promotion and customer acquisition service, maintained the highest consumption index, as evidenced by its continued increase in color intensity and its position as the highest.

[0167] Before the comprehensive heat map of the shopping mall for the D1d time period is generated, you can press... Figure 9 (c) Figure 9 (d) and Figure 9 As shown in (e), the comprehensive heat map of the shopping mall is displayed in a cycle of 10 minutes for each display period, for the time periods D1a, D1b, and D1c.

[0168] In the embodiments of the present application, the past hourly comprehensive heat maps D1a, D1b, D1c, ··· of the day that have been analyzed are displayed in a preset time period, so that the dynamic heat change curve in the commercial circle can be simulated, and for the merchants who have purchased the promotion, the intuitive change of the heat brought by the promotion of the mall can be found in the map, for example, the intuitive change of the heat brought by the value-added services such as dynamic broadcast guide, artificial guide, robot guide, and display screen guide provided by the mall manager.

[0169] In some embodiments, the displayed comprehensive heat map of the mall can also be directly compared with the historical data. For example, the current data is displayed on the right side of the map, and the historical data is displayed on the left side.

[0170] As shown in FIG. 1 (a), it is another exemplary scene schematic diagram for displaying the comprehensive heat map of the mall in the embodiments of the present application. Figure 10

[0171] As shown in FIG. 1 (a), it is another exemplary scene schematic diagram for displaying the comprehensive heat map of the mall in the embodiments of the present application. Figure 9 For example, as shown in FIG. 1 (a), after the consumption index at the D1c moment is obtained, the corresponding chroma and brightness of the consumption index at the D1c moment can be displayed on the right side of each region, and the corresponding chroma and brightness of the consumption index at the previous moments D1a and D1b can be displayed on the left side of each region in a preset period.

[0172] For example, after the consumption index at the D1c moment is obtained, the corresponding chroma and brightness of the consumption index at the D1c moment can be displayed on the right side of each region, and the corresponding chroma and brightness of the consumption index at the previous moments D1a and D1b can be displayed on the left side of each region in a preset period. Figure 10

[0173] For example, after the consumption index at the D1c moment is obtained, the corresponding chroma and brightness of the consumption index at the D1c moment can be displayed on the right side of each region, and the corresponding chroma and brightness of the consumption index at the previous moments D1a and D1b can be displayed on the left side of each region in a preset period. Figure 10

[0174] For example, after the consumption index at the D1c moment is obtained, the corresponding chroma and brightness of the consumption index at the D1c moment can be displayed on the right side of each region, and the corresponding chroma and brightness of the consumption index at the previous moments D1a and D1b can be displayed on the left side of each region in a preset period.

[0175] It can be understood that in actual application, based on different needs, the comprehensive heat map of the mall can also be in more different forms to display the consumption index of each region at each moment of the mall, which is not limited here.

[0176] ​​​The following describes the mall comprehensive heat map generation system in the embodiments of the application in combination with the foregoing embodiments.

[0177] Please refer to Figure 11 for a hardware module schematic diagram of the mall comprehensive heat map generation system 1100 in the embodiments of the application.

[0178] The mall comprehensive heat map generation system 1100 can include:

[0179] The data acquisition module 1101 is configured to acquire, at a first time, video pictures captured by video cameras in each region of a mall and region pixel temperatures collected by thermal imaging cameras.

[0180] The first data determination module 1102 is configured to determine, based on the video pictures of each region, a mood happiness degree, a number of people crowded degree, and a bag carrying rate of each region.

[0181] The second data determination module 1103 is configured to determine, based on the region pixel temperatures of each region, a region heat value and a heat crowded degree of each region.

[0182] The consumption index determination module 1104 is configured to determine, according to the mood happiness degree, the number of people crowded degree, the bag carrying rate, the region heat value, and the heat crowded degree of each region, a consumption index of each region at the first time.

[0183] The heat map display module 1105 is configured to display, in a mall region distribution diagram displayed on a display screen, the consumption index of each region at the first time as a mall comprehensive heat map at the first time in the brightness of colors, and the higher the consumption index of a region, the brighter the color in the region.

[0184] The mood happiness degree is used to represent the happiness degree of the mood of personnel in a region; the number of people crowded degree is used to represent the crowded condition of personnel in a region; the bag carrying rate is used to represent the number of goods purchased by personnel in a region; the region heat value is used to represent the average pixel temperature of personnel in a region; and the heat crowded degree is used to represent the average pixel temperature in a region.

[0185] The foregoing and the foregoing embodiments are only used to illustrate the technical solutions of the application, rather than limit the same; although the foregoing has been described in detail, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and the modification or replacement does not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the application.

[0186] In the above embodiments, the term "when" can be interpreted as meaning "if" or "after" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "upon determining" or "if detecting (the stated condition or event)" can be interpreted as meaning "if determining" or "in response to determining" or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)" depending on the context.

[0187] In the above embodiments, all or part of the methods can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the methods can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk), etc.

[0188] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be instructed by a computer program to relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. A method for generating a comprehensive heat map of a shopping mall, characterized in that, The method comprises the following steps: At a first time, video pictures captured by video cameras in each area of a shopping mall and pixel temperatures of the area collected by a thermal imaging camera are acquired; Based on the video pictures of each area, the emotional happiness degree, the number of people, and the bag rate of each area are determined; Based on the pixel temperatures of each area, the area heat value and the heat congestion degree of each area are determined; According to the emotional happiness degree, the number of people, the bag rate, the area heat value, and the heat congestion degree of each area, the consumption index of each area at the first time is determined; In the area distribution map of the shopping mall displayed on the display screen, the consumption index of each area at the first time is displayed by the brightness of the color as a comprehensive heat map of the shopping mall at the first time, and the higher the consumption index of the area, the brighter the color in the area; The emotional happiness degree is used to represent the happiness degree of the emotions of the people in the area; the number of people is used to represent the congestion of the people in the area; the bag rate is used to represent the number of the people purchasing the goods in the area; the area heat value is used to represent the average pixel temperature of the people in the area; and the heat congestion degree is used to represent the average pixel temperature in the area; The method further comprises the following steps: Based on the video pictures of each area and a human body detection model, the bag rate and the number of people of each area are acquired; Based on the video pictures of each area and a human face emotion model, the emotional happiness degree of each area is acquired; The method further comprises the following steps: Based on the pixel temperatures of each area, the area heat value of each area is determined, and the area heat value is related to the number of human body pixels and the sum of the temperatures of the human body pixels in each area; Based on the total number of pixel points and the sum of the temperatures of the human body pixels acquired by the thermal imaging camera, the heat congestion degree is calculated.

2. The method of claim 1, wherein, The method further comprises the following steps: A plurality of labeled video pictures belonging to different temperature intervals collected in the shopping mall are used as human body model training data to train a plurality of human body detection models adapted to different temperature ranges; in the human body model training data, video pictures collected in the shopping mall and the number of human bodies are used as positive samples, and video pictures collected in the shopping mall without human bodies are used as negative samples; Labeled data extracted from the video pictures in the shopping mall are used as human face emotion training data to train the human face emotion model; in the human face emotion training data, human body pictures collected in the shopping mall and labeled with human face positions and human face emotions are used as positive samples, and video pictures collected in the shopping mall without human faces are used as negative samples.

3. The method of claim 2, wherein, The method further comprises the following steps: The human body detection model is used to detect human body pictures and the number of people from the video pictures of each area; The shopping bag recognition model is used on the human body picture to identify the number of shopping bags of each regional merchant in the current season; Based on the number of people in each region and the number of shopping bags of each regional merchant in the current season, the bag taking rate of each region is calculated, which is in direct proportion to the number of shopping bags of the merchant in the current season and in inverse proportion to the number of people; Based on the area of each region and the number of people, the number of people in each region is calculated, which is in direct proportion to the number of people and in inverse proportion to the area.

4. The method of claim 3, wherein, The method further comprises: Using the labeled picture including the shopping bags of different shopping seasons of the merchants in each region of the mall as shopping bag training data, a plurality of groups of shopping bag recognition models suitable for different shopping seasons are trained; when training the shopping bag recognition model of the current season, the shopping bag training data uses the shopping bags of the merchants in each region of the mall in the current season as positive sample data, and the shopping bags that have been withdrawn from the market or the shopping bags of other malls as negative sample data.

5. The method of claim 2, wherein, The video picture of each region and the face emotion model are used to obtain the emotional happiness degree of each region, which specifically comprises: Using the face emotion model, the number and category of emotional faces in each region are determined from each human body picture detected by the human body detection model; the category includes happy, calm, and uncomfortable; Based on the number and category of emotional faces, the emotional happiness degree of each region is calculated, which is in inverse proportion to the number of emotional faces and in direct proportion to the number of emotional faces of the happy category and the number of emotional faces of the uncomfortable category.

6. A mall integrated heat map generation system, characterized by, The mall comprehensive heat map generation system comprises a plurality of video cameras and a plurality of thermal imaging cameras distributed in each region, a display screen, and a data processing center, the data processing center comprising one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code comprising computer instructions, and the one or more processors invoke the computer instructions to enable the mall comprehensive heat map generation system to perform the method of any one of claims 1-5.

7. A computer program product comprising instructions, characterized in that, When the computer program product runs on the data processing center of the mall comprehensive heat map generation system, it enables the mall comprehensive heat map generation system to perform the method of any one of claims 1-5.

Images