Multi-angle fusion hall shop scene thermodynamic diagram generation method and device

Abstract

The invention relates to the technical field of passenger flow data processing. The invention provides a multi-angle fusion hall shop scene thermodynamic diagram generation method and device, and the method comprises the steps: collecting multi-angle video data through multi-angle shooting, and extracting an image time sequence at the same moment; processing the extracted image time sequence by using a Yolov8 algorithm, and identifying human shape information; reconstructing a scene three-dimensional model, including performing image feature extraction and feature matching on the image time sequence by using an SIFT algorithm to perform object surface reconstruction and texture mapping so as to obtain a reconstructed hall shop scene three-dimensional model; human shape information in a specified historical time period is identified, human shape data is formed by using the reconstructed hall and store scene three-dimensional model, and a hall and store scene thermodynamic diagram is generated; and obtaining a passenger flow density area corresponding to the to-be-analyzed data according to the generated hall shop scene thermodynamic diagram. According to the invention, the passenger flow density area in the hall is determined, and customer shopping hobbies and habits are determined.

Description

Technical Field

[0001] The present invention belongs to the technical field of passenger flow data analysis, and provides a method and device for generating a heat map of a store scene with multi-angle fusion. Background Art

[0002] With the continuous development of information technology, heat maps, as an intuitive data visualization tool, have been widely used in various stores such as retail, catering, and commercial complexes. Through heat map analysis of customer flow data in stores, merchants can better understand customer behavior habits, consumption preferences, and the rationality of store layout, thereby optimizing store resource allocation, improving customer experience, and achieving sales growth.

[0003] However, existing technologies often rely on a single camera to generate heat maps. Due to the limited viewing angle of the camera, it is impossible to accurately capture certain important areas. This limitation may miss important information and even cause incomplete data in the heat map. In addition, heat maps generated by a single camera are usually unable to conduct in-depth analysis of human posture and movement. The lack of analysis of human posture makes it impossible to understand customer behavior and demand information, which in turn affects the effectiveness of formulating personalized marketing campaign plans. Accurate posture recognition and motion analysis are crucial to understanding customers' interests, behavior patterns, and shopping habits. Without this data, it is impossible to effectively adjust marketing strategies to meet the needs of different customer groups, resulting in poor marketing campaign results and failure to maximize customer experience and sales performance.

[0004] Therefore, it is necessary to provide a method and device for generating a heat map of a store scene with multi-angle fusion to solve the above problems. Summary of the invention

[0005] The present invention provides a method and device for generating a heat map of a hall or store scene with multi-angle fusion, so as to solve the technical problems that the heat map generated by a single camera in the prior art is usually unable to conduct in-depth analysis of human posture and movement, unable to understand customer behavior and demand information, unable to achieve accurate posture recognition and movement analysis, unable to quickly identify customers' points of interest, and thus unable to enable users to formulate personalized marketing activity plans, and the user experience is poor. The technical problem to be solved by the present invention is achieved by the following technical solution.

[0006] The first aspect of the present invention proposes a method for generating a multi-angle fused hall and store scene heat map, the method comprising: collecting multi-angle video data through multi-angle shooting to extract image time series at the same time; using Yolov8 algorithm to process the extracted image time series to identify human shape information, the human shape information including human position information and human body key point information; reconstructing the scene three-dimensional model, specifically comprising the following steps: using multi-angle template images to recalibrate the camera, and use the calibrated scene two-dimensional image collected by the camera to map to three-dimensional space to reconstruct the three-dimensional world of the hall and store scene; using SIFT algorithm to extract image features and match features of the image time series to reconstruct the object surface and texture mapping to obtain a reconstructed hall and store scene three-dimensional model; using Yolov8 algorithm to identify human shape information of a specified historical time period, using the reconstructed hall and store scene three-dimensional model to form human shape data to generate a hall and store scene heat map, and representing the activity density of the characters by color for analyzing the passenger flow density area; according to the generated hall and store scene heat map, analyzing the data to be analyzed to obtain the passenger flow density area corresponding to the data to be analyzed.

[0007] The second aspect of the present invention proposes a hall and store scene heat map generation device, which executes the hall and store scene heat map generation method described in the first aspect of the present invention, and the hall and store scene heat map generation device: an acquisition processing module, which acquires multi-angle video data by multi-angle shooting to extract the image time series at the same time; an identification processing module, which uses the Yolov8 algorithm to process the extracted image time series and identify human shape information, wherein the human shape information includes the position information of the person and the key point information of the human body; a reconstruction module, which is used to reconstruct the scene three-dimensional model, specifically comprising the following steps: using the multi-angle template image, recalibrating the camera, and using the image collected by the calibrated camera The two-dimensional image of the scene is mapped into a three-dimensional space to reconstruct the three-dimensional world of the hall and store scene; the SIFT algorithm is used to extract and match the image features of the image time series to reconstruct the object surface and perform texture mapping to obtain a reconstructed three-dimensional model of the hall and store scene; the generation module uses the Yolov8 algorithm to identify the human figure information of the specified historical time period, and uses the reconstructed three-dimensional model of the hall and store scene to form human figure data to generate a hall and store scene heat map, and the activity density of the characters is represented by color for analyzing the passenger flow density area; the analysis module analyzes the data to be analyzed according to the generated hall and store scene heat map to obtain the passenger flow density area corresponding to the data to be analyzed.

[0008] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a heat map of a hall or store scene as described in the first aspect of the present invention.

[0009] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a heat map of a hall and store scene as described in the first aspect of the present invention is implemented.

[0010] The embodiments of the present invention include the following advantages: Compared with the prior art, the present invention uses multiple cameras to capture real-time images of the store, uses the Yolov8 algorithm to identify human figure information, detects key points of the human body, reconstructs the three-dimensional model of the scene using images taken by multiple cameras, maps the human figure information to the three-dimensional model of the scene, and generates a heat map to determine the customer flow density area in the store. According to the human figure data formed by mapping the human figure information to the three-dimensional model of the scene, the human posture obtained by analyzing the human figure can accurately determine the customer's shopping preferences, preferred areas, etc., and then can accurately determine the placement area and placement height of the goods, which can effectively improve the operating efficiency, customer experience and customer satisfaction of the store. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of the steps of the method for generating a heat map of a hall and store scene by multi-angle fusion of the present invention; Figure 2 It is a flowchart of steps from another angle of the method for generating a heat map of a hall and store scene by multi-angle fusion of the present invention; Figure 3 It is a schematic diagram of an example of reconstructing a scene three-dimensional model in the method for generating a heat map of a hall and store scene by multi-angle fusion of the present invention; Figure 4 It is a structural block diagram of the multi-angle fusion hall and store scene heat map generation system of the present invention; Figure 5 is a schematic structural diagram of an electronic device embodiment according to the present invention; Figure 6 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0012] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0013] In view of the above problems, the present invention proposes a method for generating a heat map of a store scene with multi-angle fusion. The method is based on capturing real-time images of the store with multiple cameras, using the Yolov8 algorithm to identify human figures, detecting key points of the human body, reconstructing the three-dimensional model of the scene using images taken by multiple cameras, mapping the human figure information to the three-dimensional model of the scene, and generating a heat map to determine the passenger flow density area in the store. According to the human figure data formed by mapping the human figure information to the three-dimensional model of the scene, the human posture obtained by analyzing can accurately determine the customer's shopping preferences, preferred areas, etc., and effectively improve the operation efficiency, customer experience and customer satisfaction of the store.

[0014] Example 1 Refer to the following Figure 1 , Figure 2 , Figure 3 , the contents of the present invention will be described in detail.

[0015] Figure 1 It is a flowchart of the steps of the method for generating a heat map of a hall and store scene by multi-angle fusion of the present invention. Figure 2 It is a flowchart of steps from another angle of the method for generating a heat map of a hall and store scene by multi-angle fusion of the present invention.

[0016] See also Figure 1 and Figure 2 In step S101, multi-angle video data is collected by multi-angle shooting to extract the image time series at the same moment.

[0017] In a specific implementation, multiple identical high-definition cameras are installed at different angles in store A to form a comprehensive monitoring network. Figure 3 Through multi-angle shooting, the same moment can be extracted from the video. A set of image sequences, denoted as ,in, is the number of cameras, .

[0018] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0019] Next, in step S102, the extracted image time series is processed using the Yolov8 algorithm to identify human shape information, which includes human position information and human body key point information.

[0020] Specifically, the Yolov8 algorithm is used to calculate the Image time series Processing is performed to identify people, the number of people, key points of the human body, and the position information of each person in the detection frame, wherein the key point information of the human body includes the head, arms, and legs.

[0021] In a specific implementation, a framework is selected to construct a human figure recognition model, and specifically a Yolov8 framework is used to construct the human figure recognition model for target detection and posture estimation.

[0022] Specifically, image data containing human bodies are collected, and image bounding box labels (e.g., x, y, representing width and height, respectively) and posture key points (i.e., human body key points) and coordinate labels of each posture key point are annotated to establish a training data set. The posture key points include shoulders, elbows, knees, head, arms, legs, etc.

[0023] Preferably, the image data of the human body is a sequence of images at the same time t.

[0024] The image data is preprocessed such as normalization and resizing.

[0025] For the training process, it also includes setting hyperparameters, specifically setting hyperparameters such as learning rate, batch size, and number of training rounds.

[0026] Furthermore, the target detection loss and pose estimation loss are defined, and the cross entropy loss and mean square error are used.

[0027] Perform multiple rounds of iterative training, and continuously update the network weights to reduce the target detection loss and the posture estimation loss. For example, when the two loss values ​​are in the range of 0.001 to 0.006, stop training the model, that is, complete the model training, and obtain a trained human recognition model.

[0028] Specifically, at the same time The image sequence to be processed Perform preprocessing and convert the preprocessed image sequence to be processed Input the human figure recognition model and output the coordinate position of the image detection frame, such as the center position , is the total number of people identified, i represents the i-th camera, i is a positive integer, specifically i is 1, 2, 3, ..., N, is the number of cameras, At the same time, for each person identified, the key points of the human body will be accurately located and recorded as , specifically represented by the image captured by the i-th camera Personal The key point coordinate information of the human body is usually returned in the form of an array. The detection of these key points can help understand the customer's posture and movements and provide basic data for subsequent analysis.

[0029] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0030] Next, in step S103, the three-dimensional model of the scene is reconstructed, which specifically includes the following steps: using multi-angle template images to recalibrate the camera, and use the two-dimensional image of the scene captured by the calibrated camera to map it into three-dimensional space to reconstruct the three-dimensional world of the hall and store scene; using the SIFT algorithm to extract and match image features on the image time series to reconstruct the object surface and perform texture mapping to obtain the reconstructed three-dimensional model of the hall and store scene.

[0031] Specifically, reconstructing the scene three-dimensional model includes the following steps: Step S201, using the template images from multiple angles, recalibrating the camera, and using the calibrated camera to map the two-dimensional image of the scene into a three-dimensional space to reconstruct the three-dimensional world of the store scene; For the calibration of each camera, a black and white grid calibration plate is fixed on a plane, namely the template plane, and several template images are taken from different angles to detect the feature points in the template image. Assume that the template plane is in the world coordinate system. On the plane, the relationship between the projection of the three-dimensional point in space onto the template image is: ; in, Indicates the projection of the corresponding points on the template plane image to the three-dimensional points in space; (u, v) is the point on the template plane, is the intrinsic parameter matrix of the camera, and They are the rotation matrix and translation variables of the camera coordinate system relative to the world coordinate system, is the homogeneous coordinate of the corresponding point on the image plane projected from the point on the template plane, u represents the horizontal coordinate of the point on the template plane, v represents the vertical coordinate of the point on the template plane, and l represents the weight of the homogeneous coordinate, which is specifically the number 1; is the homogeneous coordinate of the point on the template plane, in order to solve , and .

[0032] It should be noted that in camera calibration, the 1 in the homogeneous coordinate (u, v, 1) represents the weight of the homogeneous coordinate, which is used to convert the plane point (u, v) into the homogeneous coordinate form (u, v, 1) for various geometric transformations and projection operations.

[0033] For the camera calibration process, calibration plate: select an object with known geometric properties as the calibration plate (i.e., template plane), such as a chessboard. The chessboard consists of black and white squares, and the actual size of each square is known, such as 10mm x 10mm. The camera is used to capture an image of the calibration plate.

[0034] For example, use the OpenCV library for image processing and calibration calculations.

[0035] Specifically, collect calibration data and take multiple pictures of the chessboard from different angles and distances. Make sure that all parts of the chessboard appear in different images, thereby providing enough data to accurately estimate the camera parameters. Next, detect the corners and use OpenCV's findChessboardCorners function to automatically detect the chessboard corners in each image. The positions of these corners in the image will be used to calculate the intrinsic and extrinsic parameters of the camera. Further, calculate the camera parameters and use the collected object points and image points to call the cv2.calibrateCamera function to calculate the camera's intrinsic parameters (focal length, principal point position, distortion coefficient) and extrinsic parameters (rotation matrix and translation vector).

[0036] Use the calculated camera intrinsic parameters (focal length, principal point position, distortion coefficients) and extrinsic parameters (rotation matrix and translation vector) to map points in the 2D image back to the 3D world.

[0037] Step S202, using the SIFT algorithm to perform image feature extraction and feature matching on the image time series to perform object surface reconstruction and texture mapping to obtain a reconstructed three-dimensional model of the store scene.

[0038] The SIFT (Scale-Invariant Feature Transform) algorithm is used to extract sparse feature points of displayed items such as shelves and cabinets in the store from the image time series to obtain a feature point set.

[0039] For example, by finding key point pairs with similar feature point descriptors in two images of a store captured, the corresponding feature points in the two images can be matched for three-dimensional reconstruction.

[0040] The descriptor of each feature point is calculated, and the nearest neighbor matching is used to obtain the point cloud corresponding to the displayed items such as shelves and cabinets in the store to reconstruct the object surface of the key items.

[0041] Specifically calculate the descriptor for each feature point , using nearest neighbor matching, we get matching pairs . Among them, the descriptor is a vector that represents the local area around the feature point.

[0042] For each feature point in image A , calculate its descriptor and the feature points in image B

[0043] Descriptor The Euclidean distance between them is used to find the feature point with the smallest distance As nearest neighbor: ; in, Represents each feature point in image A Descriptor and the feature points in image B

[0044] Descriptor The Euclidean distance between them; k is a positive integer, specifically 1, 2, ..., 128.

[0045] For each feature point , find the nearest neighbor and next nearest neighbor .

[0046] Calculate the distance ratio r: ; if <predetermined value (for example, expressed as threshold), then it is considered and is the matching point.

[0047] After completing the feature point matching, the PMVS (Patch-Match Stereo) algorithm is used to obtain a denser point cloud corresponding to the shelves, cabinets and other display items in the store. The object surface is reconstructed based on these point clouds, and texture mapping is performed to restore the three-dimensional scene and objects.

[0048] Specifically, the PMVS algorithm is a multi-view graph matching algorithm.

[0049] Taking a small supermarket as an example, the process of reconstructing the surface of an object from point cloud is explained.

[0050] Specifically obtain the camera parameters, and obtain the camera parameters in a small supermarket. Set up a calibration board in the small supermarket and take multiple photos from different angles to obtain the camera's internal parameters and distortion parameters. Perform feature point matching by identifying and matching the feature points of key objects (or target objects) such as shelves, refrigerators, and cash registers in the supermarket.

[0051] Next, a sparse 3D point cloud is generated by extracting the feature points. The shelf layout feature points such as the arrangement, spacing and height of the shelves are identified to reconstruct the entrance and exit positions of the supermarket and possible customer channels. The shelf layout feature points specifically include the promotional logo on the top.

[0052] Use algorithms such as PMVS to generate more detailed dense point clouds, specifically generating dense point clouds of various products on the shelves, including dense point clouds of details of bottled beverages, canned foods, daily necessities, etc. In addition, it also includes refining the shape and size of the refrigerator and the placement of goods inside it. In addition, it also includes capturing the shape of the cash register and the placed equipment (such as cash registers, shopping baskets, etc.). The dense point cloud generated above is converted into a three-dimensional mesh model, and the surface of the object is reconstructed to generate a mesh model of the shelf, and the models of various products are refined to ensure that their shapes and sizes are truly reflected.

[0053] Surface reconstruction includes reconstruction of the supermarket floor material (such as tiles or carpets), as well as the wall color and decoration. It also includes promotional posters and signage on the wall to enhance the supermarket’s brand atmosphere.

[0054] Preferably, real textures are added to the three-dimensional model, for example, the extracted textures of the wall and the ground are applied to the corresponding three-dimensional model to create a real shopping environment and make it more realistic.

[0055] Specifically, the extracted product packaging and label images are applied to the corresponding models on the shelves to ensure the accuracy of the information. Finally, the rendering engine is used to generate the final effect, showing the lighting effects of the supermarket, the colors of the products, and the overall shopping atmosphere.

[0056] Through the above process, the point cloud of the displayed items such as shelves and cabinets in the store is texture mapped to restore the store scene and various objects to obtain a reconstructed three-dimensional model of the store scene.

[0057] The relationship between the three-dimensional points in the space and the template image is used to map the two-dimensional image captured by the camera into the three-dimensional space, that is, the three-dimensional model of the store scene.

[0058] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0059] Next, in step S104, the Yolov8 algorithm is used to identify the human figure information of the specified historical time period, and the reconstructed three-dimensional model of the hall and store scene is used to form human figure data to generate a hall and store scene heat map, and the activity density of the characters is represented by color for analyzing the passenger flow density area.

[0060] Specifically, according to the calibrated intrinsic parameter matrix of the camera, the points representing the human figure information in the camera coordinate system are converted into corresponding mapping points in the world coordinate system, so as to convert the human figure information into human figure data in the three-dimensional model of the store scene. The human figure data includes the coordinates of the human bounding box and the human body posture data.

[0061] Camera's intrinsic matrix ,in, , ) is the coordinate of the focal length, are the coordinates of the principal point. Use The coordinates of the jth customer in the image captured by the i-th camera will be represented Convert to normalized coordinates in the camera coordinate system: Calculate the depth , is the actual height of the client, i.e. the person, and its projection height in the camera coordinate system is , z c is the calculated depth of a person in the three-dimensional world, f y Indicates the focal length of the camera on the y-axis.

[0062] Combined with external parameter matrix , convert a point in the camera coordinate system of a certain image to be processed into a corresponding point in the world coordinate system ,in, Respectively represent the coordinate information of the horizontal coordinate X, the vertical coordinate Y, and the third coordinate Z of the corresponding point converted to the world coordinate system; Respectively represent the coordinate information of the point in the camera at the horizontal coordinate X, the vertical coordinate Y, and the third coordinate Z, It represents the result after the point in the camera is rotated, and R(Xc,Yc,Zc)+T represents the result after the point in the camera is rotated and translated.

[0063] The reconstructed three-dimensional model of the hall and store scene is used to form human shape data. Specifically, the human shape information is mapped into three-dimensional space through the above method to obtain human shape data and generate a hall and store scene heat map. Specifically, the human shape data within the specified historical time end is visualized to generate a hall and store scene heat map. The specified historical time period includes the time of the generated store heat map, specifically from the start time to the end time.

[0064] For example, 2023-10-1 to 2023-10-2.

[0065] In one embodiment, the start time By deadline The human-shaped data obtained within the specified historical time period is visualized to generate a heat map of the hall and store scene.

[0066] Furthermore, the activity density of characters is represented by colors to analyze the customer flow density area. The activity density of customers in the store scene is represented by gradient colors and divided into different density areas. Specifically, color mapping technology is used to represent the activity density of customers by gradient colors (such as from yellow to red), where yellow represents low-density areas and red represents high-density areas. This visualization method allows store managers to identify at a glance the areas where customers are concentrated. For details, see Figure 3 Different passenger flow density areas are represented by circles a, b, c, d, and e, among which circle a is red and circle e is yellow.

[0067] In addition, the reconstructed three-dimensional model of the hall and store scene is used to form human shape data. Specifically, the human shape information is mapped into the three-dimensional space through the above method to obtain human shape data, and the changes in people's postures are determined based on the hall and store scene heat map analysis. This can further understand the customer's behavior patterns, identify popular products or preferred areas, and provide accurate data basis for the optimization of operation and maintenance sales layout.

[0068] Preferably, a comparative analysis of human body postures is performed based on the image time series and the identified human shape information to determine the behavioral habits and consumption preferences of each user.

[0069] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0070] Next, in step S105, the data to be analyzed is analyzed according to the generated store scene heat map to obtain the customer flow density area corresponding to the data to be analyzed.

[0071] According to the generated store scene heat map, the corresponding passenger flow density area is determined, and circles a, b, c, d, and e are used to represent different passenger flow density areas. Specifically, circles a to e represent passenger flow density areas with gradually decreasing passenger flow density. For details, see Figure 3 .

[0072] Specifically, the data to be analyzed are all human figure data analyzed from the start time to the end time period, including the coordinate frame position of the person and key point data of the person's posture.

[0073] First, based on the human shape data in the image sequence, specifically the human shape data in a cumulative time period from the start time to the end time, the overlapping effect of the human shape data can be obtained, where the more people there are, the darker the color is, represented by red, that is, a red area is formed, and the overlapping effect of the human shape data (i.e., the red area) is defined as a high-density area of ​​human flow, and the fewer people there are, the lighter the color is, represented by yellow, that is, a yellow area is formed, and the yellow area is defined as a low-density area of ​​human flow. In this way, hall and store merchants can obtain the areas that customers like and can accurately determine the areas where goods are placed.

[0074] Secondly, based on the human posture data in the image sequence, it is also possible to accumulate data over a period of time to determine which products customers are interested in. Therefore, merchants in halls and stores can accurately determine the placement height of the products.

[0075] In an optional embodiment, customer behavior data is determined based on the identified persons, the number of persons, key points of the human body, and the position information of each person in the detection frame, including the customer's stay time in front of a shelf, the action of leaning closer to the product, the action of stretching out both hands, or other actions indicating interest in the product.

[0076] Further, based on the determined customer behavior data, the product display position is adjusted or the product inventory is increased.

[0077] Specifically, a deep network algorithm is used to build an item prediction model to predict the item display position, display height, and item quantity (i.e., the quantity of each commodity).

[0078] Based on the customer behavior data determined above, including the customer's stay time in front of a shelf, actions of leaning towards the product, actions of stretching out both hands or other actions indicating interest in the product, and the customer behavior data marked with item display location, display height, and number of items, a training data set is established to train the item prediction model. When the model accuracy reaches more than 90%, the model training is stopped to obtain a trained item prediction model.

[0079] Furthermore, the customer behavior data of the customer to be predicted (based on the customer behavior data determined in step S105) is input into the item prediction model to obtain the item display position or increase the item inventory corresponding to the customer to be predicted.

[0080] In addition, for store managers, firstly, they can optimize the layout based on the analysis results, and optimize and adjust the merchandise display, shelf placement, rest area settings, etc. in the store. Secondly, according to the division of passenger flow density areas, the number of store employees and working hours are reasonably allocated, and customized marketing plans are made according to the consumption habits of customers in different areas to attract customers to consume.

[0081] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0082] Compared with the prior art, the present invention uses multiple cameras to capture real-time images of the store, uses the Yolov8 algorithm to identify human figure information, detects key points of the human body, reconstructs the three-dimensional model of the scene using images taken by multiple cameras, maps the human figure information to the three-dimensional model of the scene, and generates a heat map to determine the customer flow density area in the store. According to the human figure data formed by mapping the human figure information to the three-dimensional model of the scene, the human posture obtained by analyzing the human figure can accurately determine the customer's shopping preferences, preferred areas, etc., and then can accurately determine the placement area and placement height of the goods, which can effectively improve the operating efficiency, customer experience and customer satisfaction of the store.

[0083] Example 2 The following is a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. For details not disclosed in the device embodiment of the present invention, please refer to the method embodiment of the present invention.

[0084] Figure 4 1 is a schematic diagram of an example of a device for generating a heat map of a store scene according to the present invention. Figure 4 , a hall-store scene heat map generation device is described. The hall-store scene heat map generation device is used to execute the hall-store scene heat map generation method described in the first aspect of the present invention.

[0085] like Figure 4 As shown, the hall-store scene heat map generating device 300 includes a collection processing module 310, a recognition processing module 320, a reconstruction module 330, a generation module 340 and an analysis module 350.

[0086] In a specific embodiment, the acquisition processing module 310 acquires multi-angle video data by shooting from multiple angles to extract the image time series at the same time. The recognition processing module 320 uses the Yolov8 algorithm to process the extracted image time series and identify the human figure information, which includes the position information of the person and the key point information of the human body. The reconstruction module 330 is used to reconstruct the scene three-dimensional model, which specifically includes the following steps: using the template image of multiple angles to recalibrate the camera, and use the scene two-dimensional image acquired by the calibrated camera to map to the three-dimensional space to reconstruct the three-dimensional world of the hall and store scene; using the SIFT algorithm to extract image features and match features of the image time series to reconstruct the object surface and texture mapping to obtain the reconstructed hall and store scene three-dimensional model. The generation module 340 uses the Yolov8 algorithm to identify the human figure information of the specified historical time period, and uses the reconstructed hall and store scene three-dimensional model to form human figure data to generate a hall and store scene heat map, and the activity density of the characters is represented by color for analyzing the passenger flow density area. The analysis module 350 analyzes the data to be analyzed based on the generated store scene heat map to obtain the passenger flow density area corresponding to the data to be analyzed.

[0087] According to an optional implementation, the SIFT algorithm is used to extract sparse feature points corresponding to the displayed items in the store from the image time series to obtain a feature point set. The descriptor of each feature point is calculated, and the nearest neighbor matching is used to obtain the point cloud corresponding to the displayed items to reconstruct the object surface of the key items; after completing the feature point matching, the PMVS algorithm is used to obtain a denser point cloud corresponding to the displayed items, and the object surface is reconstructed based on these denser point clouds, and texture mapping is performed; the displayed items include shelves, refrigerators, and cash registers.

[0088] According to an optional implementation, texture mapping is performed on the point clouds of the following key objects to restore the store scene and each object to obtain a reconstructed three-dimensional model of the store scene: shelves, cabinets, and cash registers in the store; The surface reconstruction of objects includes the reconstruction of the material of the store floor, the color and decoration of the walls, the promotional posters and signs on the walls.

[0089] According to an optional implementation manner, the reconstructed three-dimensional model of the hall and store scene is used to form human shape data to generate a heat map of the hall and store scene.

[0090] According to the calibrated camera's internal parameter matrix, the points representing the human figure information in the camera coordinate system are converted into corresponding mapping points in the world coordinate system, so as to convert the human figure information into human figure data in the three-dimensional model of the hall and store scene. The human figure data in the specified historical time period is visualized to generate a hall and store scene heat map.

[0091] According to an optional implementation method, the activity density of customers in a store scene is represented by gradient colors and divided into different passenger flow density areas. The number of store clerks and working hours are configured according to the different passenger flow density areas.

[0092] According to an optional implementation manner, a comparative analysis of human body postures is performed based on the image time series and the identified human shape information to determine the behavioral habits and consumption preferences of each user.

[0093] Specifically use the Yolov8 algorithm to calculate the Image time series Processing is performed to identify people, the number of people, key points of the human body, and the position information of each person in the detection frame, wherein the key point information of the human body includes the head, arms, and legs.

[0094] Customer behavior data is determined based on the identified people, the number of people, key points of the human body, and the position information of each person in the detection frame, including the customer's stay time in front of a shelf, the action of leaning closer to the product, the action of stretching out both hands, or other actions that indicate interest in the product.

[0095] Further, based on the determined customer behavior data, the product display position is adjusted or the product inventory is increased.

[0096] According to an optional implementation, a black and white grid calibration plate is fixed on a plane, namely, a template plane, and several template images are taken from different angles to detect feature points in the template images. Assuming that the template plane is in the world coordinate system On the plane, the relationship between the projection of the three-dimensional point in space onto the template image is: ; in, Indicates the projection of the corresponding points on the template plane image to three-dimensional points in space; (u, v) is the point on the template plane; is the intrinsic parameter matrix of the camera, and They are the rotation matrix and translation variables of the camera coordinate system relative to the world coordinate system, is the homogeneous coordinate of the corresponding point on the image plane projected from the point on the template plane, u represents the horizontal coordinate of the point on the template plane, v represents the vertical coordinate of the point on the template plane, and l represents the weight of the homogeneous coordinate, which is specifically the number 1; is the homogeneous coordinate of the point on the template plane, in order to solve , and ; The relationship between the three-dimensional points in the space and the template image is used to map the two-dimensional image captured by the camera into the three-dimensional space, that is, the three-dimensional model of the store scene.

[0097] It should be noted that due to Figure 4 A method for generating a heat map of a hall and store scene executed by a hall and store scene heat map generating device and Figure 1 The method of generating the heat map of the hall and store scene in the examples is roughly the same, so the description of the same parts is omitted.

[0098] Compared with the prior art, the present invention uses multiple cameras to capture real-time images of the store, uses the Yolov8 algorithm to identify human figures, detects key points of the human body, reconstructs the three-dimensional model of the scene using images taken by multiple cameras, maps the human figure information to the three-dimensional model of the scene, and generates a heat map to determine the customer flow density area in the store. The human figure data formed by mapping the human figure information to the three-dimensional model of the scene and the human posture obtained by analysis can accurately determine the customer's shopping preferences, preferred areas, etc., effectively improving the operating efficiency, customer experience and customer satisfaction of the store.

[0099] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0100] like Figure 5 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0101] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0102] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0103] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0104] It should be understood that Figure 5The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0105] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 6 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0106] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0107] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0108] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the data interaction method of the present disclosure.

[0110] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0111] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0112] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0113] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a heat map of a store scene with multi-angle fusion, characterized in that: The method comprises: Collect multi-angle video data through multi-angle shooting to extract the image time series at the same moment; The extracted image time series is processed using the Yolov8 algorithm to identify human shape information, which includes human position information and human body key point information; Reconstructing the three-dimensional model of the scene specifically includes the following steps: using a template image from multiple angles to recalibrate the camera, and using the two-dimensional image of the scene collected by the calibrated camera to map it into a three-dimensional space to reconstruct the three-dimensional world of the hall and store scene; using the SIFT algorithm to extract and match image features on the image time series to perform object surface reconstruction and texture mapping to obtain a reconstructed three-dimensional model of the hall and store scene; Use the Yolov8 algorithm to identify human figure information in a specified historical time period, and use the reconstructed three-dimensional model of the hall and store scene to form human figure data to generate a hall and store scene heat map, and represent the activity density of the characters with colors to analyze the passenger flow density area; According to the generated heat map of the hall and store scene, the data to be analyzed is analyzed to obtain the passenger flow density area corresponding to the data to be analyzed.

2. The method for generating a heat map of a store scene according to claim 1 is characterized in that: The SIFT algorithm is used to extract and match image features from the image time series to perform object surface reconstruction and texture mapping to obtain a reconstructed three-dimensional model of the store scene, including: Using SIFT algorithm to extract sparse feature points related to the displayed items in the store from the image time series to obtain a feature point set; Calculate the descriptor of each feature point, use nearest neighbor matching to obtain the point cloud corresponding to the displayed items to reconstruct the object surface of the key items; After completing the feature point matching, the PMVS algorithm is used to obtain a denser point cloud corresponding to the displayed items, and the object surface is reconstructed based on these denser point clouds, and texture mapping is performed; the displayed items include shelves, refrigerators, and cash registers.

3. The method for generating a heat map of a store scene according to claim 1, characterized in that: Further including: Texture mapping is performed on the point clouds of the following key objects to restore the store scene and each object to obtain a reconstructed 3D model of the store scene: shelves, cabinets, and cash registers in the store; The surface reconstruction of objects includes the reconstruction of the material of the store floor, the color and decoration of the walls, the promotional posters and signs on the walls.

4. The method for generating a heat map of a store scene according to claim 1, characterized in that: The method of using the reconstructed three-dimensional model of the hall and store scene to form human shape data to generate a hall and store scene heat map includes: According to the calibrated internal parameter matrix of the camera, the points representing the human figure information in the camera coordinate system are converted into corresponding mapping points in the world coordinate system, so as to convert the human figure information into human figure data in the three-dimensional model of the store scene; The humanoid data within a specified historical time period is visualized to generate a heat map of the hall and store scene.

5. The method for generating a heat map of a store scene according to claim 1, characterized in that: Further including: The activity density of customers in the hall store scene is expressed by gradient colors and divided into different customer flow density areas. The number of store clerks and working hours are configured according to different customer flow density areas.

6. The method for generating a heat map of a store scene according to claim 1, characterized in that: Further including: Based on the image time series and the recognized human figure information, a comparative analysis of human body postures is performed to determine the behavioral habits and consumption preferences of each user.

7. The method for generating a heat map of a store scene according to claim 1, characterized in that: Further including: Use Yolov8 algorithm to calculate the Image time series Processing is performed to identify people, the number of people, key points of the human body, and the position information of each person in the detection frame, wherein the key points of the human body include the head, arms, and legs; Determine customer behavior data based on the identified people, number of people, key points of the human body, and location information of each person in the detection frame, including the customer's stay time in front of a shelf, actions of leaning towards the product, actions of stretching out both hands, or other actions indicating interest in the product; Further, based on the determined customer behavior data, the product display position is adjusted or the product inventory is increased.

8. The method for generating a heat map of a store scene according to claim 1, characterized in that: Further including: A black and white grid calibration plate is fixed on a plane, namely the template plane, and several template images are taken from different angles to detect the feature points in the template image. Assume that the template plane is in the world coordinate system. On the plane, the relationship between the projection of the three-dimensional point in space onto the template image is: ； in, Indicates the projection of the corresponding points on the template plane image to three-dimensional points in space; (u, v) is the point on the template plane; is the intrinsic parameter matrix of the camera, and They are the rotation matrix and translation variables of the camera coordinate system relative to the world coordinate system, is the homogeneous coordinate of the corresponding point on the image plane projected from the point on the template plane, u represents the horizontal coordinate of the point on the template plane, v represents the vertical coordinate of the point on the template plane, and l represents the weight of the homogeneous coordinate, which is specifically the number 1; is the homogeneous coordinate of the point on the template plane, to solve , and ; The relationship between the three-dimensional points in the space and the template image is used to map the two-dimensional image captured by the camera into the three-dimensional space, that is, the three-dimensional model of the store scene.

9. A device for generating a heat map of a store scene, characterized in that: It executes the method for generating a heat map of a hall and store scene according to any one of claims 1 to 8, and the heat map generating device for a hall and store scene comprises: The acquisition and processing module acquires video data from multiple angles by shooting from multiple angles to extract the image time series at the same moment; The recognition processing module uses the Yolov8 algorithm to process the extracted image time series and recognize the human figure information, which includes the position information of the person and the key point information of the human body; The reconstruction module is used to reconstruct the three-dimensional model of the scene, which specifically includes the following steps: using the template images from multiple angles to recalibrate the camera, and using the two-dimensional images of the scene collected by the calibrated camera to map into the three-dimensional space to reconstruct the three-dimensional world of the hall and store scene; using the SIFT algorithm to extract image features and match features on the image time series to perform object surface reconstruction and texture mapping to obtain the reconstructed three-dimensional model of the hall and store scene; The generation module uses the Yolov8 algorithm to identify human figure information in a specified historical time period, and uses the reconstructed three-dimensional model of the hall and store scene to form human figure data to generate a hall and store scene heat map, and represents the activity density of the characters by color, which is used to analyze the passenger flow density area; The analysis module analyzes the data to be analyzed according to the generated heat map of the hall and store scene, and obtains the passenger flow density area corresponding to the data to be analyzed.

10. The device for generating a heat map of a store scene according to claim 9, characterized in that: include: Using SIFT algorithm to extract sparse feature points related to the displayed items in the store from the image time series to obtain a feature point set; Calculate the descriptor of each feature point, use nearest neighbor matching to obtain the point cloud corresponding to the displayed items to reconstruct the object surface of the key items; After completing the feature point matching, the PMVS algorithm is used to obtain a denser point cloud corresponding to the displayed items, and the object surface is reconstructed based on these denser point clouds, and texture mapping is performed; the displayed items include shelves, refrigerators, and cash registers.

Images