A method and system for museum visit analysis based on BIM and video surveillance

Abstract

The present invention provides a method for museum visit analysis based on BIM and video surveillance, comprising the following steps: S1, performing laser point cloud scanning on the interior of the museum, completing museum BIM modeling, generating a voxel model, and recording camera posture fitting results into the BIM model; S2, calling a video stream, intercepting corresponding video frames, calibrating the internal parameters of the camera, and integrating the results into a matrix camera internal parameter matrix K; S3, calculating the three-dimensional voxel coordinates corresponding to the coordinates of each pixel in the video stream according to the voxel model, the camera posture and the internal parameter K of the camera, obtaining the corresponding relationship between pixels and voxels, and completing the spatial registration of the monitoring video image and the BIM model; S4, detecting human key points in the video frame, saving the pixel positions of the bipedal nodes in the human key point results, accessing the corresponding relationship between pixels and voxels, and performing indoor positioning of the audience; S5, obtaining the visiting time of all exhibition areas and exhibits, and statistically analyzing the attention.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for museum visit analysis based on BIM and video surveillance. Background Art

[0002] Visitor management is an important part of the daily work of museums. At the same time, under the requirements of normalized epidemic prevention and control, strictly controlling the number of visitors and avoiding the gathering of visitors are the top priorities of current visitor management. Museums often need to invest considerable manpower to ensure the orderly visit of visitors. In addition, the visitor's visit behavior is also an important feedback and reference for museums in exhibition planning and exhibition area exhibit layout. Against the background of the rapid development of information technology, more intelligent and efficient visitor analysis and management can be achieved with the help of three-dimensional digitization, building information modeling and visual data understanding.

[0003] Building Information Modeling (BIM) technology is a data-based tool used in engineering design, construction, and management. The core of BIM is to establish a virtual three-dimensional model of building information and use digital technology to support various management and analysis functions inside the building. Surveillance cameras are permanent security facilities in museums. In traditional security work, surveillance videos are generally watched and warned by staff. On the one hand, specific manpower needs to be arranged, and on the other hand, it is possible that warnings cannot be issued in time due to problems such as staff fatigue. Automated surveillance video analysis and warning can reduce the security burden of museum staff and provide an extra layer of protection for visit management under the background of epidemic prevention and control. In addition, in addition to meeting security needs, surveillance videos also record a large number of audience visit images, and automatically identify and count the visits of surveillance video streams, which can quantitatively analyze the exhibition effect and provide more accurate audience feedback references for exhibition planning and the layout of exhibits in the exhibition area. In view of this, the present invention provides a method and system for museum visit analysis based on BIM and video surveillance. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a method and system for museum visit analysis based on BIM and video surveillance to solve the above-mentioned technical problems.

[0005] The technical method adopted by the present invention to solve its technical problem is: a method for museum visit analysis based on BIM and video surveillance, and its improvement lies in: comprising the following steps: S1, a BIM model construction module performs laser point cloud scanning on the interior of the museum, completes the museum BIM modeling, generates a voxel model, and records the camera posture fitting result into the BIM model; S2, a video stream acquisition and calibration module calls the video stream, intercepts the corresponding video frame, calibrates the internal parameters of the camera, and integrates the results into a matrix camera internal parameter matrix K; S3, a spatial registration module performs a calibration based on the voxel model, the camera posture and the camera The internal parameter K calculates the three-dimensional voxel coordinates corresponding to the coordinates of each pixel in the video stream, obtains the correspondence between pixels and voxels, and completes the spatial alignment of the monitoring video image and the BIM model; S4, the audience detection and positioning module detects the key points of the human body in the video frame, saves the pixel positions of the bipedal nodes in the human body key point results, and accesses the correspondence between the pixels and voxels to determine the voxels where the audience's feet are located, and locates the audience indoors; S5, the attention analysis module obtains the visiting time of all exhibition areas and exhibits according to the audience positioning results, normalizes the visiting time data of the exhibition areas and exhibits, and counts the attention.

[0006] In the above method, step S6 is also included, and the attention analysis module performs accessibility analysis on the exhibition area and exhibits. The step S6 includes the following steps: S61, calculating the shortest path from the museum entrance to the ground voxel area of ​​the exhibition area; calculating the shortest path from the museum entrance to the center point of the exhibit; calculating the number of ground voxels corresponding to the exhibition area; calculating the volume of the outer cuboid of the exhibit voxel; calculating the inverse of the shortest distance between the exhibit and the wall voxel; S62, using the A* algorithm to calculate the five indicators in the step S61; S63, normalizing the five indicators to obtain the accessibility indicators of the exhibition area and exhibits:

[0007] Accessibility of exhibition area = (1 / length of exhibition area path) × exhibition area scale

[0008] Exhibit accessibility = (1 / exhibit path length) × exhibit scale × exhibit centrality.

[0009] In the above method, step S1 comprises the following steps:

[0010] S11. Use mobile laser radar scanning equipment to perform laser point cloud scanning inside the museum in a segmented scanning manner;

[0011] S12. Use the RandLA-Net algorithm to perform 3D semantic segmentation on each segmented point cloud to divide different BIM model elements;

[0012] S13, calling the Registration interface of Open3D to register each segmented point cloud to a unified spatial coordinate reference;

[0013] S14, performing axis alignment operation on the global point cloud;

[0014] S15, using the existing digital exhibit model of the museum as a three-dimensional template, performing template matching and three-dimensional spatial position fitting in the point cloud, determining the position and posture of the digital exhibit model in the point cloud, generating a three-dimensional point cloud of the digital exhibit, and using the digital exhibit point cloud to replace the exhibit point cloud obtained by scanning;

[0015] S16, creating a voxel model of the exhibit in the museum BIM model coordinate system according to the fitted three-dimensional model and posture of the exhibit;

[0016] S17. Conduct 3D modeling of the camera model used by the museum, use the camera 3D model as a template, perform template matching operations and 3D pose fitting in the point cloud, calculate the 3D position coordinates and rotation angle of the camera in the BIM coordinate system, that is, the camera's extrinsic parameter T. The 3D position coordinates are represented by a 3D vector t, and the 3D rotation angle is represented by a 3D matrix R. The two are written as the camera extrinsic parameter matrix T = [R|t], and the camera pose fitting results are recorded in the BIM model.

[0017] In the above method, the axis alignment operation in step S14 is to rotate the coordinate system of the point cloud around the z-axis, and the steps for calculating the rotation angle are as follows:

[0018] S141, calling the EstimateNormal function in the Open3D library to calculate the normal vectors of all points in the point cloud, and normalizing the normal vectors so that the three-dimensional length of each normal vector is 1;

[0019] S142, calculating the projection direction and length of the normal vector in the horizontal direction. If the length is greater than a threshold value of 0.5, the point is judged to belong to a vertical structure and needs to be retained for calculation of the rotation angle. If the length is less than the threshold value of 0.5, the point is removed and does not participate in the calculation of the rotation angle.

[0020] S143, establish optimization objective function: Δθ i is the difference between the fitted angle and the horizontal projection angle of the normal vector of a certain point, and N is the number of points involved in the calculation of the rotation angle;

[0021] S144. Use derivative-free optimization method to solve, call nlopt library to complete the solution process, and obtain the rotation angle.

[0022] In the above method, step S2 comprises the following steps:

[0023] S21, determining the camera models used in the museum, and selecting a camera from each model for calibration;

[0024] S22, calling the video stream through the API provided by the camera manufacturer, placing the Zhang Zhengyou calibration chessboard in front of each calibration camera, the camera shoots the selected fixed position, and intercepts the corresponding video frame in the video stream;

[0025] S23. Use the findChessboardCorners and cal ibrateCamera functions in the opencv library to calibrate the camera's intrinsic parameters and obtain the intrinsic parameter K of each camera model.

[0026] In the above method, in step S3, the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream are calculated, and the pixel coordinates P i (u,v) and P c The relationship is: The camera coordinate system is established with the camera optical center as the origin, the front of the camera as the z axis, and the horizontal and vertical directions of the imaging plane as the x and y axes respectively. K is the camera internal parameter. In the camera coordinate system, the coordinates of the captured point are P c (x c ,y c ,z c ), z c is the distance from the photographed point to the optical center of the camera, and the coordinates of the photographed point are P c and the coordinate P of the point in the BIM model coordinate system w (x w ,y w ,z w ) has a spatial relationship of P C =TP w , T is the external parameter of the camera, that is, the rotation and translation of the camera coordinate system relative to the BIM model coordinate system [R|t].

[0027] In the above method, step S4 comprises the following steps:

[0028] S41, using the Mask R-CNN architecture in the computer vision processing library Detectron to detect key points of the human body in the video frame;

[0029] S42, create a data set of video images, annotate the instance contours and human key points of the audience in the data set, and train it on the pre-trained model of the Detectron library;

[0030] S43, intermittently run detection, save the pixel positions of the two-foot nodes in the human body key point results, and access the correspondence between the pixels and voxels to determine the voxels where the audience's feet are located, and perform indoor positioning of the audience.

[0031] In the above method, step S5 comprises the following steps:

[0032] S51, adding a "whether to view the exhibits" branch in the Mask R-CNN, adding a "whether to view the exhibits" annotation in the data set, and training it simultaneously with the human key point detection branch;

[0033] S52, when detecting a new video stream, synchronously outputting the bipedal node pixels of the audience, and judging whether the audience is viewing the exhibits. If it is judged that the audience is “not viewing the exhibits”, the audience is not counted in the number of people viewing the exhibits; if it is judged that the audience is “viewing the exhibits”, the voxel corresponding to the detected bipedal pixel is obtained through the mapping relationship between the pixel and the voxel;

[0034] S53, judging the voxels, when the voxels are divided into the visiting area of ​​a specific exhibit, the detected audience is counted into the number of viewers of the exhibit in the frame; when the voxels are divided into a specific exhibition area, the detected audience is counted into the number of viewers of the exhibition area in the frame. The number of visitors measured in each frame is the visiting time of the exhibition area and the exhibits. The time data of the exhibition area and the exhibits are normalized and the attention is counted.

[0035] In the above method, it also includes step S7, the crowd density analysis and early warning module generates a ground thermal voxel model according to the audience positioning result, displays the voxel color according to the density, and completes the crowd density analysis and early warning;

[0036] The step S7 comprises the following steps:

[0037] S71, generating a ground thermal voxel model according to the ground voxels corresponding to the bipedal pixels;

[0038] S72, displaying voxel colors according to density through a three-dimensional visualization interface;

[0039] S73. Set a density threshold. When the number of people in a voxel exceeds the set threshold, a gathering alarm message pops up in the three-dimensional visualization interface. Click on the message and the three-dimensional view locates the voxel position where the density is higher than the threshold.

[0040] The present invention also provides a system for museum visit analysis based on BIM and video surveillance, including a BIM model construction module, a video stream acquisition and calibration module, a space registration module, an audience detection and positioning module, and an attention analysis module.

[0041] The BIM model building module is used to perform laser point cloud scanning inside the museum, complete the museum BIM modeling, generate a voxel model, and record the camera pose fitting results into the BIM model;

[0042] The video stream acquisition and calibration module is used to call the video stream, intercept the corresponding video frame, calibrate the camera's internal parameters, and obtain the internal parameter K of each model of camera;

[0043] The spatial registration module is connected to the BIM model construction module and the video stream acquisition and calibration module, and is used to calculate the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream according to the voxel model, the camera posture and the camera's internal parameter K, obtain the corresponding relationship between pixels and voxels, and complete the spatial registration of the monitoring video image and the BIM model;

[0044] The audience detection and positioning module is connected to the video stream acquisition and calibration module and the spatial registration module, and is used to detect the key points of the human body in the video frame, save the pixel positions of the bipedal nodes in the human body key point results, and access the corresponding relationship between the pixels and voxels to determine the voxels where the audience's feet are located, and perform indoor positioning of the audience;

[0045] The attention analysis module is connected to the audience detection and positioning module, and after obtaining the visiting time of all exhibition areas and exhibits according to the audience positioning result, the visiting time data of the exhibition areas and exhibits are normalized to calculate the attention.

[0046] The beneficial effects of the present invention are as follows: based on the point cloud and the existing three-dimensional models of exhibits and cameras in the museum, a museum BIM model is constructed, the pixels in the BIM model and the surveillance video are spatially registered, the pixel points of the audience's feet are detected, and the measured two-foot pixel coordinates are mapped to the three-dimensional space coordinates of the BIM model to complete the audience positioning; based on the positioning result, combined with the accessibility of exhibits and exhibition areas, the number of visitors who visit the exhibition area and view exhibits in a given period of time is counted to analyze the audience's attention to each exhibit and exhibition area; and real-time crowd density monitoring and early warning can be achieved. Museum staff can set a crowd density alarm threshold. Once there is a voxel or voxel area that maintains a high crowd density for a certain period of time, the audience can be properly guided to avoid crowd gathering; the existing surveillance video network of the museum is used, and there is no need to install and set up new equipment, and no additional equipment cost is added, so low-cost museum audience density analysis and exhibit area attention analysis are achieved, which has high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Attached Figure 1 The present invention is a flow chart of a method for museum visit analysis based on BIM and video surveillance.

[0048] Attached Figure 2 It is a schematic diagram of the correspondence between the bipedal coordinates and the three-dimensional coordinate system of the BIM model in the present invention.

[0049] Attached Figure 3 It is a schematic diagram of the correspondence between pixel coordinates and voxel coordinates in the video image of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0052] Reference Figure 1 As shown, a method for museum visit analysis based on BIM and video surveillance of the present invention comprises the following steps:

[0053] S1, BIM model building module performs laser point cloud scanning inside the museum, completes the museum BIM modeling, generates a voxel model, and records the camera pose fitting results into the BIM model;

[0054] Specifically, the step S1 includes the following steps:

[0055] S11. Use mobile laser radar scanning equipment to scan the laser point cloud inside the museum. To avoid positioning drift caused by long-track scanning, segmented scanning is used, and the horizontal area of ​​each segment is controlled within 50 square meters to facilitate subsequent semantic segmentation;

[0056] S12. Use the RandLA-Net algorithm to perform 3D semantic segmentation on each segmented point cloud to divide different BIM model elements such as walls, floors, and steps. RandLA-Net is a neural network model based on random point sampling and local feature aggregation for large-scale point cloud semantic segmentation tasks.

[0057] S13, calling the Registration interface of Open3D to register each segmented point cloud to a unified spatial coordinate reference, Open3D is an open source algorithm library for three-dimensional data, and the Registration interface is a point cloud registration interface;

[0058] S14. To reduce the precision loss of subsequent point cloud and voxel processing, the global point cloud of the museum is axis-aligned. The axis-aligned operation is to rotate the coordinate system of the point cloud around the z-axis (vertical direction) so that most of the vertical structures (walls, etc.) in the point cloud are parallel to the x and y axes in the new coordinate system. The key to axis alignment is the calculation of the rotation angle, and the calculation steps of the rotation angle are as follows: S141. Call the EstimateNormal function in the Open3D library to calculate the normal vectors of all points in the point cloud, and normalize the normal vectors so that the three-dimensional length of each normal vector is 1. The EstimateNormal function is the normal vector estimation function; S142. Calculate the projection direction and length of the normal vector in the horizontal direction. If the length is greater than the threshold value of 0.5, it is judged that the point belongs to the vertical structure and needs to be retained to participate in the calculation of the rotation angle. If it is less than the threshold value of 0.5, the point is eliminated and does not participate in the calculation of the rotation angle; S143. Establish the optimization objective function: Δθ i is the difference between the fitted angle and the horizontal projection angle of the normal vector of a certain point, and N is the number of points involved in the calculation of the rotation angle; S144, using the derivative-free optimization method to solve, calling the nlopt library to complete the solution process, and obtain the rotation angle, the nlopt library is a nonlinear derivative-free optimization algorithm library.

[0059] S15. After the axis alignment operation is completed, the existing digital exhibit model of the museum is regarded as a three-dimensional template, and template matching and three-dimensional spatial position fitting are performed in the point cloud: a three-dimensional sliding window operation is performed on the point cloud, and in a sliding window of the appropriate size, the average point error of the model at each pose (including angle and position) is calculated. If the average point error is less than a threshold, it is considered that the pose of the digital exhibit model in the point cloud is determined. After the pose of the digital exhibit model in the point cloud is determined, a three-dimensional point cloud of the digital exhibit is generated, and the digital exhibit point cloud is used to replace the exhibit point cloud obtained by scanning;

[0060] S16. Create a voxel model of the exhibit in the museum BIM model coordinate system based on the fitted three-dimensional model and position of the exhibit. After the voxel model is generated, the museum staff marks the ground voxels corresponding to each exhibition area in the voxel model interaction software. In this embodiment, the voxel model is stored in three parts: (1) Independent voxel model: The file header records the origin coordinates and voxel side length of the voxel model. Each voxel records the three-dimensional coordinates and attributes according to (vid, x, y, z, tid, pid, rid), where vid is the voxel id, x, y and z are the three-dimensional coordinates of the voxel, all of which are integers. The attributes include tid, which indicates the type of voxel (wall: 0, ground: 1, step: 2, exhibit: 3); pid, if the voxel is an exhibit voxel, pid is the exhibit id in the corresponding digital exhibit information system; rid, if the voxel is a ground voxel and belongs to a certain exhibition area, rid is the id of the corresponding exhibition area. Independent voxel files are stored in text files and can be compressed into binary files as needed; (2) Digital exhibits associated storage: In the museum's existing digital exhibits information system, a corresponding voxel field is added, and the voxel vid corresponding to the exhibit is recorded in the field in a collection manner; (3) Exhibition area associated storage: In the museum's existing operation and management database, a new exhibition area table is added, or the original exhibition area table is directly expanded, and a corresponding voxel field is added, and the voxel vid corresponding to the exhibition area is recorded in the field in a collection manner;

[0061] S17. Conduct 3D modeling of the camera model used by the museum. The model uses absolute size and uses the camera 3D model as a template. Template matching operations and 3D pose fitting are performed in the point cloud. The matching method is similar to the template matching of the 3D model of the digital exhibit. The 3D position coordinates and rotation angle of the camera in the BIM coordinate system are calculated, that is, the camera's extrinsic parameters T (hereinafter referred to as camera extrinsic parameters). The 3D position coordinates can be represented by a 3D vector t, and the 3D rotation angle can be represented by a 3D matrix R. These two can be written as the camera extrinsic parameter matrix T = [R|t], and the camera pose fitting results (including camera extrinsic parameters and camera model) are recorded in the BIM model.

[0062] S2, the video stream acquisition and calibration module calls the video stream, intercepts the corresponding video frame, calibrates the intrinsic parameters of the camera, the intrinsic parameters of the camera include the camera focal length, imaging plane translation and distortion, etc., and integrates the results into the matrix camera intrinsic parameter matrix K;

[0063] Specifically, the step S2 includes the following steps:

[0064] S21, determining the camera models used in the museum, and selecting a camera from each model for calibration;

[0065] S22, calling the video stream through the API (video stream acquisition interface) provided by the camera manufacturer, placing the Zhang Zhengyou calibration chessboard in front of each calibration camera (that is, the Zhang Zhengyou calibration method is used in this embodiment), selecting several fixed positions to be photographed by the camera, and intercepting the corresponding video frames in the video stream;

[0066] S23. Use the findChessboardCorners and cal ibrateCamera functions in the opencv library to calibrate the camera's intrinsic parameters and obtain the intrinsic parameter K of each camera model. The intrinsic parameter K is recorded in the BIM model to support subsequent real-time and batch solutions. The opencv library is an open source computer vision algorithm library, findChessboardCorners is a chessboard corner detection function, and cal ibrateCamera is a camera parameter calibration function.

[0067] S3, the spatial registration module calculates the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream according to the voxel model, the camera posture and the camera's internal parameter K, obtains the corresponding relationship between pixels and voxels, and completes the spatial registration of the monitoring video image and the BIM model;

[0068] Specifically, in step S3, the three-dimensional voxel coordinates corresponding to the pixel coordinates of the video stream are calculated, referring to Figure 2 As shown, the pixel coordinate P i (u,v) and P c The relationship is: The camera coordinate system is established with the camera optical center as the origin, the front of the camera as the z axis, and the horizontal and vertical directions of the imaging plane as the x and y axes respectively. K is the camera internal parameter. In the camera coordinate system, the coordinates of the captured point are P c (x c ,y c ,z c ), z c is the distance from the photographed point to the optical center of the camera, and the coordinates of the photographed point P c and the coordinate P of the point in the BIM model coordinate system w (x w ,y w ,z w ) has a spatial relationship of P C =TP w , T is the external parameter of the camera, that is, the rotation and translation of the camera coordinate system relative to the BIM model coordinate system [R|t]. When there is only a monocular camera, z c This solution uses the established 3D voxel model to search for different z pixels one by one. cThe nearest voxel found in the BIM model is used as the 3D spatial position corresponding to the pixel, thus realizing the registration of the pixel to the BIM model. Furthermore, the voxel coordinates corresponding to each pixel successfully solved by the camera are recorded in the camera attributes to support the subsequent audience detection and positioning module to locate the audience indoors.

[0069] S4, the audience detection and positioning module detects the key points of the human body in the video frame, saves the pixel positions of the bipedal nodes in the human body key point results, accesses the correspondence between the pixels and the voxels, determines the voxels where the audience's feet are located, and positions the audience indoors;

[0070] Specifically, step S4 includes the following steps:

[0071] S41. Use Mask R-CNN in the computer vision processing library Detectron to detect key points of human bodies in video frames. This architecture can handle occlusion well. When the audience blocks each other in the video frame, it can also estimate the occluded key positions well. The Mask R-CNN architecture is an instance segmentation algorithm based on masks and convolutional neural networks.

[0072] S42. In order to enable the Mask R-CNN model to obtain good detection results under the camera view of the museum, a data set containing 500 frames of video images was created, the instance contours and human key points of the audience in the data set were annotated, and the training was performed on the pre-trained model of the Detectron library.

[0073] S43, run a test every 5 seconds, save the pixel positions of the two-foot nodes in the human key point results, access the correspondence between the pixels and the voxels, determine the voxels where the audience's feet are located, and locate the audience indoors.

[0074] S5, the attention analysis module obtains the visiting time of all exhibition areas and exhibits according to the audience positioning result, normalizes the visiting time data of the exhibition areas and exhibits, and calculates the attention;

[0075] Specifically, step S5 includes the following steps:

[0076] S51. Add a whether to view the exhibit branch similar to the classification branch to the existing three branches of the Mask R-CNN. Except that the output is converted into a real number instead of a vector, the other structures are the same as the original classification branch structure of the Mask R-CNN.

[0077] In order to train the branch of viewing exhibits in collaboration with the audience detection and positioning module, a new annotation of viewing exhibits is added to the data set of the audience detection and positioning module, and is trained simultaneously with the human body key point detection branch of the audience detection and positioning module;

[0078] S52, after the model training is completed, that is, when detecting a new video stream, the bipedal node pixels of the audience are synchronously output to determine whether the audience is viewing the exhibits. If it is determined that the audience is "not viewing the exhibits", the audience is not counted in the number of people viewing the exhibits; if it is determined that the audience is "viewing the exhibits", the voxel corresponding to the detected bipedal pixel is obtained through the mapping relationship between the pixel and the voxel;

[0079] S53. Reference Figure 3 As shown, the voxels are judged. When the voxels are divided into the visiting area of ​​a specific exhibit, the detected audience is counted into the number of viewers of the exhibit in the frame; when the voxels are divided into a specific exhibition area, the detected audience is counted into the number of viewers of the exhibition area in the frame. The visiting time of a certain exhibit or exhibition area in a certain time period is the number of visitors to the exhibit measured in each frame. After obtaining the visiting time of all exhibition areas and exhibits, the time data of the exhibition areas and exhibits are normalized respectively to calculate the final attention.

[0080] Visitors staying in the exhibition viewing area does not directly mean that they are visiting the exhibits, so it is necessary to identify the behavior of the visitors. Therefore, supervised learning methods can be used to determine whether visitors are visiting the corresponding exhibits when they stay in the exhibition area. Furthermore, in order to conduct supervised learning of viewing behavior judgment, the corresponding true value data needs to be labeled; further, an image processing method based on deep learning is used, based on the cutting-edge classification convolutional neural network, and the true value labeled data is used to perform parameter fine-tuning.

[0081] Furthermore, the method further includes step S6, wherein the attention analysis module performs accessibility analysis on the exhibition area and exhibits.

[0082] Specifically, step S6 includes the following steps:

[0083] S61, (1) Exhibition area path length: calculate the shortest path from the museum entrance to the exhibition area's ground voxel area; (2) Exhibit path length: calculate the shortest path from the museum entrance to the exhibit center point; (3) Exhibition area scale: calculate the number of ground voxels corresponding to the exhibition area; (4) Exhibit scale: calculate the volume of the outer cuboid of the exhibit voxel; (5) Exhibit centrality: calculate the inverse of the shortest distance (path) between the exhibit and the wall voxel, that is, the farther away from the wall, the stronger the centrality;

[0084] S62, using the A* algorithm to calculate the five indicators in step S61;

[0085] S63, normalizing the five indicators to obtain the accessibility indicators of the exhibition area and exhibits:

[0086] Accessibility of exhibition area = (1 / length of exhibition area path) × exhibition area scale

[0087] Exhibit accessibility = (1 / exhibit path length) × exhibit scale × exhibit centrality.

[0088] According to the audience detection and positioning results, the number of visitors recorded in the voxel area corresponding to the exhibition area and the exhibits at different timestamps is recorded, and it is judged whether the audience staying in front of the exhibits is viewing the exhibits. In addition, the existing exhibition area location distribution in the museum leads to different spatial accessibility, and the difference in accessibility will greatly affect the possibility of the exhibition area being visited by the audience. The scheme of the present invention conducts a comprehensive analysis of the attention based on the visit time of the audience and the accessibility of the exhibition area, providing a more accurate curatorial reference for museum staff. Museum staff can analyze the visiting time and accessibility of the exhibition area and exhibits respectively, and can also calculate the "net attention" of the exhibits and exhibition areas, that is, the normalized visiting time / accessibility, to view the attention of each exhibit and exhibition area after eliminating the influence of accessibility, which helps museum managers to find exhibition areas or exhibits with high accessibility but not enthusiastic audience response, or exhibits that are not high in accessibility but still attract audiences.

[0089] Furthermore, the method further includes step S7, wherein the crowd density analysis and early warning module generates a ground thermal voxel model according to the audience positioning result, displays the voxel color according to the density, and completes the crowd density analysis and early warning;

[0090] Specifically, step S7 includes the following steps:

[0091] S71, generating a ground thermal voxel model according to the ground voxels corresponding to the two-foot pixels, if one foot falls in a certain voxel, the number of people in the voxel in a certain video frame is increased by 1;

[0092] S72. Provide a three-dimensional visualization interface to managers to display voxel colors according to density;

[0093] S73. Set a density threshold. When the number of people in a voxel exceeds the set threshold, a gathering alarm message pops up in the 3D visualization interface. Click on the message and the 3D view locates the voxel position where the density is higher than the threshold. The staff can use this information to determine whether to provide path guidance to the audience at that position.

[0094] The present invention also provides a system for museum visit analysis based on BIM and video surveillance, including a BIM model construction module, a video stream acquisition and calibration module, a space registration module, an audience detection and positioning module, and an attention analysis module.

[0095] The BIM model construction module is used to perform laser point cloud scanning on the interior of the museum, complete the museum BIM modeling, generate a voxel model, and record the camera pose fitting results into the BIM model; further, considering that some museums already have ready-made 3D exhibit models, this solution can use the existing digital 3D models of exhibits in the pilot museums to match and fit the 3D spatial positions in the LiDAR point clouds, and determine the 3D pose of each exhibit in the museum, i.e., the position coordinates and angles, and the data and the corresponding exhibit model numbers are stored in the BIM model; further, this solution can create a voxel model of each exhibit in the museum BIM model coordinate system based on the fitted 3D model and pose of the exhibits; further, considering that the setting of the exhibition area is generally flexible and changeable, this solution is for staff or modelers to circle and mark on the ground voxel model, which can be referred to Figure 3 The ground voxels divided by the middle exhibition area 1; further, set the distance threshold for viewing the exhibits, and divide the viewing area voxels corresponding to each exhibit in the ground voxels, which can be referred to Figure 3 Ground voxel segmentation of exhibits AD;

[0096] The video stream acquisition and calibration module is used to call the video stream, intercept the corresponding video frame, calibrate the camera's internal parameters, and obtain the internal parameter K of each model of camera;

[0097] The spatial registration module is connected to the BIM model construction module and the video stream acquisition and calibration module, and is used to calculate the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream according to the voxel model, the camera posture and the camera's internal parameter K, obtain the corresponding relationship between pixels and voxels, and complete the spatial registration of the monitoring video image and the BIM model;

[0098] The audience detection and positioning module is connected to the video stream acquisition and calibration module and the spatial registration module, and is used to detect the key points of the human body in the video frame, save the pixel positions of the bipedal nodes in the human body key point results, and access the corresponding relationship between the pixels and voxels to determine the voxels where the audience's feet are located, and perform indoor positioning of the audience;

[0099] The attention analysis module is connected to the audience detection and positioning module, and after obtaining the visiting time of all exhibition areas and exhibits according to the audience positioning result, the visiting time data of the exhibition areas and exhibits are normalized to calculate the attention.

[0100] The present invention constructs a museum BIM model based on point clouds and the existing three-dimensional models of exhibits and cameras in the museum, performs spatial registration on pixels in the BIM model and surveillance video, detects the bipedal pixel points of visitors, and maps the measured bipedal pixel coordinates to the three-dimensional spatial coordinates of the BIM model to complete the audience positioning. Based on the positioning result and in combination with the accessibility of exhibits and exhibition areas, the number of visitors who visit exhibition areas and view exhibits in a given period of time is counted to analyze the audience's attention to each exhibit and exhibition area; and real-time crowd density monitoring and early warning can be achieved. Museum staff can set a crowd density alarm threshold. Once there is a voxel or voxel area that maintains a high crowd density for a certain period of time, the audience can be properly guided to avoid crowd gathering; the existing surveillance video network of the museum is used, and there is no need to install and set up new equipment, and no additional equipment cost is added, so low-cost museum audience density analysis and exhibit area attention analysis are achieved, and it has high practicability.

[0101] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for museum visit analysis based on BIM and video surveillance, characterized in that: The steps include: S1, BIM model building module performs laser point cloud scanning inside the museum, completes the museum BIM modeling, generates a voxel model, and records the camera pose fitting results into the BIM model; S2, the video stream acquisition and calibration module calls the video stream, intercepts the corresponding video frame, calibrates the intrinsic parameters of the camera, and integrates the results into the matrix camera intrinsic parameter matrix K; S3, the spatial registration module calculates the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream according to the voxel model, the camera posture and the camera's internal parameter K, obtains the corresponding relationship between pixels and voxels, and completes the spatial registration of the monitoring video image and the BIM model; S4, the audience detection and positioning module detects the key points of the human body in the video frame, saves the pixel positions of the bipedal nodes in the human body key point results, accesses the correspondence between the pixels and the voxels, determines the voxels where the audience's feet are located, and positions the audience indoors; S5. After obtaining the visiting time of all exhibition areas and exhibits according to the audience positioning result, the attention analysis module normalizes the visiting time data of the exhibition areas and exhibits and calculates the attention.

2. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 1, characterized in that: The step S6 is also included, where the attention analysis module performs accessibility analysis on the exhibition area and exhibits, and the step S6 includes the following steps: S61, calculating the shortest path from the museum entrance to the ground voxel area of ​​the exhibition area; Calculate the shortest path from the museum entrance to the center of the exhibit; calculate the number of ground voxels corresponding to the exhibition area; calculate the volume of the outer cuboid of the exhibit voxels; calculate the inverse of the shortest distance between the exhibit and the wall voxel; S62, use the A* algorithm to calculate the five indicators in step S61; S63, normalize the five indicators to obtain the accessibility indicators of the exhibition area and the exhibits: Accessibility of exhibition area = (1 / length of exhibition area path) × exhibition area scale Exhibit accessibility = (1 / exhibit path length) × exhibit scale × exhibit centrality.

3. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 1, characterized in that: The step S1 comprises the following steps: S11. Use mobile laser radar scanning equipment to perform laser point cloud scanning inside the museum in a segmented scanning manner; S12. Use the RandLA-Net algorithm to perform 3D semantic segmentation on each segmented point cloud to divide different BIM model elements; S13, calling the Registration interface of Open3D to register each segmented point cloud to a unified spatial coordinate reference; S14, performing axis alignment operation on the global point cloud; S15, using the existing digital exhibit model of the museum as a three-dimensional template, performing template matching and three-dimensional spatial position fitting in the point cloud, determining the position and posture of the digital exhibit model in the point cloud, generating a three-dimensional point cloud of the digital exhibit, and using the digital exhibit point cloud to replace the exhibit point cloud obtained by scanning; S16, creating a voxel model of the exhibit in the museum BIM model coordinate system according to the fitted three-dimensional model and posture of the exhibit; S17. Conduct 3D modeling of the camera model used by the museum, use the camera 3D model as a template, perform template matching operations and 3D pose fitting in the point cloud, calculate the 3D position coordinates and rotation angle of the camera in the BIM coordinate system, that is, the camera's extrinsic parameter T. The 3D position coordinates are represented by a 3D vector t, and the 3D rotation angle is represented by a 3D matrix R. The two are written as the camera extrinsic parameter matrix T = [R|t], and the camera pose fitting results are recorded in the BIM model.

4. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 3, characterized in that: The axis alignment operation in step S14 is to rotate the coordinate system of the point cloud around the z-axis. The calculation steps of the rotation angle are as follows: S141, calling the EstimateNormal function in the Open3D library to calculate the normal vectors of all points in the point cloud, and normalizing the normal vectors so that the three-dimensional length of each normal vector is 1; S142, calculating the projection direction and length of the normal vector in the horizontal direction. If the length is greater than a threshold value of 0.5, the point is judged to belong to a vertical structure and needs to be retained for calculation of the rotation angle. If the length is less than the threshold value of 0.5, the point is removed and does not participate in the calculation of the rotation angle. S143, establish optimization objective function: Δθ i is the difference between the fitted angle and the horizontal projection angle of the normal vector of a certain point, and N is the number of points involved in the calculation of the rotation angle; S144. Use derivative-free optimization method to solve, call nlopt library to complete the solution process, and obtain the rotation angle.

5. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 4, characterized in that: The step S2 comprises the following steps: S21, determining the camera models used in the museum, and selecting a camera from each model for calibration; S22, calling the video stream through the API provided by the camera manufacturer, placing the Zhang Zhengyou calibration chessboard in front of each calibration camera, the camera shoots the selected fixed position, and intercepts the corresponding video frame in the video stream; S23. Use the findChessboardCorners and calibrateCamera functions in the opencv library to calibrate the camera's intrinsic parameters and obtain the intrinsic parameter K of each camera model.

6. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 5, characterized in that: In step S3, the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream are calculated. i (u,v) and P c The relationship is: The camera coordinate system is established with the optical center of the camera as the origin, the front of the camera as the z axis, and the horizontal and vertical directions of the imaging plane as the x and y axes respectively. K is the camera internal parameter. In the camera coordinate system, the coordinates of the captured point are P c (x c ,y c ,z c ), z c is the distance from the photographed point to the optical center of the camera, and the coordinates of the photographed point P c and the coordinate P of the point in the BIM model coordinate system w (x w ,y w ,z w ) has a spatial relationship of P C =TP w , T is the external parameter of the camera, that is, the rotation and translation of the camera coordinate system relative to the BIM model coordinate system [R|t].

7. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 6, characterized in that: The step S4 comprises the following steps: S41, using Mask R-CNN in the computer vision processing library Detectron to detect key points of the human body in the video frame; S42, create a data set of video images, annotate the instance contours and human key points of the audience in the data set, and train it on the pre-trained model of the Detectron library; S43, intermittently run detection, save the pixel positions of the two-foot nodes in the human body key point results, and access the correspondence between the pixels and voxels to determine the voxels where the audience's feet are located, and perform indoor positioning of the audience.

8. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 7, characterized in that: The step S5 comprises the following steps: S51, adding a "whether to view the exhibits" branch in the Mask R-CNN, adding a "whether to view the exhibits" annotation in the data set, and training it simultaneously with the human key point detection branch; S52, when detecting a new video stream, synchronously output the bipedal node pixels of the audience to determine whether the audience is viewing the exhibits. If it is determined that the audience is "not viewing the exhibits", the audience is not counted in the number of people viewing the exhibits; if it is determined that the audience is "viewing the exhibits", the voxel corresponding to the detected bipedal pixel is obtained through the mapping relationship between the pixel and the voxel; S53, judging the voxels, when the voxels are divided into the visiting area of ​​a specific exhibit, the detected audience is counted into the number of viewers of the exhibit in the frame; when the voxels are divided into a specific exhibition area, the detected audience is counted into the number of viewers of the exhibition area in the frame. The number of visitors measured in each frame is the visiting time of the exhibition area and the exhibits. The time data of the exhibition area and the exhibits are normalized and the attention is counted.

9. The method for museum visit analysis based on BIM and video surveillance as claimed in claim 8, characterized in that: The method also includes step S7, wherein the crowd density analysis and early warning module generates a ground thermal voxel model according to the audience positioning result, displays the voxel color according to the density, and completes the crowd density analysis and early warning; The step S7 comprises the following steps: S71, generating a ground thermal voxel model according to the ground voxels corresponding to the bipedal pixels; S72, displaying voxel colors according to density through a three-dimensional visualization interface; S73. Set a density threshold. When the number of people in a voxel exceeds the set threshold, a gathering alarm message pops up in the three-dimensional visualization interface. Click on the message and the three-dimensional view locates the voxel position where the density is higher than the threshold.

10. A system for museum visit analysis based on BIM and video surveillance, characterized in that: It includes BIM model construction module, video stream acquisition and calibration module, space registration module, audience detection and positioning module and attention analysis module. The BIM model building module is used to perform laser point cloud scanning inside the museum, complete the museum BIM modeling, generate a voxel model, and record the camera pose fitting results into the BIM model; The video stream acquisition and calibration module is used to call the video stream, intercept the corresponding video frame, calibrate the camera's internal parameters, and obtain the internal parameter K of each model of camera; The spatial registration module is connected to the BIM model construction module and the video stream acquisition and calibration module, and is used to calculate the three-dimensional voxel coordinates corresponding to each pixel coordinate of the video stream according to the voxel model, the camera posture and the camera's internal parameter K, obtain the corresponding relationship between pixels and voxels, and complete the spatial registration of the monitoring video image and the BIM model; The audience detection and positioning module is connected to the video stream acquisition and calibration module and the spatial registration module, and is used to detect the key points of the human body in the video frame, save the pixel positions of the bipedal nodes in the human body key point results, and access the corresponding relationship between the pixels and voxels to determine the voxels where the audience's feet are located, and perform indoor positioning of the audience; The attention analysis module is connected to the audience detection and positioning module, and after obtaining the visiting time of all exhibition areas and exhibits according to the audience positioning result, the visiting time data of the exhibition areas and exhibits are normalized to calculate the attention.

Images