Calibration method and equipment for virtual-real mapping of video stream and BIM model

Abstract

The present invention provides a calibration method and device for virtual-real mapping of video stream and BIM model. The present invention can reuse the existing fixed monitoring probes in the building, calibrate the internal and external parameters of the monitoring probes, and convert the two-dimensional coordinates of the personnel in the video stream into three-dimensional coordinates in the BIM three-dimensional model through spatial mapping, so as to realize the accurate position positioning of the personnel and restore the movement trajectory of the personnel; the present invention uses the spatial data and structural information of BIM to accurately analyze and understand the environmental layout inside the building, thereby improving the positioning accuracy and efficiency of the personnel, drawing the movement trajectory, and realizing digital security management; the present invention not only retains the low-cost advantage of the existing monitoring system, but also significantly improves the positioning accuracy and real-time tracking capability, saving deployment and maintenance costs. At the same time, coordinate conversion and trajectory generation are realized through software algorithms, which greatly improves management efficiency, and has broad application prospects and significant technical advantages.

Description

Technical Field

[0001] The invention relates to a calibration method and equipment for virtual-real mapping of a video stream and a BIM model. Background Art

[0002] Building Information Modeling (BIM) is a new digital tool for architecture, engineering and civil engineering. It integrates all kinds of building information into a three-dimensional model information database, improving the information integration and digitalization of construction projects. With the continuous development of digital twin technology for public buildings, smart buildings are increasingly using BIM to achieve refined management of the interior of the building, but the precise positioning and mapping of personnel in the refined BIM three-dimensional model is still one of the difficulties.

[0003] Traditional indoor positioning methods are limited by hardware equipment and user participation, and often cannot meet the flexibility and immediacy of real-world needs. Currently, the commonly used positioning card solutions have problems such as insufficient accuracy and positioning errors; face capture machines cannot meet open security scenarios and the cost of deploying new equipment is high. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for calibrating virtual-reality mapping of a video stream and a BIM model.

[0005] In order to solve the above problems, the present invention provides a calibration method for virtual-reality mapping of a video stream and a BIM model, comprising:

[0006] Step 1: Build a BIM 3D model of the building;

[0007] Step 2: Obtain the calibrated camera image calibrated by the camera's intrinsic parameters through a standard black and white checkerboard calibration plate;

[0008] Step 3: Compare and map the feature points of the calibrated camera image with the BIM 3D model to obtain the 2D coordinates of the feature points in the calibrated camera image. and the 3D coordinates Q in the BIM 3D model;

[0009] Step 4: 2D coordinates of feature points in the calibrated camera image and the three-dimensional coordinates Q of the BIM three-dimensional model, and obtain the mapping relationship matrix between the calibrated camera image and the BIM three-dimensional model;

[0010] Step 5: Capture and identify the face, and generate the movement trajectory of the corresponding person based on the face capture and recognition and coordinate mapping relationship matrix.

[0011] Furthermore, in the above method, step 1: constructing a BIM three-dimensional model of a building includes:

[0012] Step 1.1: Use BIM software to create a BIM 3D model of the building, which includes: the structure of the building, the internal layout, and the fixed configuration objects in the room; obtain the BIM floor elevation d; add the location of existing video surveillance equipment such as cameras in the building to the refined BIM 3D model, and determine the spatial information of each camera, including the camera coordinates R （ , , ) and the camera's orientation angle , integrate the spatial information of each camera into the BIM 3D model;

[0013] Step 1.2: Based on the spatial information of the room walls and fixed configuration objects in the room in the BIM 3D model, determine the layout of each room, corridor and passage, as well as the distribution of stairs and elevators, and use gridding to establish the spatial path topology in the building to obtain a grid coordinate point set. .

[0014] Furthermore, in the above method, step 2: obtaining a calibrated camera image calibrated by the camera's intrinsic parameters through a standard black and white checkerboard calibration plate, includes:

[0015] Step 2.1: Make a standard black and white checkerboard calibration board for internal reference calibration. The side length of each small square grid on the standard black and white checkerboard calibration board is defined as s, with a total of A grid of small squares;

[0016] Step 2.2: Use the standard black and white checkerboard calibration plate to capture the camera internal reference calibration image: Place the calibration plate in the image captured by the camera, such as Figure 2 As shown in the figure, calibration images are taken in the corresponding i areas in the captured image, where in each area, the grids of the calibration plate are made to occupy the corresponding area as much as possible under the premise of being fully visible, and the grids of the calibration plate are ensured to be clearly visible in the camera image. K screenshots are taken from each area as the calibrated camera images of the area, where the calibrated camera images of the five areas are respectively .

[0017] Furthermore, in the above method, step 3: mapping the feature points of the calibrated camera image with the BIM three-dimensional model to obtain the two-dimensional coordinates of the feature points in the calibrated camera image. And the 3D coordinates Q in the BIM 3D model, including:

[0018] Step 3.1: Based on the calibrated camera image and the refined BIM 3D model, obtain the feature points in the calibrated camera image. And the corresponding feature points on the BIM 3D model ; Among them, the definition of feature points is: point It is clearly visible in the calibrated camera image and can be found in the BIM 3D model. Corresponding points ;

[0019] Step 3.2: If 8 or more pairs of feature points are found in the calibrated camera image and the refined BIM 3D model , When 8 or more pairs of feature points are directly extracted, the two-dimensional coordinates of all feature points in the calibrated camera image are and the three-dimensional coordinates Q in the BIM three-dimensional model;

[0020] Step 3.3: If only 7 or fewer pairs of feature points are found, select one pair of feature points from the 7 or fewer pairs of feature points as the base points , based on the base point Create feature points for center fit fixed calibration tools.

[0021] Furthermore, in the above method, step 3.3: if only 7 or fewer pairs of feature points are found, one pair of feature points is selected from the 7 or fewer pairs of feature points as the base points. , based on the base point Create feature points for center fit fixed calibration tools, including:

[0022] 1) If only 7 or fewer pairs of feature points are found, select one pair of feature points from the 7 or fewer pairs of feature points as the base points ;

[0023] 2) Select a pole of fixed length e and place it vertically on the ground for calibration;

[0024] 3) Based on base points Starting from the current origin, select a nearby wall as the reference surface and compare the angle of the reference surface to the Select the current moving distance outward. The ground point obtained is ;

[0025] 4) Place a pole of fixed length e perpendicular to the ground at a point on the ground , define the top point of the pole as , and take a screenshot from the calibrated camera screen to get the first external parameter calibration image , among which, it can be clearly confirmed that and ;

[0026] 5) Using the newly acquired ground point As the current origin, repeat the above steps 3) and 4) at least four times, and use the current length Get at least four pairs of points and , and , and , and , that is, at least get the base point A total of 9 points, including , and the corresponding external reference calibration image set is obtained ,in As the base point images;

[0027] 6) If Figure 4a and 4b As shown, the image set is calibrated by external parameters Get base points And the two-dimensional coordinates of other points P, and obtain the corresponding point Q in the BIM three-dimensional model, where the base point Corresponding ground point The coordinates of can be directly read from the BIM 3D model, and the 3D coordinates of the remaining points Q are calculated by the formula:

[0028] ;

[0029] ;

[0030] in, is the BIM 3D coordinate of the ground point Q corresponding to the current origin P, The BIM three-dimensional coordinates of the top point Q corresponding to the current origin P;

[0031] X, Y are the BIM two-dimensional coordinates of the ground point Q corresponding to the origin P before the current origin P;

[0032] Z is the height of the ground point Q corresponding to the origin P before the current point P. If it is a ground point, it is the BIM floor elevation d. If it is the top point of the pole, it is the BIM floor height plus the pole length e.

[0033] is the moving distance compared to the current origin P;

[0034] It is the angle between the moving vector and the reference wall, which is used to determine the moving direction.

[0035] Furthermore, in the above method, step 4: based on the two-dimensional coordinates of the feature point in the calibrated camera image And the three-dimensional coordinates Q in the BIM three-dimensional model, the mapping relationship matrix between the calibrated camera image and the BIM three-dimensional model is obtained, including:

[0036] Step 4.1: Correct the camera lens distortion and set the radial distortion model to:

[0037] ;

[0038] where (x, y) is the undistorted normalized image coordinate, ( ) is the distorted image coordinate, is the radial distance, is the radial distortion coefficient;

[0039] Step 4.2: The internal reference calibration image obtained in step 2.2 is the calibrated camera image Input into the algorithm to obtain a square in the calibration plate The three-dimensional coordinates of , where s is the side length of a square on the chessboard, is the number of squares in the width direction of the chessboard, is the number of squares in the height direction of the chessboard, , is a positive integer;

[0040] Step 4.3: The camera image will be calibrated Convert to grayscale image;

[0041] Step 4.4: Traverse the grayscale image to find the corner points of each square in the calibration plate, and record the 2D image coordinates of each corner point , and add the 3D coordinates of each square in the calibration plate to the object point list

[0042] ;

[0043] Step 4.5: Use the fisheye model to calibrate the camera through the calibrate function to calculate the camera's intrinsic parameter matrix K and distortion coefficient D:

[0044] ;

[0045] in, and is the focal length in the x and y directions, and is the optical center coordinate;

[0046] ;

[0047] in, is the radial distortion coefficient,

[0048] Step 4.6: Based on the radial distortion model, the intrinsic parameter matrix K and the distortion coefficient D, the distorted image coordinates are obtained. , to map the distorted image coordinates to the undistorted image coordinates:

[0049] ;

[0050] in, are the coordinates of the rectified undistorted image;

[0051] Step 4.7: Based on the intrinsic parameter matrix K, distortion coefficient D and undistorted image coordinates , calculate the mapping matrix group of the undistorted feature points P and Q, and obtain the intrinsic parameter matrix of the calibration camera , rotation matrix and translation vectors ;

[0052] Step 4.8: Combine the camera coordinates R and BIM floor elevation d, based on the internal parameter matrix , rotation matrix and translation vectors , and obtain the coordinate mapping relationship matrix.

[0053] Furthermore, in the above method, step 5: capturing and identifying the face, and generating the movement trajectory of the corresponding person based on the capturing and identifying the face and the coordinate mapping relationship matrix, includes:

[0054] Step 5.1: Based on the image of the calibrated camera, capture the face and obtain the face capture record;

[0055] Step 5.2: Based on Retrieve all facial capture records of the same person within a certain period of time;

[0056] Step 5.3: Extract the two-dimensional coordinates of all the faces of the same person retrieved in the picture to form a two-dimensional coordinate group , through the coordinate relationship mapping matrix of step 4, calculate the three-dimensional coordinate group mapped to the BIM three-dimensional model ;

[0057] Step 5.4: Group the three-dimensional coordinates Pre-converted to , used to match gridded coordinates;

[0058] The conversion formula is:

[0059] ,

[0060] Step 5.5: Process the completed The gridded coordinate point set obtained in step 1.2 Match according to the minimum distance method and map it to the BIM 3D model ground to get the minimum distance point ;

[0061] The calculation formula for the minimum distance point is:

[0062] ,

[0063] The coordinate points are expressed as , The coordinate points are expressed as ;

[0064] Step 5.6: The minimum distance point is obtained According to the shortest path method, the final personnel movement trajectory is obtained , draw it in the BIM 3D model to get the final personnel trajectory.

[0065] Furthermore, in the above method, step 5.1: capturing a face based on the image of the calibration camera to obtain a face capture record includes:

[0066] 1) When the face of a person q appears in the picture for the first time, a face capture record is captured, and all face capture records captured before the face leaves the picture are marked accordingly , q is the number of different people;

[0067] 2) When the face remains in the picture, a face capture record is captured every preset seconds;

[0068] 3) When the face leaves the screen, capture a face capture record and release the corresponding q of a certain person mark.

[0069] According to another aspect of the present invention, there is further provided a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: execute any of the methods described above.

[0070] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0071] Processor; and

[0072] A memory arranged to store computer executable instructions, which when executed cause the processor to: perform any of the methods described above.

[0073] Compared with the prior art, the present invention can reuse the existing fixed monitoring probes in the building, calibrate the internal and external parameters of the monitoring probes, and convert the two-dimensional coordinates of the personnel in the video stream into three-dimensional coordinates in the BIM three-dimensional model through spatial mapping, so as to realize the accurate positioning of the personnel and restore the movement trajectory of the personnel. The present invention uses the spatial data and structural information of BIM to accurately analyze and understand the environmental layout inside the building, thereby improving the positioning accuracy and efficiency of personnel, drawing movement trajectories, and realizing digital security management. Compared with the traditional positioning card or face capture machine solution, this method not only retains the low-cost advantage of the existing monitoring system, but also significantly improves the positioning accuracy and real-time tracking capabilities, saving deployment and maintenance costs. At the same time, coordinate conversion and trajectory generation are realized through software algorithms, which greatly improves management efficiency, and has broad application prospects and significant technical advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flow chart of a method for calibrating virtual-real mapping of a video stream and a BIM model according to an embodiment of the present invention;

[0075] Figure 2 is a schematic diagram of a chessboard-calibrated screen ratio according to an embodiment of the present invention;

[0076] Figure 3 is a schematic diagram of obtaining a mapping calibration image according to an embodiment of the present invention;

[0077] Figure 4a and 4b It is a schematic diagram of the coordinates of the points inversely calculated in the BIM three-dimensional model and the comparison mapping according to an embodiment of the present invention;

[0078] Figure 5 is a schematic diagram of a room space grid according to an embodiment of the present invention;

[0079] Figure 6 is a schematic diagram of an image before and after distortion correction according to an embodiment of the present invention;

[0080] Figure 7a~7d It is a schematic diagram of comparison mapping calibration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0082] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (CPU), input / output interfaces, network interfaces and memories.

[0083] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0084] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0085] like Figure 1 As shown, the present invention provides a calibration method for virtual-real mapping of a video stream and a BIM model, the method comprising:

[0086] Step 1: Build a BIM 3D model of the building;

[0087] Step 1.1: Build a detailed BIM 3D model and video surveillance model of the building:

[0088] 1) Use BIM software (such as Revit, AutoCAD, etc.) to create a detailed BIM 3D model of the building and obtain basic information about the interior. The BIM 3D model of the building includes: the structure of the building, the internal layout, and the fixed configuration objects in the room. Specifically, the location and size of key areas such as each room, corridor, and staircase can be marked in the BIM 3D model to ensure the accuracy and detail integrity of the model to reach the level of digital twins;

[0089] 2) Obtain the BIM floor elevation d, which is used to determine the spatial position of the identified person;

[0090] Specifically, this implementation case takes a camera on the first floor of a building as an example to obtain the BIM floor elevation d=0m, which is then used to determine the spatial position of a person or object;

[0091] 3) Add the location of existing video surveillance equipment such as cameras in the building to the refined BIM 3D model, and determine the spatial information of each camera, including the camera coordinates R （ , , ) and the camera's orientation angle , integrate the spatial information of each camera into the BIM 3D model;

[0092] Specifically, the location of existing video surveillance equipment in the building can be added to the refined BIM 3D model, and the spatial information of each camera can be determined, including: camera coordinates R (9.599, -40.498, -3.670) and the camera's orientation angle =41.2°, integrated into the BIM 3D model;

[0093] Step 1.2: Based on the spatial information of the room walls and fixed configuration objects in the room in the BIM 3D model, determine the layout of each room, corridor and passage, as well as the distribution of stairs and elevators, and use gridding to establish the spatial path topology in the building to obtain a grid coordinate point set. ,like Figure 5 As shown;

[0094] Step 2: Obtain the calibrated camera image calibrated by the camera's intrinsic parameters through a standard black and white checkerboard calibration plate;

[0095] Step 2.1: Make a standard black and white checkerboard calibration board for internal reference calibration. The side length of each small square grid on the standard black and white checkerboard calibration board is defined as s, with a total of A grid of small squares;

[0096] Here, a standard black and white chessboard can be made for internal reference calibration: the side length s of each square grid on the chessboard calibration plate is , a total of A grid of small squares;

[0097] Step 2.2: Use the standard black and white checkerboard calibration plate to capture the camera internal reference calibration image: Place the calibration plate in the image captured by the camera, such as Figure 2As shown in the figure, calibration images are taken in the corresponding i areas in the captured image, where in each area, the grids of the calibration plate are made to occupy the corresponding area as much as possible under the premise of being fully visible, and the grids of the calibration plate are ensured to be clearly visible in the camera image. K screenshots are taken from each area as the calibrated camera images of the area, where the calibrated camera images of the five areas are respectively ;

[0098] Here, you can use a standard black and white chessboard calibration plate to capture the camera internal reference calibration image. Place the chessboard in the camera image and take calibration images in the corresponding 5 areas. Each area should be filled as much as possible under the premise that all the chessboard grids are visible, and ensure that the calibration plate is clearly visible in the camera image. Take a screenshot of each area. Figure 3 The calibrated camera images of the five areas are ;

[0099] Step 3: Compare and map the feature points of the calibrated camera image with the BIM 3D model to obtain the 2D coordinates of the feature points in the calibrated camera image. and the 3D coordinates Q in the BIM 3D model;

[0100] Step 3.1: Based on the calibrated camera image and the refined BIM 3D model, obtain the feature points in the calibrated camera image. And the corresponding feature points on the BIM 3D model ,like Figure 3 As shown; the definition of feature points is: point It is clearly visible in the calibrated camera image and can be found in the BIM 3D model. Corresponding points ;

[0101] Step 3.2: If 8 or more pairs of feature points are found in the calibrated camera image and the refined BIM 3D model , When the 2D coordinates of all the feature points in the calibrated camera image are directly extracted, 8 or more pairs of feature points are directly extracted. The three-dimensional coordinates Q of the BIM three-dimensional model and the coordinates of the feature points are recorded in a table, as shown in Table 1;

[0102] ;

[0103] Step 3.3: If only 7 or fewer pairs of feature points are found, select one pair of feature points from the 7 or fewer pairs of feature points as the base points , based on the base point Create feature points for the center fit fixed calibration tool:

[0104] Specifically, Figure 7a~7d As shown in the figure, if only 7 or fewer pairs of feature points are found in step 3.3, select a more obvious feature point on the ground. In this case, by comparing the calibrated camera image with the BIM 3D model, the corner point of the elevator door can be used as the reference point, and the space to the right is large, which is suitable as a base point to extend outward, recorded as the base point (820,448), based on the base point Create feature points for the center fit fixed calibration tool:

[0105] 1) If only 7 or fewer pairs of feature points are found, select one pair of feature points from the 7 or fewer pairs of feature points as the base points ;

[0106] 2) Select a pole of fixed length e and place it vertically on the ground for calibration;

[0107] Here, a pole with a fixed length of 1.5m can be selected for vertical placement on the ground for calibration;

[0108] 3) Based on base points Starting from the current origin, select a nearby wall as the reference surface and compare the angle of the reference surface to the Select the current moving distance outward. The ground point obtained is ;

[0109] Here, we can start from the base point and select the elevator wall as the reference surface, and compare the angle of the reference surface to the The length of the outward selection is The ground point obtained is ;

[0110] 4) Place a pole of fixed length e perpendicular to the ground at a point on the ground , define the top point of the pole as , and take a screenshot from the calibrated camera screen to get the first external parameter calibration image , among which, it can be clearly confirmed that and ;

[0111] 5) Using the newly acquired ground point As the current origin, repeat the above steps 3) and 4) at least four times, and use the current length Get at least four pairs of points and , and , and , and , that is, at least get the base point A total of 9 points, including , and the corresponding external reference calibration image set is obtained ,in As the base point images;

[0112] Specifically, here The rod with a fixed length e in the previous step is used as a reference. The value of can be equal, for example, it can be a fixed length e of the pole; The values ​​can also be different. For example, different lengths can be set with a tape measure as several sets of distance values. The core purpose is to be able to reverse calculate the position in the BIM model according to the actual length recorded in the BIM 3D model.

[0113] Here, each pair of points is obtained, and the ground points are As the current origin;

[0114] 6) If Figure 4a and 4b As shown, the image set is calibrated by external parameters Get base points And the two-dimensional coordinates of other points P, and obtain the corresponding point Q in the BIM three-dimensional model, where the base point Corresponding ground point The coordinates of can be directly read from the BIM 3D model, and the 3D coordinates of the remaining points Q are calculated by the formula:

[0115] ;

[0116] ;

[0117] in, is the BIM 3D coordinate of the ground point Q corresponding to the current origin P, The BIM three-dimensional coordinates of the top point Q corresponding to the current origin P;

[0118] X, Y are the BIM two-dimensional coordinates of the ground point Q corresponding to the origin P before the current origin P;

[0119] Z is the height of the ground point Q corresponding to the origin P before the current point P. If it is a ground point, it is the BIM floor elevation d. If it is the top point of the pole, it is the BIM floor height plus the pole length e.

[0120] is the moving distance compared to the current origin P;

[0121] It is the angle between the moving vector and the reference wall, which is used to determine the moving direction.

[0122] For example, as shown in Table 2,

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] Among them, if the current origin is The current origin Corresponding ground point BIM 3D coordinates, The top point corresponding to the current origin BIM 3D coordinates; X, Y are the current origin The previous origin Corresponding ground point BIM two-dimensional coordinates; Z is the current origin. The previous origin Corresponding ground point The height, if it is a ground point, it is the BIM floor height; if it is a pole point, it is the BIM floor height plus the pole length e;

[0128] If the current origin The current origin Corresponding ground point BIM 3D coordinates, The current origin The corresponding top point BIM 3D coordinates; X, Y are the current origin The previous origin Corresponding ground point BIM 2D coordinates; Z is the current origin The previous origin The height of the corresponding ground point. If it is a ground point, it is the BIM floor elevation d. If it is the top point of the pole, it is the BIM floor height plus the pole length e. Other And so on.

[0129] The calculated feature point coordinates are recorded in a table, as shown in Table 2:

[0130]

[0131] The calculated coordinates are recorded in a table, step 3.3, feature point coordinate recording table, which can be specifically shown in Table 3.

[0132]

[0133] Step 4: 2D coordinates of feature points in the calibrated camera image and the three-dimensional coordinates Q of the BIM three-dimensional model, and obtain the mapping relationship matrix between the calibrated camera image and the BIM three-dimensional model;

[0134] Step 4.1: Since camera lenses usually have distortion, and radial distortion is dominant, it is necessary to correct the distortion of the camera lens and set the radial distortion model as:

[0135] ;

[0136] where (x, y) is the undistorted normalized image coordinate, ( ) is the distorted image coordinate, is the radial distance, is the radial distortion coefficient;

[0137] Step 4.2: The internal reference calibration image obtained in step 2.2 is the calibrated camera image Input into the algorithm to obtain a square in the calibration plate The three-dimensional coordinates of , where s is the side length of a square on the chessboard, is the number of squares in the width direction of the chessboard, is the number of squares in the height direction of the chessboard, , is a positive integer;

[0138] Step 4.3: The camera image will be calibrated Convert to grayscale image to improve recognition rate;

[0139] Step 4.4: Traverse the grayscale image to find the corner points of each square in the calibration plate, and record the 2D image coordinates of each corner point , and add the 3D coordinates of each square in the calibration plate to the object point list ;

[0140] Step 4.5: Use the fisheye model to calibrate the camera through the calibrate function to calculate the camera's intrinsic parameter matrix K and distortion coefficient D:

[0141] ;

[0142] in, and is the focal length in the x and y directions, and are the optical center coordinates.

[0143] ;

[0144] in, is the radial distortion coefficient.

[0145] Step 4.6: Based on the radial distortion model, the intrinsic parameter matrix K and the distortion coefficient D, the distorted image coordinates are obtained. , to map the distorted image coordinates to the undistorted image coordinates:

[0146]

[0147] in, are the rectified undistorted image coordinates.

[0148] Output the corrected image for testing and get the following Figure 6 Check the corrected image to see if all the checkerboard grids in the image are straight lines.

[0149] Step 4.7: Based on the intrinsic parameter matrix K, distortion coefficient D and undistorted image coordinates , calculate the mapping matrix group of the undistorted feature points P and Q, and obtain the intrinsic parameter matrix of the calibration camera , rotation matrix and translation vectors ;

[0150] Specifically, the internal parameter matrix:

[0151] =644.9247154562587,0.0,0.0,0.0,638.3905994279746,0.0,943.5751663665941,576.1587253437231,1.0;

[0152] Rotation matrix:

[0153] =-0.9784997951843409,-0.1379467759240181,0.15332592030172662,-0.20621696222874442,0.6415091196914178,-0.738877942451858,0.003565853803968899,-0.7546103208709191,-0.6561635073072392;

[0154] Translation vector:

[0155] =17.745225770455487,-22.48965468175566,30.25380849346079.

[0156] Step 4.8: Combine the camera coordinates R ( ) and BIM floor elevation d, based on the internal parameter matrix , rotation matrix and translation vectors , and obtain the coordinate mapping relationship matrix.

[0157] Step 5: Capture and identify the face, and generate the movement trajectory of the corresponding person based on the face capture and recognition and coordinate mapping relationship matrix:

[0158] Step 5.1: Based on the image of the calibrated camera, capture the face to obtain the face capture record;

[0159] 1) When the face of a person q appears in the picture for the first time, a face capture record is captured, and all face capture records captured before the face leaves the picture are marked accordingly , q is the number of different people;

[0160] 2) When the face remains in the picture, a face capture record is captured every K seconds (usually 2 seconds);

[0161] 3) When the face leaves the screen, capture a face capture record and release the corresponding q of a certain person Mark; here, different people correspond to different q, different people's marks Different, through different To distinguish different people’s face capture records.

[0162] Step 5.2: Based on Retrieve all facial capture records of the same person within a certain period of time;

[0163] Step 5.3: Extract the two-dimensional coordinates of all the faces of the same person retrieved in the picture to form a two-dimensional coordinate group , through the coordinate relationship mapping matrix of step 4, calculate the three-dimensional coordinate group mapped to the BIM three-dimensional model ;

[0164] Step 5.4: Group the three-dimensional coordinates Pre-converted to , used to match gridded coordinates;

[0165] The conversion formula is:

[0166] ,

[0167] Step 5.5: Process the completed The gridded coordinate point set obtained in step 1.2 Match according to the minimum distance method and map it to the BIM 3D model ground to get the minimum distance point ;

[0168] Among them, the calculation formula for the minimum distance point is:

[0169] ,

[0170] The coordinate points are expressed as , The coordinate points are expressed as ;

[0171] Step 5.6: The minimum distance point is obtained According to the shortest path method, the final personnel movement trajectory is obtained , draw it in the BIM 3D model to get the final personnel trajectory.

[0172] In addition, the elevation of the detected person can be set. By adding the input face coordinates to the person's elevation, the position of the person's feet on the ground can be calculated to obtain the person's true location; here, the person's elevation is the person's height.

[0173] Among them, the coordinate mapping matrix is ​​used to calculate the three-dimensional spatial position, and then based on the three-dimensional spatial position and the elevation of the detected person, the position of the person's feet relative to the ground can be calculated.

[0174] According to another aspect of the present invention, there is further provided a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: execute any of the methods described above.

[0175] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0176] Processor; and

[0177] A memory arranged to store computer executable instructions, which when executed cause the processor to: perform any of the methods described above.

[0178] The present invention can reuse the existing fixed monitoring probes in the building, calibrate the internal and external parameters of the monitoring probes, and convert the two-dimensional coordinates of the personnel in the video stream into three-dimensional coordinates in the BIM three-dimensional model through spatial mapping, so as to realize the accurate positioning of the personnel and restore the movement trajectory of the personnel. The present invention uses the spatial data and structural information of BIM to accurately analyze and understand the environmental layout inside the building, thereby improving the positioning accuracy and efficiency of personnel, drawing movement trajectories, and realizing digital security management. Compared with the traditional positioning card or face capture machine solution, this method not only retains the low-cost advantage of the existing monitoring system, but also significantly improves the positioning accuracy and real-time tracking capability, saving deployment and maintenance costs. At the same time, coordinate conversion and trajectory generation are realized through software algorithms, which greatly improves management efficiency, and has broad application prospects and significant technical advantages.

[0179] The detailed contents of the device embodiments of the present invention can be found in the corresponding parts of the method embodiments, which will not be described in detail here.

[0180] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0181] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0182] In addition, a part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. The program instruction for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present invention, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present invention.

[0183] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A calibration method for virtual-real mapping between video stream and BIM model, characterized in that: include: Step 1: Build a BIM 3D model of the building; Step 2: Obtain the calibrated camera image calibrated by the camera's intrinsic parameters through a standard black and white checkerboard calibration plate; Step 3: Compare and map the feature points of the calibrated camera image with the BIM 3D model to obtain the 2D coordinates P of the feature points in the calibrated camera image and the 3D coordinates Q of the BIM 3D model; Step 4: Based on the two-dimensional coordinates P of the feature point in the calibrated camera image and the three-dimensional coordinates Q of the BIM three-dimensional model, a mapping relationship matrix between the calibrated camera image and the BIM three-dimensional model is obtained; Step 5: Capture and identify the face, and generate the movement trajectory of the corresponding person based on the face capture and recognition and coordinate mapping relationship matrix; Step 1: Build a BIM 3D model of the building, including: Step 1.1: Use BIM software to create a BIM three-dimensional model of the building, the BIM three-dimensional model of the building includes: the structure of the building, the internal layout and the fixed configuration objects in the room; obtain the BIM floor elevation d; add the position of the camera in the building to the refined BIM three-dimensional model, and determine the spatial information of each camera, including the camera coordinates R (X0, Y0, Z0) and the camera's orientation angle α0, and integrate the spatial information of each camera into the BIM three-dimensional model; Step 1.2: Based on the spatial information of the room walls and fixed configuration objects in the room in the BIM 3D model, determine the layout of each room, corridor and passage, as well as the distribution of stairs and elevators, and use gridding to establish the spatial path topology in the building to obtain the gridded coordinate point set coor[j]; Step 2: Obtain the calibrated camera image calibrated by the camera's intrinsic parameters through a standard black and white checkerboard calibration plate, including: Step 2.1: Make a standard black and white checkerboard calibration plate for internal reference calibration, where the side length of each small square grid on the standard black and white checkerboard calibration plate is defined as s, and there are a total of m×n small square grids; Step 2.2: Use the calibration plate with the standard black and white checkerboard to capture the camera intrinsic calibration image: place the calibration plate in the picture captured by the camera, and take calibration images in the corresponding i areas in the captured picture, wherein in each area, the grids of the calibration plate are made to occupy the corresponding area as much as possible under the premise that all of them are visible, and ensure that the grids of the calibration plate are clearly visible in the camera picture, and take k screenshots from each area as the calibrated camera pictures of the area, wherein the calibrated camera pictures of the five areas are Array[i], i=1, 2, 3, 4, 5 respectively.

2. The method for calibrating the virtual-real mapping of a video stream and a BIM model according to claim 1, characterized in that: Step 3: Compare and map the feature points of the calibrated camera image with the BIM 3D model to obtain the 2D coordinates P of the feature points in the calibrated camera image and the 3D coordinates Q of the BIM 3D model, including: Step 3.1: Based on the calibrated camera image and the refined BIM 3D model, obtain {P = P1, P2, P3, ..., P i ,…,P j ,…,P n } and the corresponding feature points {P = Q1, Q2, Q3, ..., Q i , …, Q j , …, Q n }; Among them, the definition of feature point is: point P i It is clearly visible in the calibrated camera image and can be found in the BIM 3D model. i The corresponding point Q i ; Step 3.2: If 8 or more pairs of feature points {P = P1, P2, P3, ..., P8, ...}, {Q = Q1, Q2, Q3, ..., Q8, ...} are found in the calibrated camera image and the refined BIM 3D model, directly extract the 2D coordinates P of all feature points in the calibrated camera image and the 3D coordinates Q of the BIM 3D model; Step 3.3: If only 7 or fewer pairs of feature points are found, select a pair of feature points from the 7 or fewer pairs of feature points and record them as the base points P O (X Po ,Y Po ), with base point P O Create feature points for center fit fixed calibration tools.

3. The method for calibrating the virtual-real mapping of a video stream and a BIM model as claimed in claim 2, characterized in that: Step 3.3: If only 7 or fewer pairs of feature points are found, select a pair of feature points from the 7 or fewer pairs of feature points and record them as the base points P O (X Po ,Y Po ), with base point P O Create feature points for center fit fixed calibration tools, including: 1) If only 7 or fewer pairs of feature points are found, select one pair of feature points from the 7 or fewer pairs of feature points and record them as the base points P O (X Po ,Y Po ); 2) Select a pole of fixed length e and place it vertically on the ground for calibration; 3) Based on the base point P O (X Po ,Y Po ) as the current origin, select the nearby wall as the reference surface, and select the current moving distance Δl1 outward at an angle β=n·90° relative to the reference surface. The ground point obtained is P 1_1 ; where n∈Z; 4) Place a pole of fixed length e perpendicular to the ground at point P 1_1 , define the top point of the pole as P 1_2 At the same time, a screenshot is taken from the calibrated camera image to obtain the first external parameter calibration image dir[0], where P 1_1 With P 1_2 ; 5) Take the newly acquired ground point P 1_1 As the current origin, repeat the above steps 3) and 4) at least four times. Similarly, obtain at least four pairs of point positions P with the current lengths Δl2, Δl3, and Δl4 respectively. 1_1 With P 1_2 , P 2_1 With P 2_2 , P 3_1 With P 3_2 , P 4_1 With P 4_2 , that is, at least get the base point P O A total of 9 points are included, and the corresponding external parameter calibration image set dir[i] is obtained, i = 0, 1, 2, 3, 4, where i = 0 is the base point P O images; 6) Obtain the base point P through the external reference calibration image set dir[i] O and the two-dimensional coordinates of other points P, and obtain the corresponding point Q in the BIM three-dimensional model, where the base point P O Corresponding ground point Q O The coordinates of can be directly read from the BIM 3D model, and the 3D coordinates of the remaining points Q are calculated by the formula: Q tools1 =(X Q +Δlcosβ—Δlysinβ,Y Q +Δlsinβ+Δlcosβ,Z Q ); Q tools2 =(X Q +Δlcosβ-Δlysinβ,Y Q +Δlsinβ+Δlcosβ,Z Q +e); Among them, Q tools1 is the BIM 3D coordinate of the ground point Q corresponding to the current origin P, Q tools2 The BIM three-dimensional coordinates of the top point Q corresponding to the current origin P; X, Y are the BIM two-dimensional coordinates of the ground point Q corresponding to the origin P before the current origin P; Z is the height of the ground point Q corresponding to the origin P before the current point P. If it is a ground point, it is the BIM floor elevation d. If it is the top point of the pole, it is the BIM floor height plus the pole length e. Δl is the moving distance compared to the current origin P; β is the angle between the moving vector and the reference wall, which is used to determine the moving direction.

4. The method for calibrating the virtual-real mapping of a video stream and a BIM model as claimed in claim 3, characterized in that: Step 4: Based on the two-dimensional coordinates P of the feature point in the calibrated camera image and the three-dimensional coordinates Q of the BIM three-dimensional model, the mapping relationship matrix between the calibrated camera image and the BIM three-dimensional model is obtained, including: Step 4.1: Correct the camera lens distortion and set the radial distortion model to: Where (x, y) is the undistorted normalized image coordinate, (x distorted ,y distorted ) is the distorted image coordinate, r 2 =x 2 +y 2 is the radial distance, k1, k2, k3 are the radial distortion coefficients; Step 4.2: Input the intrinsic calibration image obtained in step 2.2, that is, the calibrated camera image Array[i], into the algorithm to obtain the three-dimensional coordinates obj of a square p in the calibration plate p ={(i·s, j·s, 0)|i∈[0, width], j∈[0, height]}, where s is the side length of a square on the chessboard, width is the number of squares in the width direction of the chessboard, height is the number of squares in the height direction of the chessboard, and i and j are positive integers; Step 4.3: Convert the calibrated camera image Array[i] into a grayscale image; Step 4.4: Traverse the grayscale image to find the corner points of each square in the calibration plate, and record the two-dimensional image coordinates of each corner point image_points = {(x i，j ,y i，j )}, and add the 3D coordinates of each square in the calibration plate to the object point list obj_points = obj p ; Step 4.5: Use the fisheye model to calibrate the camera through the calibrate function to calculate the camera's intrinsic parameter matrix K and distortion coefficient D: Among them, f x and f y is the focal length in the x and y directions, c x and c y is the optical center coordinate; D=(k1 k2 k3 k4) T Among them, k1, k2, k3, k4 are radial distortion coefficients; Step 4.6: Based on the radial distortion model, the intrinsic parameter matrix K and the distortion coefficient D, the distorted image coordinates (x undistorted ,y undistorted ) to map the distorted image coordinates to the undistorted image coordinates: Among them, (x undistorted ,y undistorted ) is the coordinate of the rectified undistorted image; Step 4.7: Based on the intrinsic parameter matrix K, distortion coefficient D and undistorted image coordinates (x undistorted ,y undistorted ), calculate the mapping matrix group of the undistorted feature points P and Q, and obtain the intrinsic parameter matrix MtxK, rotation matrix MtxR and translation vector VecT of the calibration camera; Step 4.8: Combine the camera coordinates R (X0, Y0, Z0) and the BIM floor elevation d, and obtain the coordinate mapping relationship matrix based on the intrinsic parameter matrix MtxX, the rotation matrix MtxR and the translation vector VecT.

5. The method for calibrating the virtual-real mapping of a video stream and a BIM model as claimed in claim 4, characterized in that: Step 5: Capture and identify the face, and generate the movement trajectory of the corresponding person based on the face capture and recognition and coordinate mapping relationship matrix, including: Step 5.1: Based on the image of the calibrated camera, capture the face and obtain the face capture record; Step 5.2: Based on F q Retrieve all facial capture records of the same person within a certain period of time; Step 5.3: Extract the two-dimensional coordinates of all retrieved faces of the same person in the picture to form a two-dimensional coordinate group face[i], and calculate the three-dimensional coordinate group path[i] mapped to the BIM three-dimensional model through the coordinate relationship mapping matrix of step 4; Step 5.4: Pre-convert the three-dimensional coordinate group path[i] into path′[i] for matching the gridded coordinates; The conversion formula is: Step 5.5: Match the processed path′[i] with the grid coordinate point set coor[j] obtained in step 1.2 according to the minimum distance method, map it to the ground of the BIM 3D model, and obtain the minimum distance point coor′[j]; The calculation formula for the minimum distance point is: The coordinate point of path′[i] is expressed as (x i ,y i , 0), the coordinate point of coor[j] is expressed as (x gj ,y gj , z gj ); Step 5.6: The obtained minimum distance point coor′[j] is sequentially processed according to the shortest path method to obtain the final personnel movement trajectory path″[i], which is then plotted in the BIM 3D model to obtain the final personnel trajectory.

6. The method for calibrating the virtual-real mapping of a video stream and a BIM model as claimed in claim 5, characterized in that: Step 5.1: Based on the image of the calibrated camera, capture the face and obtain the face capture record, including: 1) When the face of a person q appears in the picture for the first time, a face capture record is captured, and all face capture records captured before the face leaves the picture are marked with a mark F q , q is the number of different people; 2) When the face remains in the picture, a face capture record is captured every preset seconds; 3) When the face leaves the screen, capture a face capture record and release the F corresponding to a certain person q q mark.

7. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to: perform the method according to any one of claims 1 to 6.

8. A computer device, wherein: include: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to: perform a method as claimed in any one of claims 1 to 6.

Images