Airport terminal video digital twinning method, device, equipment and medium

Abstract

The invention provides an airport terminal video digital twinning method, device, equipment and medium, and the method comprises the steps: firstly obtaining data collected by a plurality of sensors, then constructing a high-precision airport terminal static three-dimensional model, and taking a world coordinate system as a reference, a mapping relation among a two-dimensional image coordinate system, an airport terminal static three-dimensional model coordinate system and a camera coordinate system is constructed, an airport terminal space initial layout is obtained, two-dimensional layout estimation is used as a key point, image textures are reconstructed, registration from the two-dimensional image textures to an airport terminal static three-dimensional model under a single camera is achieved, and the registration result is obtained. And finally, calculating a suture line in a fusion rendering process, and fusing a multi-camera two-dimensional image with the constructed static three-dimensional model to realize the three-dimensional panoramic digital twinning of the terminal building. Through the scheme of the invention, three-dimensional live-action visualization under the fusion of multiple cameras, structured data and the terminal indoor model in a large scene of the terminal is realized, comprehensive real-time monitoring is realized, and safe operation of an airport is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an airport terminal video digital twin method, device, equipment and medium. Background Art

[0002] Video digital twin is an emerging technology in the past two years. It mainly solves the mapping between spatial data and real space, and solves the problem that when staff watch massive videos at the same time, the spatial positions of various facilities / objects are difficult to reflect, and abnormal events are difficult to detect. At present, airports mainly monitor events through panoramic stitching and manual polling of videos. This method is difficult to monitor and discover various events in a timely and effective manner, and to associate physical locations at the same time. Existing solutions mainly focus on the following points:

[0003] (1) Monitor the entire terminal scene through panoramic stitching;

[0004] (2) Analyze single-camera scenes through video analysis;

[0005] (3) Linked viewing through the combination of camera and GIS (geographic information system);

[0006] (4) Digital twin method.

[0007] The above four mainstream solutions have the following problems:

[0008] (1) The first solution requires that the parallax between the images to be stitched should not be too large and that there should be a certain overlap between the two frames. However, the airport terminal has a complex structure and there are problems of field of view obstruction and large parallax between surveillance cameras. Therefore, this method is not applicable to indoor surveillance scenarios in the terminal.

[0009] (2) The second solution can monitor passenger behavior with a single camera, but it is difficult to implement multiple cameras to connect and monitor events in the area;

[0010] (3) Although the third solution associates the surveillance camera with the real geographic location, it is limited to the display of a single camera and cannot be linked with multiple cameras;

[0011] (4) The fourth digital twin method is mainly based on static modeling, but it cannot achieve real-time modeling of dynamic scenes, and in the twin process, it relies on the completeness and accuracy of the collected data. Summary of the invention

[0012] In response to the above problems, the present invention proposes a method, device, equipment and medium for digital twinning of airport terminal video, which uses multiple sensors to reconstruct the three-dimensional model of the static model of the large terminal, establishes a mapping relationship between the two-dimensional image and the three-dimensional model, and realizes the registration of the two-dimensional image texture to the three-dimensional model. Finally, it performs multi-camera fusion and splicing, and maps the structured data based on video analysis to realize real-time three-dimensional panoramic monitoring of the large terminal scene.

[0013] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0014] The data involved in the entire solution include: image data collected by RGB-D (depth camera) sensors, point cloud data collected by LIDAR (laser radar scanner) sensors, video data collected by surveillance cameras, airport CAD (computer-aided design) drawings, and camera installation parameters (including camera internal parameters and installation location).

[0015] In a first aspect, the present invention provides a method for video digital twinning of an airport terminal, the method comprising the following steps:

[0016] Step S1, data acquisition, acquiring image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera;

[0017] Step S2, three-dimensional reconstruction of the static scene of the terminal, constructing a high-precision three-dimensional model of the static scene of the airport terminal based on the image data, point cloud data and video data, including using the position and posture image data of the RGB-D sensor to build a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine three-dimensional model of the indoor static digital surface of the terminal;

[0018] Step S3, registering the two-dimensional image texture to the three-dimensional model under a single camera, registering the two-dimensional image texture to the fine three-dimensional model of the terminal's indoor static digital surface under a single camera for the video data collected by the camera and the fine three-dimensional model of the terminal's indoor static digital surface, including constructing a mapping relationship between the two-dimensional image, the fine three-dimensional model of the terminal's indoor static digital surface, the camera coordinate system and the world coordinate system, and then obtaining the initial layout of the terminal space according to the initial position of the monitoring camera, solving the rotation matrix of the monitoring camera, updating the terminal's indoor space layout, estimating key points according to the two-dimensional layout, performing image texture division, reconstructing the image texture, and projecting it to the three-dimensional model of the terminal's indoor static digital surface;

[0019] Step S4, the fusion of multiple cameras and the constructed static digital surface 3D model, includes screening the terminal cameras, calculating the initial layout estimate corresponding to each camera, determining whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, and fusing different textures according to the viewing angle difference to realize the terminal video digital twin.

[0020] In a second aspect, the present invention provides an airport terminal video digital twin device, comprising: a data acquisition module, a construction module, a registration module, and a fusion module;

[0021] The data acquisition module is used to acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera;

[0022] The construction module is used for three-dimensional reconstruction of the static scene of the terminal. According to the image data, point cloud data and video data, a high-precision three-dimensional model of the static scene of the airport terminal is constructed, including using the position and posture image data of the RGB-D sensor to construct a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine three-dimensional model of the indoor static digital surface of the terminal;

[0023] The registration module is used for registering the two-dimensional image texture to the three-dimensional model under a single camera. For the video data collected by the camera and the fine three-dimensional model of the indoor static digital surface of the terminal, the two-dimensional image texture under a single camera is registered to the fine three-dimensional model of the indoor static digital surface of the terminal, including constructing a mapping relationship between the two-dimensional image, the fine three-dimensional model of the indoor static digital surface of the terminal, the camera coordinate system and the world coordinate system, obtaining the initial layout of the terminal space according to the mapping relationship, estimating the key points according to the two-dimensional layout, reconstructing the image texture, and projecting it to the three-dimensional model of the indoor static digital surface of the terminal;

[0024] The fusion module is used to fuse multiple cameras with the constructed static digital surface three-dimensional model, including screening the terminal cameras, calculating the initial layout estimate corresponding to each camera, judging whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, and fusing different textures according to the viewing angle difference to realize the terminal video digital twin.

[0025] Furthermore, the building blocks are specifically used for:

[0026] Construct a high-precision static 3D model of the terminal, including collecting RGB-D depth images, performing denoising on the depth images, and removing outliers;

[0027] For the image data depth map, a frame model tracking method is used to perform posture estimation, and the initial data frame depth map is selected as valid data to participate in the reconstruction of the terminal static scene 3D model. The corresponding points between the selected data frame and the remaining frames are identified, and the error measurement is performed. The incremental transformation with the previous input frame depth map is estimated, and the estimation is connected to the previous posture estimation result to obtain the terminal static scene model.

[0028] The point cloud data and RGB-D image data collected by the LIDAR sensor are used for registration to obtain the texture information of the three-dimensional point cloud data. The structured information of the terminal plane and the edge of the plane is obtained through point cloud segmentation. The corresponding objects are identified using deep learning methods to realize the construction of component information in the terminal. The stereo image pair point cloud and the point cloud data are fused according to the sensor position information to realize the construction of a complete and detailed static digital surface model of the terminal interior.

[0029] Further, the registration module includes a coordinate system association relationship construction submodule, a two-dimensional scene layout estimation submodule, a two-dimensional scene layout estimation update submodule and a projection submodule;

[0030] The coordinate system association relationship construction submodule is used to construct the mapping relationship between the coordinate system of the indoor static digital surface 3D model of the terminal building and the world coordinate system, the camera coordinate system and the world coordinate system, and the camera coordinate system and the image coordinate system based on the world coordinate system;

[0031] A two-dimensional scene layout estimation submodule is used to annotate the three-dimensional model layout of the indoor static digital surface of the terminal in the world coordinate system, analyze the effective distance of the camera, obtain the scene layout under the effective distance of the camera, and project and obtain a preliminary estimate of the two-dimensional scene layout of the camera based on the scene layout;

[0032] The 2D scene layout estimation and update submodule is used to calculate the rotation matrix when the camera rotates, and to update the initial layout according to the rotation matrix;

[0033] The projection submodule is used to project the two-dimensional scene onto the three-dimensional model of the static digital surface of the terminal building. It includes dividing the texture area of ​​the video image based on the two-dimensional layout nodes, performing affine transformation and reconstruction on the image texture of each area, building the relationship between the two-dimensional initial layout and the scene layout, and projecting it into the three-dimensional model.

[0034] Furthermore, the fusion module is specifically used for:

[0035] Different perspective differences are processed differently during fusion: For image areas with large perspective differences, directly select The video image texture corresponding to the smaller surveillance camera is overlaid; image fusion is performed on the image area with small viewing angle difference; for the image fusion, the stitching line in the fusion rendering process is calculated, and image fusion is performed based on the stitching line to realize the video digital twin of the entire scene of the terminal.

[0036] In a third aspect, the present invention provides an electronic device, comprising: at least one processor and a memory;

[0037] The memory stores computer-executable instructions;

[0038] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the computing method as described in any one of the first aspects.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the computing method described in any one of the first aspects.

[0040] The present invention combines the construction of a static three-dimensional model of the airport terminal with video analysis to achieve accurate monitoring of all elements of the airport, while also achieving real-time monitoring of the entire terminal scene. Specifically, it is based on multi-source point cloud data and video data obtained by RGB-D and LIDAR sensors, and by constructing a mapping relationship between the real world, three-dimensional models, cameras, and two-dimensional images, with a single-camera local dynamic scene three-dimensional reconstruction technology as the core, to achieve three-dimensional real-scene visualization in a large scene that integrates multiple cameras, structured data, and indoor models of the terminal, and to combine digital space with real space to achieve comprehensive real-time monitoring of the terminal in a large scene, ultimately allowing airport staff to "see quickly", "understand", and "see clearly" in massive scattered monitoring videos. Compared with the prior art, the present invention has the following beneficial technical effects:

[0041] (1) Based on the construction of the mapping relationship between physical space, two-dimensional space, and world space, accurate estimation of indoor layout structure is achieved without depth information;

[0042] (2) By fusing real-time video images with the static three-dimensional model of the terminal, real-time twinning of digital space and real space is achieved, and comprehensive real-time monitoring of the terminal is realized, providing strong support for the safe operation of the airport. This method can also be extended to the airport public areas and flight areas, and is also suitable for complex indoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a business process diagram of an airport terminal video digital twin method provided by an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of a local three-dimensional model constructed according to an embodiment of the present invention;

[0046] Figure 3 This is a panoramic real-time monitoring example after the fusion of multiple cameras and a three-dimensional static scene model provided by an embodiment of the present invention;

[0047] Figure 4 It is a structural schematic diagram of an airport terminal video digital twin device provided by an embodiment of the present invention;

[0048] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The following is a detailed application description of the present invention based on the content of the present invention and combined with the actual scene of the airport terminal. The present invention provides a video digital twin method of an airport terminal, such as Figure 1 As shown, the method comprises the following steps:

[0051] (1) Data acquisition

[0052] Acquire image data collected by RGB-D sensors, point cloud data collected by LIDAR sensors, and video data collected by surveillance cameras.

[0053] (2) Based on the characteristics of the terminal scene, a detailed 3D model of the terminal interior static digital surface is constructed.

[0054] This includes using the image data of the RGB-D sensor’s position and posture to construct a static scene 3D model of the airport terminal, fusing the constructed static scene 3D model with the point cloud data collected by the LIDAR sensor, performing semantic segmentation on the point cloud data, and constructing the terminal components, thereby achieving the construction of a detailed static digital surface 3D model of the terminal interior. The details are as follows:

[0055] The static scene of the terminal is reconstructed in three dimensions based on the data collected by the RGB-D sensor. The normal of each point is calculated using a bilateral filter and finite difference, and the depth map of the RGB-D image data is denoised and outliers are removed.

[0056] For the acquired image data depth map, the frame model tracking method is used for posture estimation. The initial data frame depth map is collected as the input for reconstructing the static scene 3D model of the terminal. The corresponding points between the selected data frame and the remaining frames are identified, and the error measurement is performed. The incremental transformation with the previous input frame depth map is estimated, and the estimation is connected to the previous posture estimation result, thereby reducing the RGB-D local trajectory error accumulation problem. In this way, the static digital surface 3D model of the terminal is obtained.

[0057] Based on the static 3D model of the terminal that has been constructed, the static texture model of the terminal interior is constructed by fusing the stereo image pair of video data with the acquired point cloud data. The LIDAR sensor is used to obtain the 3D point cloud data of the complex texture scene of the terminal, and the texture information of the 3D point cloud data of the laser scanning is obtained by registering the image data with the 3D point cloud data. The stereo image pair point cloud and the 3D laser scanning point cloud data are fused according to the sensor position information to build the model. The registration is based on geometric transformation, and the purpose is to align the point cloud data with the image.

[0058] The acquired point cloud data is semantically segmented to obtain structured information such as the plane and edge of the terminal building. The corresponding objects are identified using deep learning methods to build component information in the terminal building (such as counters, etc.). Finally, a fine digital 3D model of the static scene in the terminal building is built. The constructed local 3D model is as follows: Figure 2 shown.

[0059] (3) Registration of 2D image texture to 3D model using a single camera

[0060] It includes constructing a two-dimensional image, a detailed three-dimensional model of the terminal's indoor static digital surface, and the mapping relationship between the camera coordinate system and the world coordinate system. According to the initial position of the surveillance camera, the initial layout of the terminal space is obtained, the rotation matrix of the surveillance camera is solved, and the indoor space layout of the terminal is updated. According to the two-dimensional layout, the key points are estimated, the image texture is divided, and the image texture is reconstructed and projected onto the three-dimensional model of the terminal's indoor static digital surface. The details are as follows:

[0061] In world coordinate system As a benchmark, a three-dimensional model coordinate system of the static digital surface of the terminal is established. And the surveillance camera coordinate system , Image coordinate system The mapping relationship between them;

[0062] Specifically, under the premise that the static digital surface 3D model of the terminal is scaled in proportion to the real scene, the scale parameter s is determined by the scaling ratio. For the constructed static digital surface 3D model of the terminal, a static digital surface model coordinate system is established with its center as the origin , let the coordinates of each vertex in the model be the set , where n is a natural number. Similarly, construct the world coordinate system (The z-axis is the vertical direction of the ground). The mapping relationship between the two coordinate systems is shown in the following formula: ;

[0063] The origin of the coordinate system is selected so that its position in the three-dimensional model of the static digital surface of the terminal is consistent with its relative position in the real world. , ,in, is the identity matrix, from which we can calculate: ;

[0064] Surveillance Cameras In the world coordinate system The position below is It represents the coordinate value of the point in the constructed coordinate system. Represents the actual location of the surveillance camera on the plane of the airport terminal. Indicates the actual installation height of the surveillance camera.

[0065] Command surveillance camera The initial pose in the world coordinate system is , and Represents surveillance cameras Relative to the world coordinate system The rotation matrix and displacement vector of the origin, surveillance camera Corresponding coordinate system and the world coordinate system The mapping relationship between the two is expressed by the following formula: ;

[0066] From the surveillance camera coordinate system The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix of the surveillance camera Determine the intrinsic parameter matrix of the surveillance camera Obtained by monitoring camera calibration, monitoring camera coordinate system and the image coordinate system The mapping relationship is: ,in, It is the depth of the corresponding node in the surveillance camera coordinate system, thereby realizing the transformation and mapping relationship between the terminal indoor static digital surface model coordinate system, the world coordinate system, the surveillance camera coordinate system and the image coordinate system.

[0067] When the surveillance camera rotates, the displacement vector , we solve the rotation matrix of the surveillance camera by the following formula: ,in, is a set of three-dimensional points that satisfy geometric constraints, R is the rotation matrix, and || || represents the norm of the matrix.

[0068] The overall layout of the static digital surface model of the terminal building based on the world coordinate system is marked as follows: , where n is a natural number, Represented in the form of point pairs, Represents layout line segments The starting point and end point of For surveillance cameras The effective distance that can be analyzed is the monitoring camera Analyze scene layout within a certain distance for:

[0069]

[0070] in, is a unit vector representing the surveillance camera The main axis direction, is the step size of the search area, For all layout segments analyzed, surveillance cameras The corresponding two-dimensional layout in the image coordinate system Can be Projection is performed to obtain is the depth of the corresponding node in the surveillance camera coordinate system: ;

[0071] When the surveillance camera's viewing angle rotates, the layout is updated through the rotation matrix R.

[0072] Acquisition-based 2D layout , divide the video image into texture regions, and perform affine transformation and reconstruction on the image texture of each region. The following formula is used to establish and The relationship between them is projected into a three-dimensional model, where Represents the layout in the coordinate system of the static digital surface 3D model of the terminal: .

[0073] (4) Multi-camera video images are integrated with the static digital surface model of the terminal to achieve panoramic real-time monitoring of the terminal

[0074] The fusion of multiple cameras and 3D models includes screening the cameras of the terminal, calculating the initial layout estimation corresponding to each camera, judging whether there is overlap between the cameras, and calculating the viewing angle difference between the cameras. The size of the viewing angle difference is related to the focal length of the camera, which represents the size of the illumination range of the surveillance camera. Then, different texture fusion methods are used according to the viewing angle difference to realize the digital twin of the full-scene video of the terminal, as follows:

[0075] In the terminal scene, there will be a phenomenon that multiple surveillance cameras cover the same area, so it is necessary to perform multi-camera multi-view image fusion. The present invention performs different processing on different viewing angle differences when implementing fusion: for image areas with large viewing angle differences, directly select The video image texture corresponding to the smaller surveillance camera is covered; for the image area with small viewing angle difference, the image stitching line is used as shown in the following formula (1) for image splicing, and the image fusion method is shown in formula (2). The stitching line calculation and fusion formula in the fusion rendering process are as follows:

[0076] (1)

[0077] (2)

[0078] Among them, p and q are two adjacent output pixels in the overlapping area. , Indicates that point p is in the image and images Colors in , Indicates that point q is in the image and images Color, , Represents surveillance cameras and In the image frame at the same time, RGB represents the image format of RGB, α ​​is the calculation parameter, 0<α<1, and the panoramic real-time monitoring image after the multi-camera video image is fused with the static digital surface 3D model of the terminal building is shown in Figure 2. Figure 3 shown.

[0079] Figure 4 A schematic diagram of the structure of an airport terminal video digital twin device provided by an embodiment of the present invention, such as Figure 4 As shown, the device of this embodiment may include: a data acquisition module, a construction module, a registration module and a fusion module;

[0080] The data acquisition module is used to acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the camera;

[0081] The construction module is used for three-dimensional reconstruction of the static scene of the terminal. According to the image data, point cloud data and video data, a high-precision three-dimensional model of the static scene of the airport terminal is constructed, including using the position and posture image data of the RGB-D sensor to construct a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine static digital surface model of the terminal indoors;

[0082] The registration module is used for registering the two-dimensional image texture to the three-dimensional model under a single camera. For the video data collected by the camera and the fine indoor static digital surface model of the terminal, the two-dimensional image texture under a single camera is registered to the fine indoor static digital surface model of the terminal, including constructing a mapping relationship between the two-dimensional image, the fine indoor static digital surface three-dimensional model of the terminal, and the camera coordinate system and the world coordinate system, obtaining the initial layout of the terminal space according to the initial position of the monitoring camera, solving the rotation matrix of the monitoring camera, updating the indoor space layout of the terminal, estimating key points according to the two-dimensional layout, dividing the image texture, reconstructing the image texture, and projecting it to the indoor static digital surface three-dimensional model of the terminal;

[0083] The fusion module is used to fuse multiple cameras and build a high-precision static digital surface three-dimensional model, including screening the terminal cameras, calculating the initial layout estimate corresponding to each camera, determining whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, and fusing different textures according to the viewing angle difference to realize the terminal video digital twin.

[0084] Optionally, a construction module specifically includes denoising the depth map of the collected RGB-D data and removing outliers, using a bilateral filter and finite difference to calculate the normal of each point, and performing denoising on the depth map; using a frame model tracking method to estimate the posture of the surveillance camera, selecting the collected initial data frame as valid data to participate in the reconstruction of the static three-dimensional model of the terminal, identifying corresponding points between the selected frame and the remaining frames, performing error measurement, estimating the incremental transformation with the previous input frame, and connecting the estimation to the previous posture estimation result, thereby realizing the reconstruction of the static three-dimensional model in the large scene of the terminal;

[0085] Based on the static 3D model of the terminal, the static texture model of the terminal is constructed by fusing the stereo image pair of video data with the acquired point cloud data. The LIDAR sensor is used to obtain the 3D point cloud data of the terminal's complex texture scene, and the texture information of the laser scanning 3D point cloud data is obtained by registering the image data with the 3D point cloud data. The stereo image pair point cloud and the 3D laser scanning point cloud data are fused according to the sensor position information to build the model.

[0086] Semantic segmentation is performed on the acquired point cloud data to obtain structured information such as the terminal plane and plane edges, and the corresponding objects are identified using deep learning methods to build component information in the terminal (such as counters, etc.), and finally complete the construction of a detailed digital 3D model of the static scene in the terminal.

[0087] Optionally, a registration module is used to register As a benchmark, a static digital surface model coordinate system of the terminal is established. And the surveillance camera coordinate system , Image coordinate system Mapping relationship between: For the constructed static digital surface 3D model of the terminal, the 3D model coordinate system is established with its center as the origin , let the coordinates of each vertex in the model be the set , where n is a natural number. Similarly, construct the world coordinate system (The z-axis is the vertical direction of the ground). The mapping relationship between the two coordinate systems is shown in the following formula: ;

[0088] The origin of the coordinate system is selected so that its position in the three-dimensional model of the static digital surface of the terminal is consistent with its relative position in the real world. , ,in, is the identity matrix, from which we can calculate: ;

[0089] Surveillance Cameras In the world coordinate system The position below is It represents the coordinate value of the point in the constructed coordinate system. Represents the actual location of the surveillance camera on the plane of the airport terminal. Indicates the actual installation height of the surveillance camera.

[0090] Command surveillance camera The initial pose in the world coordinate system is , and Represents surveillance cameras Relative to the world coordinate system The rotation matrix and displacement vector of the origin, surveillance camera Corresponding coordinate system and the world coordinate system The mapping relationship between the two is expressed by the following formula: ;

[0091] From the surveillance camera coordinate system The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix of the surveillance camera Determine the intrinsic parameter matrix of the surveillance camera Obtained by monitoring camera calibration, monitoring camera coordinate system and the image coordinate system The mapping relationship is: ,in, It is the depth of the corresponding node in the surveillance camera coordinate system, thereby realizing the transformation and mapping relationship between the terminal indoor static digital surface model coordinate system, the world coordinate system, the surveillance camera coordinate system and the image coordinate system.

[0092] When the surveillance camera rotates, the displacement vector , we solve the rotation matrix of the surveillance camera by the following formula: ,in, is a set of three-dimensional points that satisfy geometric constraints, R is the rotation matrix, and || || represents the norm of the matrix.

[0093] The overall layout of the static digital surface model of the terminal building based on the world coordinate system is marked as follows: , where n is a natural number, Represented in the form of point pairs, Represents layout line segments The starting point and end point of For surveillance cameras The effective distance that can be analyzed is the monitoring camera Analyze scene layout within a certain distance for:

[0094]

[0095] in, is a unit vector representing the surveillance camera The main axis direction, is the step size of the search area, For all layout segments analyzed, surveillance cameras The corresponding two-dimensional layout in the image coordinate system Can be Projection is performed to obtain is the depth of the corresponding node in the surveillance camera coordinate system: ;

[0096] When the surveillance camera's viewing angle rotates, the layout is updated through the rotation matrix R.

[0097] Acquisition-based 2D layout , divide the video image into texture regions, and perform affine transformation and reconstruction on the image texture of each region. The following formula is used to establish and The relationship between them is projected into a three-dimensional model, where Represents the layout in the coordinate system of the static digital surface 3D model of the terminal: .

[0098] Optionally, a fusion module is used to process different perspective differences differently during fusion: In the terminal scene, multiple surveillance cameras may cover the same area, so multi-camera multi-perspective image fusion is required. For image areas with large perspective differences, directly select The video image texture corresponding to the smaller surveillance camera is overlaid; for the image area with small viewing angle difference, the seam line calculation and image fusion formulas in the fusion rendering process are: , , where p and q are two adjacent output pixels in the overlapping area, , Indicates that point p is in the image and images Colors in , Indicates that point q is in the image and images Color, , Represents surveillance cameras and In the image frame at the same time, RGB represents that the image is in RGB format, α is a calculation parameter, 0<α<1.

[0099] The device of this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle is similar and will not be repeated here.

[0100] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the electronic device of this embodiment may include: a memory and a processor.

[0101] A memory, used to store computer programs (such as an application program, a functional module, etc. for implementing the above-mentioned airport terminal video digital twin method), computer instructions, etc.;

[0102] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0103] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment.

[0104] For details, please refer to the relevant description in the previous method embodiment.

[0105] The memory and the processor may be independent structures or integrated structures. When the memory and the processor are independent structures, the memory and the processor may be coupled and connected via a bus.

[0106] In addition, an embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When at least one processor of a user device executes the computer-executable instructions, the user device executes the above-mentioned various possible methods.

[0107] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0108] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video digital twin method for airport terminals, It is characterized in that The method comprises the following steps: Step S1, data acquisition, acquiring image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera; Step S2, three-dimensional reconstruction of the static scene of the terminal, constructing a high-precision three-dimensional model of the static scene of the airport terminal based on the image data, point cloud data and video data, including using the position and posture image data of the RGB-D sensor to build a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine three-dimensional model of the indoor static digital surface of the terminal; Step S3, registering the two-dimensional image texture to the three-dimensional model under a single camera, registering the two-dimensional image texture to the fine three-dimensional model of the terminal's indoor static digital surface under a single camera for the video data collected by the camera and the fine three-dimensional model of the terminal's indoor static digital surface, including constructing a mapping relationship between the two-dimensional image, the fine three-dimensional model of the terminal's indoor static digital surface, the camera coordinate system and the world coordinate system, and then obtaining the initial layout of the terminal space according to the initial position of the monitoring camera, solving the rotation matrix of the monitoring camera, updating the terminal's indoor space layout, estimating key points according to the two-dimensional layout, performing image texture division, reconstructing the image texture, and projecting it to the three-dimensional model of the terminal's indoor static digital surface; Step S4, the fusion of multiple cameras and the constructed static digital surface 3D model, includes screening the terminal cameras, calculating the initial layout estimate corresponding to each camera, determining whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, and fusing different textures according to the viewing angle difference to realize the terminal video digital twin.

2. The method according to claim 1, It is characterized in that The step S2 specifically includes: Acquire RGB-D image data as data for constructing a static scene model of the terminal, use a bilateral filter and finite difference to calculate the normal of each point, and perform depth map denoising and outlier removal processing on the RGB-D image data; For the image data depth map, use the frame model tracking method to perform posture estimation, select the initial data frame depth map as valid data to participate in the reconstruction of the terminal static scene 3D model, identify the corresponding points between the selected data frame and the remaining frames, perform error measurement, estimate the incremental transformation with the previous input frame depth map, connect the estimation to the previous posture estimation result, and obtain the terminal static scene model; The point cloud data and RGB-D image data collected by the LIDAR sensor are used for registration to obtain the texture information of the three-dimensional point cloud data. The structured information of the terminal plane and the edge of the plane is obtained through point cloud segmentation. The corresponding objects are identified using deep learning methods to realize the construction of component information in the terminal. The stereo image pair point cloud and the point cloud data are fused according to the sensor position information to realize the construction of a complete and detailed static digital surface model of the terminal interior.

3. The method according to claim 1, It is characterized in that The step S3 specifically includes: Constructing the world coordinate system , Terminal static digital surface 3D model coordinate system And the surveillance camera coordinate system , Image coordinate system The mapping relationship between them is: The transformation relationship between the three-dimensional model coordinate system of the terminal static digital surface and the world coordinate system is ,in, , , is the identity matrix, from which we can calculate: ; Command surveillance camera The initial pose in the world coordinate system is , and Represents surveillance cameras Relative to the world coordinate system The rotation matrix and displacement vector of the origin, surveillance camera Corresponding coordinate system and the world coordinate system The mapping relationship between the two is expressed by the following formula: ; From the surveillance camera coordinate system The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix of the surveillance camera Determine the intrinsic parameter matrix of the surveillance camera Obtained by monitoring camera calibration, monitoring camera coordinate system and the image coordinate system The mapping relationship is: ,in, is the depth of the corresponding node in the surveillance camera coordinate system; When the surveillance camera rotates, the displacement vector , solve the rotation matrix of the surveillance camera by the following formula: ,in, is a set of three-dimensional points that satisfy geometric constraints, R is the rotation matrix, and || || represents the norm of the matrix; The overall layout of the static digital surface model of the terminal building based on the world coordinate system is marked as follows: , where n is a natural number, Represented in the form of point pairs, Represents layout line segments The starting and ending points of For surveillance cameras The effective distance that can be analyzed is the monitoring camera Analyze scene layout within a certain distance for: ； in, is a unit vector representing the surveillance camera The main axis direction, is the step size of the search area, For all layout segments analyzed, surveillance cameras The corresponding two-dimensional layout in the image coordinate system pass Projection is performed to obtain is the depth of the corresponding node in the surveillance camera coordinate system: ; When the surveillance camera's viewing angle rotates, the layout is updated through the rotation matrix R; Acquisition-based 2D layout , divide the video image into texture regions, and perform affine transformation and reconstruction on the image texture of each region. The following formula is used to establish and The relationship between Represents the layout in the coordinate system of the static digital surface 3D model of the terminal: .

4. The method according to claim 3, It is characterized in that The step S4 specifically includes: Different perspective differences are processed differently during fusion: For image areas with large perspective differences, directly select The video image texture corresponding to the smaller surveillance camera is overlaid; for the image area with small viewing angle difference, image fusion is performed, and for the image fusion, the stitching line in the fusion rendering process is calculated: , and perform image fusion based on the stitching line , where p and q are two adjacent output pixels in the overlapping area, , Indicates that point p is in the image and images Colors in , Indicates that point q is in the image and images Color, , Represents the camera and In the image frame at the same moment, RGB represents the image in RGB format, α is the calculation parameter, 0<α<1, thereby realizing the video digital twin of the entire scene of the terminal.

5. An airport terminal video digital twin device, It is characterized in that include: Data acquisition module, construction module, registration module and fusion module; The data acquisition module is used to acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera; The construction module is used for three-dimensional reconstruction of the static scene of the terminal. According to the image data, point cloud data and video data, a high-precision three-dimensional model of the static scene of the airport terminal is constructed, including using the position and posture image data of the RGB-D sensor to construct a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine three-dimensional model of the indoor static digital surface of the terminal; The registration module is used for registering the two-dimensional image texture to the three-dimensional model under a single camera. For the video data collected by the camera and the fine three-dimensional model of the indoor static digital surface of the terminal, the two-dimensional image texture under a single camera is registered to the fine three-dimensional model of the indoor static digital surface of the terminal, including constructing a mapping relationship between the two-dimensional image, the fine three-dimensional model of the indoor static digital surface of the terminal, the camera coordinate system and the world coordinate system, obtaining the initial layout of the terminal space according to the mapping relationship, estimating the key points according to the two-dimensional layout, reconstructing the image texture, and projecting it to the three-dimensional model of the indoor static digital surface of the terminal; The fusion module is used to fuse multiple cameras with the constructed static digital surface three-dimensional model, including screening the terminal cameras, calculating the initial layout estimate corresponding to each camera, judging whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, and fusing different textures according to the viewing angle difference to realize the terminal video digital twin.

6. The device according to claim 5, It is characterized in that The building blocks are specifically used for: Construct a high-precision static 3D model of the terminal, including collecting RGB-D depth images, performing denoising on the depth images, and removing outliers; For the image data depth map, use the frame model tracking method to perform posture estimation, select the initial data frame depth map as valid data to participate in the reconstruction of the terminal static scene 3D model, identify the corresponding points between the selected data frame and the remaining frames, perform error measurement, estimate the incremental transformation with the previous input frame depth map, connect the estimation to the previous posture estimation result, and obtain the terminal static scene model; The point cloud data and RGB-D image data collected by the LIDAR sensor are used for registration to obtain the texture information of the three-dimensional point cloud data. The structured information of the terminal plane and the edge of the plane is obtained through point cloud segmentation. The corresponding objects are identified using deep learning methods to realize the construction of component information in the terminal. The stereo image pair point cloud and the point cloud data are fused according to the sensor position information to realize the construction of a complete and detailed static digital surface model of the terminal interior.

7. The device according to claim 5, It is characterized in that The registration module includes a coordinate system association relationship construction submodule, a two-dimensional scene layout estimation submodule, a two-dimensional scene layout estimation update submodule and a projection submodule; The coordinate system association relationship construction submodule is used to construct the mapping relationship between the coordinate system of the indoor static digital surface 3D model of the terminal building and the world coordinate system, the camera coordinate system and the world coordinate system, and the camera coordinate system and the image coordinate system based on the world coordinate system; A two-dimensional scene layout estimation submodule is used to annotate the three-dimensional model layout of the indoor static digital surface of the terminal in the world coordinate system, analyze the effective distance of the camera, obtain the scene layout under the effective distance of the camera, and project and obtain a preliminary estimate of the two-dimensional scene layout of the camera based on the scene layout; The 2D scene layout estimation and update submodule is used to calculate the rotation matrix when the camera rotates, and to update the initial layout according to the rotation matrix; The projection submodule is used to project the two-dimensional scene onto the three-dimensional model of the static digital surface of the terminal building. It includes dividing the video image into texture areas based on the two-dimensional layout nodes, performing affine transformation and reconstruction on the image texture of each area, building the relationship between the two-dimensional initial layout and the scene layout, and projecting it into the three-dimensional model.

8. The device according to claim 7, It is characterized in that The fusion module is specifically used for: Different perspective differences are processed differently during fusion: For image areas with large perspective differences, directly select The video image texture corresponding to the smaller surveillance camera is overlaid; for the image area with small viewing angle difference, image fusion is performed; For the image fusion, the stitching lines in the fusion rendering process are calculated, and image fusion is performed based on the stitching lines to realize the video digital twin of the entire scene of the terminal.

9. An electronic device, It is characterized in that include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 4 is implemented.

Images