A method, device, equipment and medium for video digital twinning of airport terminal

Abstract

The present invention provides a video digital twin method, device, equipment and medium for an airport terminal. First, data collected by multiple sensors is obtained, and then a high-precision static three-dimensional model of the airport terminal is constructed. Based on the world coordinate system, a mapping relationship between a two-dimensional image coordinate system, a static three-dimensional model coordinate system of the terminal, and a camera coordinate system is constructed to obtain the initial layout of the terminal space. The image texture is reconstructed with the two-dimensional layout estimation as the key point to achieve the registration of the two-dimensional image texture to the static three-dimensional model of the terminal under a single camera. Finally, the suture line in the fusion rendering process is calculated, and the two-dimensional images of multiple cameras are fused with the constructed static three-dimensional model to achieve a three-dimensional panoramic digital twin of the terminal. Through the scheme of the present invention, three-dimensional real scene visualization in a large scene of fusion of multiple cameras, structured data, and indoor models of the terminal is achieved, and comprehensive real-time monitoring is achieved to ensure the safe operation of the airport.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, equipment and medium for creating a digital twin of an airport terminal video. Background Technology

[0002] Video digital twins are an emerging technology that has emerged in the last two years. They primarily address the mapping of spatial data to real-world spaces and solve the problems of difficulty in visualizing the spatial locations of various facilities / objects and detecting abnormal events when staff simultaneously view massive amounts of video. Currently, airports mainly monitor events through panoramic stitching and manual review of video footage. This method is insufficient for timely and effective monitoring and detection of various events, as well as for establishing physical location correlations. Existing solutions mainly focus on the following points:

[0003] (1) Monitor the entire terminal building using panoramic stitching;

[0004] (2) Analyze single-camera scenes using video analysis techniques;

[0005] (3) Viewing is conducted in conjunction with a camera and a GIS (Geographic Information System);

[0006] (4) Digital twin methods.

[0007] Among the four mainstream solutions mentioned above, the following problems exist:

[0008] (1) The first method requires that the parallax between the images to be stitched should not be too large and there should be some overlap between the two frames. However, the airport terminal has a complex structure and there are obstructions to the field of view and large parallax between the monitoring cameras. Therefore, this method is not suitable for indoor monitoring scenarios in the terminal.

[0009] (2) The second option can monitor passenger behavior with a single camera, but it is difficult to implement multiple cameras to connect and monitor events within the area.

[0010] (3) Although the third option associates the surveillance camera with the real geographical location, it is limited to the display of a single camera and cannot link multiple cameras together.

[0011] (4) The fourth method of digital twin mainly uses static modeling, but it cannot achieve real-time modeling of dynamic scenes, and in the twinning process, it depends on the completeness and accuracy of the collected data. Summary of the Invention

[0012] To address the aforementioned issues, this invention proposes a method, apparatus, equipment, and medium for digital twin of airport terminal video. It employs multiple sensors to reconstruct the three-dimensional model of a large terminal building from a static model, establishes a mapping relationship between two-dimensional images and three-dimensional models, registers the texture of two-dimensional images to the three-dimensional model, and finally performs multi-camera fusion and stitching, and maps the structured data based on video analysis to achieve real-time three-dimensional panoramic monitoring of a large terminal building scene.

[0013] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0014] The data involved in the entire solution includes: image data acquired by RGB-D (depth camera) sensors, point cloud data acquired by LIDAR (LiDAR scanner) sensors, video data acquired by surveillance cameras, airport CAD (computer-aided design) drawings, and camera installation parameters (including camera intrinsic parameters and installation location).

[0015] In a first aspect, the present invention provides a method for creating a digital twin of an airport terminal video, the method comprising the following steps:

[0016] Step S1, data acquisition: acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera;

[0017] Step S2, 3D reconstruction of the static scene of the terminal building: Based on the image data, point cloud data and video data, a high-precision 3D model of the static scene of the airport terminal building is constructed. This includes constructing a 3D model of the static scene using RGB-D sensor position and attitude image data, fusing point cloud data collected by LIDAR sensor according to the 3D model of the static scene, performing semantic segmentation on the point cloud data, and constructing a detailed 3D model of the static digital surface of the terminal building's interior.

[0018] Step S3: Registration of 2D image texture from single camera to 3D model. Based on the video data collected by the camera and the detailed 3D model of the static digital surface of the terminal building, the registration of 2D image texture from single camera to the detailed 3D model of the static digital surface of the terminal building is carried out. This includes constructing the 2D image, the detailed 3D model of the static digital surface of the terminal building, and the mapping relationship between the camera coordinate system and the world coordinate system. Then, based on the initial pose of the monitoring camera, the initial layout of the terminal building space is obtained, the rotation matrix of the monitoring camera is solved, the layout of the indoor space of the terminal building is updated, the key points are estimated based on the 2D layout, the image texture is divided, the image texture is reconstructed, and projected onto the 3D model of the static digital surface of the terminal building.

[0019] Step S4 involves fusing multiple cameras with the constructed static digital surface 3D model, including screening terminal cameras, calculating the initial layout estimate for each camera, determining whether there is overlap between cameras, calculating the viewing angle difference between cameras, and fusing different textures based on the viewing angle difference to achieve a digital twin of the terminal video.

[0020] Secondly, the present invention provides an airport terminal video digital twin device, comprising: a data acquisition module, a construction module, a registration module, and a fusion module;

[0021] The data acquisition module is used to acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera.

[0022] The construction module is used for 3D reconstruction of static scenes of the terminal building. It constructs a high-precision 3D model of the static scene of the airport terminal building based on the image data, point cloud data and video data. This includes constructing a 3D model of the static scene using position and attitude image data from RGB-D sensors, fusing point cloud data collected by LIDAR sensors based on the 3D model of the static scene, performing semantic segmentation on the point cloud data, and constructing a detailed 3D model of the indoor static digital surface of the terminal building.

[0023] The registration module is used for registering 2D image textures to 3D models from a single camera. It registers 2D image textures to 3D models from a single camera based on video data collected by the camera and a detailed 3D model of the static digital surface of the terminal building. This includes constructing a 2D image, a detailed 3D model of the static digital surface of the terminal building, and a mapping relationship between the camera coordinate system and the world coordinate system. Based on the mapping relationship, it obtains the initial layout of the terminal building space, estimates key points based on the 2D layout, reconstructs the image texture, and projects it onto the 3D model of the static digital surface of the terminal building.

[0024] The fusion module is used to fuse multiple cameras with a constructed static digital surface 3D model. It includes screening terminal cameras, calculating the initial layout estimate for each camera, determining whether there is overlap between cameras, calculating the viewing angle difference between cameras, and fusing different textures based on the viewing angle difference to achieve a digital twin of terminal video.

[0025] Furthermore, the building module is specifically used for:

[0026] Constructing a high-precision static 3D model of the terminal building includes acquiring RGB-D depth images, performing noise reduction on the depth maps, and removing outliers;

[0027] For the image data depth map, a frame model tracking method is used for attitude estimation. The initial data frame depth map is selected as valid data to participate in the reconstruction of the 3D model of the terminal static scene. Corresponding points between the selected data frame and the remaining frames are identified, error is measured, and the incremental transformation with the previous input frame depth map is estimated. This estimate is then connected to the previous attitude estimation results to obtain the terminal static scene model.

[0028] The point cloud data and RGB-D image data collected by the LIDAR sensor are registered to obtain three-dimensional point cloud texture information. The structured information of the terminal plane and its edges is obtained through point cloud segmentation. The corresponding objects are identified using deep learning methods to construct the information of the components inside the terminal. The stereo image pair point cloud and the point cloud data are fused according to the sensor position information to realize the construction of a complete and detailed static digital surface model of the terminal interior.

[0029] Furthermore, the registration module includes a coordinate system association construction submodule, a two-dimensional scene layout estimation submodule, a two-dimensional scene layout estimation update submodule, and a projection submodule;

[0030] The coordinate system association construction submodule is used to construct the mapping relationship between the coordinate system of the 3D model of the indoor static digital surface of the terminal building and the world coordinate system, the camera coordinate system and the world coordinate system, and the camera coordinate system and the image coordinate system, with the world coordinate system as the reference.

[0031] The 2D scene layout estimation submodule is used to annotate the 3D model layout of the static digital surface of the terminal building in the world coordinate system, analyze the effective distance of the camera, obtain the scene layout under the effective distance of the camera, and project the scene layout to obtain a preliminary estimate of the 2D scene layout of the camera based on the scene layout.

[0032] The 2D scene layout estimation and update submodule is used to calculate the rotation matrix when the camera rotates, and update the initial layout based on the rotation matrix;

[0033] The projection submodule is used to project a two-dimensional scene onto a three-dimensional model of a static digital surface inside the terminal building. It includes dividing the video image into texture regions based on two-dimensional layout nodes, performing affine transformation and reconstruction on the image texture of each region, constructing the relationship between the initial two-dimensional layout and the scene layout, and projecting it onto the three-dimensional model.

[0034] Furthermore, the fusion module is specifically used for:

[0035] Different viewpoint differences are handled differently during fusion: for image regions with large viewpoint differences, d is directly selected. iThe video image texture corresponding to the smaller surveillance camera is covered; for image areas with small viewing angle differences, image fusion is performed; for the image fusion, the stitching line in the fusion rendering process is calculated, and the image fusion is performed based on the stitching line to realize the video digital twin of the entire terminal scene.

[0036] Thirdly, the present invention provides an electronic device, comprising: at least one processor and a memory;

[0037] The memory stores computer-executed instructions;

[0038] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the computation method as described in any of the first aspects.

[0039] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the computation method described in any of the first aspects.

[0040] This invention achieves precise monitoring of all airport elements and real-time monitoring of the entire terminal building by combining video analysis with the construction of a static 3D model of the airport terminal. Specifically, it is based on multi-source point cloud data and video data acquired by RGB-D and LiDAR sensors. By constructing a mapping relationship between the real world, 3D models, cameras, and 2D images, and using 3D reconstruction technology of local dynamic scenes under a single camera as the core, it realizes 3D real-scene visualization of the terminal building in a large scene by fusing multiple cameras, structured data, and the terminal building's indoor model. It combines digital space with real space to achieve comprehensive real-time monitoring of the large terminal building scene, ultimately enabling airport staff to "see quickly," "understand," and "see clearly" from massive amounts of fragmented surveillance video. Compared with existing technologies, this invention has the following beneficial technical effects:

[0041] (1) Based on the construction of the mapping relationship between physical space, two-dimensional space and world space, accurate estimation of indoor layout structure is achieved without depth information;

[0042] (2) By integrating real-time video images with the static 3D model of the terminal, the digital space and the real space are made into a real-time twin, enabling comprehensive real-time monitoring of the terminal in a large scene, providing strong support for the safe operation of the airport. This method can also be extended to the airport public area and flight area, and is also applicable to complex indoor scenes. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a business process diagram of an airport terminal video digital twin method provided by an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a local three-dimensional model constructed according to an embodiment of the present invention;

[0046] Figure 3 This is an example of a panoramic real-time monitoring image obtained by fusing multiple cameras with a three-dimensional static scene model, as provided in an embodiment of the present invention.

[0047] Figure 4 This is a schematic diagram of the structure of an airport terminal video digital twin device provided in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] The following is a detailed application description of the present invention, using an actual airport terminal scenario. The present invention provides a method for creating a digital twin of airport terminal video, such as... Figure 1 As shown, the method includes the following steps:

[0051] (1) Data Acquisition

[0052] It acquires image data from RGB-D sensors, point cloud data from LIDAR sensors, and video data from surveillance cameras.

[0053] (2) Based on the characteristics of the terminal building scene, construct a detailed 3D model of the static digital surface of the terminal building interior.

[0054] This involves constructing a static 3D model of the airport terminal using image data from RGB-D sensors, fusing the constructed static 3D model with point cloud data collected by LiDAR sensors, performing semantic segmentation on the point cloud data, and constructing terminal components to achieve a detailed 3D model of the terminal's indoor static digital surfaces. Details are as follows:

[0055] Based on the data collected by the RGB-D sensor, a 3D reconstruction of the static scene of the terminal building is performed. The normal of each point is calculated using a bilateral filter and finite difference. The RGB-D image data is then processed for depth map noise reduction and outlier removal.

[0056] For the acquired image data depth map, the frame model tracking method is used for attitude estimation. The initial data frame depth map is used as the input for the reconstruction of the 3D model of the static scene of the terminal. The corresponding points between the selected data frame and the other frames are identified, the error is measured, the incremental transformation with the depth map of the previous input frame is estimated, and the estimation is connected to the previous attitude estimation results. This reduces the problem of RGB-D local trajectory error accumulation. In this way, the 3D model of the static digital surface of the terminal is obtained.

[0057] Based on the existing static 3D model of the terminal building, an indoor static texture model of the terminal building is constructed by fusing stereo image pairs from video data with acquired point cloud data. A LiDAR sensor is used to acquire 3D point cloud data of the complex texture scene of the terminal building. Image data and 3D point cloud data are registered to obtain texture information from laser-scanned 3D point cloud data. Based on sensor position information, stereo image pairs and 3D laser-scanned point cloud data are fused to establish the model. The registration is based on geometric transformation to align the point cloud data with the image.

[0058] The acquired point cloud data is semantically segmented to obtain structured information such as the terminal's floor plan and edges. Deep learning methods are then used to identify corresponding objects, enabling the construction of information about components within the terminal (such as counters). Ultimately, a detailed digital 3D model of the static scene within the terminal is constructed, as shown in the example below. Figure 2 As shown.

[0059] (3) Registration of 2D image textures to 3D models under single camera

[0060] This includes constructing a 2D image, a detailed 3D model of the terminal's indoor static digital surface, and the mapping relationship between the camera coordinate system and the world coordinate system. Based on the initial pose of the surveillance cameras, the initial spatial layout of the terminal is obtained, the camera rotation matrix is ​​solved, the indoor spatial layout of the terminal is updated, key points are estimated based on the 2D layout, image texture is segmented, the image texture is reconstructed, and then projected onto the 3D model of the terminal's indoor static digital surface. The details are as follows:

[0061] Using world coordinate system O W Based on this, a coordinate system O for the three-dimensional model of the static digital surface of the terminal building is established. M and the coordinate system of the surveillance camera Image coordinate system The mapping relationship between them;

[0062] Specifically, under the premise of scaling the 3D model of the terminal building's static digital surface to the real scene, the scaling parameter s is determined by the scaling ratio. For the constructed 3D model of the terminal building's static digital surface, a coordinate system O is established with its center as the origin. M Let the coordinates of each vertex in the model be a set {V}. Mi} i＝1，2，3，…n Where n is a natural number, similarly, construct the world coordinate system O. W (with the vertical direction of the ground as the z-axis), the mapping relationship between the two coordinate systems is shown in the following equation: s·O M =R MW O W +t MW ;

[0063] We choose the origin of the coordinate system so that its position in the 3D model of the terminal's static digital surface matches its relative position in the real world. Therefore, R... MW =I,t MW =0, where I is the identity matrix, from which we can calculate: s·O M =O W ;

[0064] Surveillance camera c i In world coordinate system O W The lower position is used This indicates that it represents the coordinate values ​​of a point in the constructed coordinate system. This represents the actual location of the surveillance cameras on the plane of the airport terminal. This represents the actual installation height of the surveillance camera.

[0065] Let the surveillance camera c i The initial pose in the world coordinate system is and They represent surveillance cameras ci Relative to world coordinate system O W The rotation matrix and displacement vector of the origin, and the monitoring camera c i Corresponding coordinate system and world coordinate system O W The mapping relationship between the two is expressed by the following formula:

[0066] From the coordinate system of the surveillance camera The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix K of the surveillance camera. i The intrinsic parameter matrix K of the surveillance camera is determined. i The coordinate system of the surveillance camera is obtained from the calibration of the surveillance camera. and image coordinate system The mapping relationship is as follows: Where z is the depth of the corresponding node in the coordinate system of the surveillance camera, thereby realizing the establishment of the transformation and mapping relationship between the coordinate system of the static digital surface model of the terminal building, the world coordinate system, the coordinate system of the surveillance camera, and the image coordinate system.

[0067] When the surveillance camera rotates, the displacement vector t = 0. We can solve for the rotation matrix of the surveillance camera using the following formula: ||[x′] i ] × Rx i ||=0, where x i ,x′ i R is a set of three-dimensional points that satisfy geometric constraints, where R is the rotation matrix and |‖‖| represents the norm of the matrix.

[0068] The overall layout of the static digital surface model of the terminal building in the world coordinate system is L={(s k e k Let L be a list of pairs of numbers, where k = 0, 1, ..., n, and n is a natural number. Let L be a list of pairs of numbers. k ,e k These represent the start and end points of the layout line segment k, respectively. In practical applications, d is defined as... i For surveillance camera c i The analyzable effective distance is then the surveillance camera c i Scene layout within the analysis distance for:

[0069]

[0070] Where, r i Let c be a unit vector representing the surveillance camera. i The principal axis direction, t i For the step size in the search domain, L WFor all layout line segments analyzed, surveillance camera c i The corresponding two-dimensional layout l in the image coordinate system i It can be done The depth of the corresponding node in the camera's coordinate system is obtained by projection.

[0071] When the viewpoint of the surveillance camera rotates, the layout is updated using the rotation matrix R.

[0072] Based on the acquired two-dimensional layout l i The video image is divided into texture regions, and the texture of each region is subjected to affine transformation and reconstruction. The l is established according to the following formula. i and The relationship between them is projected onto a 3D model, where... Represents the layout of the terminal building in the coordinate system of the 3D model of the static digital surface:

[0073] (4) Video images acquired by multiple cameras are fused with the static digital surface model of the terminal building to achieve real-time panoramic monitoring of the terminal building.

[0074] The fusion of multiple cameras with a 3D model includes screening terminal building cameras, calculating the initial layout estimate for each camera, determining whether there is overlap between cameras, calculating the viewing angle difference between cameras (the magnitude of which is related to the camera's focal length and represents the size of the surveillance camera's illumination range), and then using different texture fusion methods based on the viewing angle difference to achieve a full-scene digital twin of the terminal building video, as detailed below:

[0075] In airport terminal scenarios, multiple surveillance cameras often cover the same area, necessitating multi-camera, multi-view image fusion. This invention addresses this by processing different viewing angle differences differently during fusion: for image areas with large viewing angle differences, the df value is directly selected. i The video image texture corresponding to the smaller surveillance camera is covered; for image areas with small viewing angle differences, the image stitching is performed using the image stitching line as shown in Equation (1), and the image fusion method is shown in Equation (2). The stitching line calculation and fusion formula in the fusion rendering process are as follows:

[0076] M(p, q, I) i I j )=‖I i (p)-I j (p)‖+‖I i (q)-I j (q)‖ (1)

[0077] RGB(I)=(1-α)RGB(Ii )+αRGB(I j (2)

[0078] Where p and q are two adjacent output pixels in the overlapping region, I i (p), I j (p) indicates that point p is in image I. i and Image I j The color in I i (q), I j (q) indicates that point q is in image I. i and Image I j The color, I i I j They represent surveillance cameras c i and c j Image frames at the same time, RGB represents the image format as RGB, α ​​is a calculation parameter, 0 < α < 1, and the panoramic real-time monitoring image is obtained by fusing video images acquired by multiple cameras with the static digital surface 3D model of the terminal building. Figure 3 As shown.

[0079] Figure 4 This is a structural schematic diagram of an airport terminal video digital twin device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the apparatus in this embodiment may include: a data acquisition module, a construction module, a registration module, and a fusion module;

[0080] The data acquisition module is used to acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the camera.

[0081] The construction module is used for 3D reconstruction of static scenes of the terminal building. It constructs a high-precision 3D model of the static scene of the airport terminal building based on the image data, point cloud data and video data. This includes constructing a 3D model of the static scene using position and attitude image data from RGB-D sensors, fusing point cloud data collected by LIDAR sensors based on the 3D model of the static scene, performing semantic segmentation on the point cloud data, and constructing a refined indoor static digital surface model of the terminal building.

[0082] The registration module is used for registering 2D image textures to 3D models from a single camera. It registers the 2D image textures from a single camera to a detailed 3D model of the terminal building's indoor static digital surface based on the video data collected by the camera and the detailed indoor static digital surface model. This includes constructing a 2D image, a detailed 3D model of the terminal building's indoor static digital surface, and the mapping relationship between the camera coordinate system and the world coordinate system. Based on the initial pose of the monitoring camera, it obtains the initial layout of the terminal building space, solves the camera rotation matrix, updates the indoor spatial layout of the terminal building, estimates key points based on the 2D layout, performs image texture segmentation, reconstructs the image texture, and projects it onto the 3D model of the terminal building's indoor static digital surface.

[0083] The fusion module is used to fuse multiple cameras with a high-precision static digital surface 3D model. This includes screening terminal cameras, calculating the initial layout estimate for each camera, determining whether there is overlap between cameras, calculating the viewing angle difference between cameras, and fusing different textures based on the viewing angle difference to achieve a digital twin of the terminal video.

[0084] Optionally, the construction module specifically includes denoising the depth map and removing outliers from the acquired RGB-D data, using bilateral filters and finite difference to calculate the normal of each point for depth map denoising; using frame model tracking method to estimate the pose of the surveillance camera, selecting the initial data frame as valid data to participate in the reconstruction of the static 3D model of the terminal, identifying the corresponding points between the selected frame and the remaining frames, measuring the error, estimating the incremental transformation with the previous input frame, and connecting this estimate to the previous pose estimation results, thereby realizing the reconstruction of the static 3D model of the terminal in a large scene;

[0085] Based on the existing static 3D model of the terminal building, an indoor static texture model of the terminal building is constructed by fusing stereo image pairs from video data with acquired point cloud data. A LiDAR sensor is used to acquire 3D point cloud data of the complex texture scene of the terminal building. Image data and 3D point cloud data are registered to obtain texture information from laser-scanned 3D point cloud data. Finally, based on sensor position information, stereo image pairs and 3D laser-scanned point cloud data are fused to build the model.

[0086] The acquired point cloud data is semantically segmented to obtain structured information such as the terminal floor plan and edges. Deep learning methods are then used to identify corresponding objects, enabling the construction of information about components within the terminal (such as counters). Ultimately, a detailed digital 3D model of the static scene within the terminal is constructed.

[0087] Optionally, a registration module, specifically used in world coordinate system O W Based on this, establish the coordinate system O of the static digital surface model of the terminal building.M and the coordinate system of the surveillance camera Image coordinate system Mapping relationship between them: For the constructed static digital surface 3D model of the terminal building, a 3D model coordinate system O is established with its center as the origin. M Let the coordinates of each vertex in the model be a set {V}. Mi} i＝1，2，3，…n Where n is a natural number, similarly, construct the world coordinate system O. W (with the vertical direction of the ground as the z-axis), the mapping relationship between the two coordinate systems is shown in the following equation: s·O M =R MW O W +t MW ;

[0088] We choose the origin of the coordinate system so that its position in the 3D model of the terminal's static digital surface matches its relative position in the real world. Therefore, R... MW =I,t MW =0, where I is the identity matrix, from which we can calculate: s·O M =O W ;

[0089] Surveillance camera c i In world coordinate system O W The lower position is used This indicates that it represents the coordinate values ​​of a point in the constructed coordinate system. This represents the actual location of the surveillance cameras on the plane of the airport terminal. This represents the actual installation height of the surveillance camera.

[0090] Let the surveillance camera c i The initial pose in the world coordinate system is and They represent surveillance cameras c i Relative to world coordinate system O W The rotation matrix and displacement vector of the origin, and the monitoring camera c i Corresponding coordinate system and world coordinate system O W The mapping relationship between the two is expressed by the following formula:

[0091] From the coordinate system of the surveillance camera The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix K of the surveillance camera. i The intrinsic parameter matrix K of the surveillance camera is determined. i The coordinate system of the surveillance camera is obtained from the calibration of the surveillance camera. and image coordinate system The mapping relationship is as follows: Where z is the depth of the corresponding node in the coordinate system of the surveillance camera, thereby realizing the establishment of the transformation and mapping relationship between the coordinate system of the static digital surface model of the terminal building, the world coordinate system, the coordinate system of the surveillance camera, and the image coordinate system.

[0092] When the surveillance camera rotates, the displacement vector t = 0. We can solve for the rotation matrix of the surveillance camera using the following formula: ||[x′] i ] × Rx i ||=0, where x i ,x′ i R is a set of three-dimensional points that satisfy geometric constraints, where R is the rotation matrix and |‖‖| represents the norm of the matrix.

[0093] The overall layout of the static digital surface model of the terminal building in the world coordinate system is L={(s k e k Let L be a list of pairs of numbers, where k = 0, 1, ..., n, and n is a natural number. Let L be a list of pairs of numbers. k ,e k These represent the start and end points of the layout line segment k, respectively. In practical applications, d is defined as... i For surveillance camera c i The analyzable effective distance is then the surveillance camera c i Scene layout within the analysis distance for:

[0094]

[0095] Where, r i Let c be a unit vector representing the surveillance camera. i The principal axis direction, t i For the step size in the search domain, L W For all layout line segments analyzed, surveillance camera c i The corresponding two-dimensional layout l in the image coordinate system i It can be done The depth of the corresponding node in the camera's coordinate system is obtained by projection.

[0096] When the viewpoint of the surveillance camera rotates, the layout is updated using the rotation matrix R.

[0097] Based on the acquired two-dimensional layout l i The video image is divided into texture regions, and the texture of each region is subjected to affine transformation and reconstruction. The l is established according to the following formula. i and The relationship between them is projected onto a 3D model, where... Represents the layout of the terminal building in the coordinate system of the 3D model of the static digital surface:

[0098] Optionally, the fusion module is specifically used to handle different viewing angle differences differently during fusion: In a terminal building scenario, multiple surveillance cameras may cover the same area, thus requiring multi-camera, multi-view image fusion. For image areas with large viewing angle differences, d is directly selected. i The video image texture corresponding to smaller surveillance cameras is covered; for image areas with small viewing angle differences, the stitching line calculation and image fusion formula in the fusion rendering process are respectively: M(p, q, I) i I j )=‖I i (p)-I j (p)‖+‖I i (q)-I j (q)‖,RGB(I)=(1-α)RGB(I i )+αRGB(I j ), where p and q are two adjacent output pixels in the overlapping region, I i (p), I j (p) indicates that point p is in image I. i and Image I j The color in I i (q), I j (q) indicates that point q is in image I. i and Image I j The color, I i I j They represent surveillance cameras c i and c j Image frames at the same time, RGB represents the image in RGB format, α is a calculation parameter, 0 < α < 1.

[0099] The apparatus in this embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principle is similar, so it will not be described again here.

[0100] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device in this embodiment may include: a memory and a processor.

[0101] The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned method for digital twinning of airport terminal video), computer instructions, etc.

[0102] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0103] A processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

[0104] For details, please refer to the relevant descriptions in the preceding method embodiments.

[0105] Memory and processor can be separate structures or integrated structures. When memory and processor are separate structures, they can be coupled together via a bus.

[0106] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions, wherein when at least one processor of a user device executes the computer-executable instructions, the user device performs the various possible methods described above.

[0107] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0108] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video digital twin method for an airport terminal, characterized in that: The method comprises the following steps: Step S1, data acquisition, acquiring image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera; Step S2, three-dimensional reconstruction of the static scene of the terminal, constructing a high-precision three-dimensional model of the static scene of the airport terminal based on the image data, point cloud data and video data, including using the position and posture image data of the RGB-D sensor to build a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine three-dimensional model of the indoor static digital surface of the terminal; Step S3, registering the two-dimensional image texture to the three-dimensional model under a single camera, registering the two-dimensional image texture to the fine three-dimensional model of the terminal's indoor static digital surface under a single camera for the video data collected by the camera and the fine three-dimensional model of the terminal's indoor static digital surface, including constructing a mapping relationship between the two-dimensional image, the fine three-dimensional model of the terminal's indoor static digital surface, the camera coordinate system and the world coordinate system, and then obtaining the initial layout of the terminal space according to the initial position of the monitoring camera, solving the rotation matrix of the monitoring camera, updating the terminal's indoor space layout, estimating key points according to the two-dimensional layout, performing image texture division, reconstructing the image texture, and projecting it to the three-dimensional model of the terminal's indoor static digital surface; Step S4, multiple cameras are integrated with the constructed static digital surface 3D model, including screening the terminal cameras, calculating the initial layout estimation corresponding to each camera, determining whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, and integrating different textures according to the viewing angle difference to realize the terminal video digital twin; The step S3 specifically includes: Constructing the world coordinate system O W , the terminal building static digital surface 3D model coordinate system O M And the surveillance camera coordinate system Image coordinate system The mapping relationship between them is: the transformation relationship between the three-dimensional model coordinate system of the terminal static digital surface and the world coordinate system is s·O M =R MW O W +t MW , where R MW =I,t MW =0, I is the unit matrix, from which we can calculate: s·O M =O W ; Command surveillance camera c i The initial pose in the world coordinate system is and Represents surveillance camera c i Relative to the world coordinate system O W The rotation matrix and displacement vector of the origin, surveillance camera c i Corresponding coordinate system and the world coordinate system O W The mapping relationship between the two is expressed by the following formula: From the surveillance camera coordinate system The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix K of the surveillance camera. i Determine the intrinsic parameter matrix K of the surveillance camera i Obtained by monitoring camera calibration, monitoring camera coordinate system and the image coordinate system The mapping relationship is: Among them, z is the depth of the corresponding node in the surveillance camera coordinate system; When the surveillance camera rotates, the displacement vector t = 0, and the rotation matrix of the surveillance camera is solved by the following formula: ||[x′ i ] × Rx i ||=0, where x i ,x′ i is a set of three-dimensional points that satisfy geometric constraints, R is the rotation matrix, and |||| represents the norm of the matrix; The overall layout of the static digital surface model of the terminal building based on the world coordinate system is marked as L = {(s k , e k )|k=0,1,…n}, where n is a natural number, L is represented in the form of a dot pair, s k ,e k Represent the starting point and end point of the layout line segment k respectively, and define d i For surveillance cameras i The effective distance that can be analyzed is the monitoring camera c i Analyze scene layout within a certain distance for: Among them, r i is a unit vector representing the surveillance camera c i The main axis direction, t i is the step length of the search area, L W For all layout segments analyzed, the surveillance camera c i The corresponding two-dimensional layout l in the image coordinate system i pass Projection is performed to obtain, z is the depth of the corresponding node in the surveillance camera coordinate system: When the surveillance camera's viewing angle rotates, the layout is updated through the rotation matrix R; Based on the acquisition of two-dimensional layout l i , divide the video image into texture regions, and perform affine transformation and reconstruction on the image texture of each region. The l established according to the following formula i and The relationship between Represents the layout in the coordinate system of the static digital surface 3D model of the terminal: The step S4 specifically includes: Different perspective differences are processed differently during fusion: for image areas with large perspective differences, d i The video image texture corresponding to the smaller surveillance camera is covered; for the image area with small viewing angle difference, image fusion is performed, and for the image fusion, the stitching line in the fusion rendering process is calculated: M(p, q, I i , I j )=‖I i (p)-I j (p)‖+‖I i (q)-I j (q)‖, and perform image fusion based on the stitching lineRGB(I)=(1-α)RGB(I i )+αRGB(I j ), where p and q are two adjacent output pixels in the overlapping area, I i (p), I j (p) represents point p in image I i and image I j Color in I i (q), I j (q) represents point q in image I i and image I j Color, I i , I j Represents camera c i and c j In the image frame at the same moment, RGB represents the image in RGB format, α is the calculation parameter, 0<α<1, thereby realizing the video digital twin of the entire scene of the terminal.

2. The method according to claim 1, characterized in that The step S2 specifically includes: Acquire RGB-D image data as data for constructing a static scene model of the terminal, use a bilateral filter and finite difference to calculate the normal of each point, and perform depth map denoising and outlier removal processing on the RGB-D image data; For the image data depth map, use the frame model tracking method to perform posture estimation, select the initial data frame depth map as valid data to participate in the reconstruction of the terminal static scene 3D model, identify the corresponding points between the selected data frame and the remaining frames, perform error measurement, estimate the incremental transformation with the previous input frame depth map, connect the estimation to the previous posture estimation result, and obtain the terminal static scene model; The point cloud data and RGB-D image data collected by the LIDAR sensor are used for registration to obtain the texture information of the three-dimensional point cloud data. The structured information of the terminal plane and the edge of the plane is obtained through point cloud segmentation. The corresponding objects are identified using deep learning methods to realize the construction of component information in the terminal. The stereo image pair point cloud and the point cloud data are fused according to the sensor position information to realize the construction of a complete and detailed static digital surface model of the terminal interior.

3. An airport terminal video digital twin device, characterized in that: include: Data acquisition module, construction module, registration module and fusion module; The data acquisition module is used to acquire image data collected by the RGB-D sensor, point cloud data collected by the LIDAR sensor, and video data collected by the surveillance camera; The construction module is used for three-dimensional reconstruction of the static scene of the terminal. According to the image data, point cloud data and video data, a high-precision three-dimensional model of the static scene of the airport terminal is constructed, including using the position and posture image data of the RGB-D sensor to construct a static scene three-dimensional model, fusing the point cloud data collected by the LIDAR sensor according to the static scene three-dimensional model, performing semantic segmentation on the point cloud data, and constructing a fine three-dimensional model of the indoor static digital surface of the terminal; The registration module is used for registering the two-dimensional image texture to the three-dimensional model under a single camera. For the video data collected by the camera and the fine three-dimensional model of the indoor static digital surface of the terminal, the two-dimensional image texture under a single camera is registered to the fine three-dimensional model of the indoor static digital surface of the terminal, including constructing a mapping relationship between the two-dimensional image, the fine three-dimensional model of the indoor static digital surface of the terminal, the camera coordinate system and the world coordinate system, obtaining the initial layout of the terminal space according to the mapping relationship, estimating the key points according to the two-dimensional layout, reconstructing the image texture, and projecting it to the three-dimensional model of the indoor static digital surface of the terminal; The fusion module is used to fuse multiple cameras with the constructed static digital surface 3D model, including screening the terminal cameras, calculating the initial layout estimation corresponding to each camera, judging whether there is overlap between the cameras, calculating the viewing angle difference between the cameras, fusing different textures according to the viewing angle difference, and realizing the terminal video digital twin; Specifically, the registration module is used to construct the world coordinate system O W , the terminal building static digital surface 3D model coordinate system O M And the surveillance camera coordinate system Image coordinate system The mapping relationship between them is: the transformation relationship between the three-dimensional model coordinate system of the terminal static digital surface and the world coordinate system is s·O M =R MW O W +t MW , where R MW =I,t MW =0, I is the unit matrix, from which we can calculate: s·O M =O W ; Command surveillance camera c i The initial pose in the world coordinate system is and Represents surveillance camera c i Relative to the world coordinate system O W The rotation matrix and displacement vector of the origin, surveillance camera c i Corresponding coordinate system and the world coordinate system O W The mapping relationship between the two is expressed by the following formula: From the surveillance camera coordinate system The mapping transformation to the corresponding image coordinate system is determined by the intrinsic parameter matrix K of the surveillance camera. i Determine the intrinsic parameter matrix K of the surveillance camera i Obtained by monitoring camera calibration, monitoring camera coordinate system and the image coordinate system The mapping relationship is: Among them, z is the depth of the corresponding node in the surveillance camera coordinate system; When the surveillance camera rotates, the displacement vector t = 0, and the rotation matrix of the surveillance camera is solved by the following formula: ||[x′ i ] × Rx i ||=0, where x i ,x′ i is a set of three-dimensional points that satisfy geometric constraints, R is the rotation matrix, and |||| represents the norm of the matrix; The overall layout of the static digital surface model of the terminal building based on the world coordinate system is marked as L = {(s k , e k )|k=0,1,…n}, where n is a natural number, L is represented in the form of a dot pair, s k ,e k Represent the starting point and end point of the layout line segment k respectively, and define d i For surveillance cameras i The effective distance that can be analyzed is the monitoring camera c i Analyze scene layout within a certain distance for: Among them, r i is a unit vector, which represents the surveillance camera c i The main axis direction, t i is the step length of the search area, L W For all layout segments analyzed, the surveillance camera c i The corresponding two-dimensional layout l in the image coordinate system i pass Projection is performed to obtain, z is the depth of the corresponding node in the surveillance camera coordinate system: When the surveillance camera's viewing angle rotates, the layout is updated through the rotation matrix R; Based on the acquisition of two-dimensional layout l i , divide the video image into texture regions, and perform affine transformation and reconstruction on the image texture of each region. The l established according to the following formula i and The relationship between Represents the layout in the coordinate system of the static digital surface 3D model of the terminal: The fusion module is used to process different perspective differences in different ways during fusion: for image areas with large perspective differences, directly select d i The video image texture corresponding to the smaller surveillance camera is covered; for the image area with small viewing angle difference, image fusion is performed, and for the image fusion, the stitching line in the fusion rendering process is calculated: M(p, q, I i , I j )=‖I i (p)-I j (p)‖+‖I i (q)-I j (q)‖, and perform image fusion based on the stitching lineRGB(I)=(1-α)RGB(I i )+αRGB(I j ), where p and q are two adjacent output pixels in the overlapping area, I i (p), I j (p) represents point p in image I i and image I j Color in I i (q), I j (q) represents point q in image I i and image I j Color, I i , I j Represents camera c i and c j In the image frame at the same time, RGB represents that the image is in RGB format, α is a calculation parameter, 0<α<1.

4. The device according to claim 3, characterized in that The building blocks are specifically used for: Construct a high-precision static 3D model of the terminal, including collecting RGB-D depth images, performing denoising on the depth images, and removing outliers; For the image data depth map, use the frame model tracking method to perform posture estimation, select the initial data frame depth map as valid data to participate in the reconstruction of the terminal static scene 3D model, identify the corresponding points between the selected data frame and the remaining frames, perform error measurement, estimate the incremental transformation with the previous input frame depth map, connect the estimation to the previous posture estimation result, and obtain the terminal static scene model; The point cloud data and RGB-D image data collected by the LIDAR sensor are used for registration to obtain the texture information of the three-dimensional point cloud data. The structured information of the terminal plane and the edge of the plane is obtained through point cloud segmentation. The corresponding objects are identified using deep learning methods to realize the construction of component information in the terminal. The stereo image pair point cloud and the point cloud data are fused according to the sensor position information to realize the construction of a complete and detailed static digital surface model of the terminal interior.

5. The device according to claim 4, characterized in that The registration module includes a coordinate system association relationship construction submodule, a two-dimensional scene layout estimation submodule, a two-dimensional scene layout estimation update submodule and a projection submodule; The coordinate system association relationship construction submodule is used to construct the mapping relationship between the coordinate system of the indoor static digital surface 3D model of the terminal building and the world coordinate system, the camera coordinate system and the world coordinate system, and the camera coordinate system and the image coordinate system based on the world coordinate system; A two-dimensional scene layout estimation submodule is used to annotate the three-dimensional model layout of the indoor static digital surface of the terminal in the world coordinate system, analyze the effective distance of the camera, obtain the scene layout under the effective distance of the camera, and project and obtain a preliminary estimate of the two-dimensional scene layout of the camera based on the scene layout; The 2D scene layout estimation and update submodule is used to calculate the rotation matrix when the camera rotates, and to update the initial layout according to the rotation matrix; The projection submodule is used to project the two-dimensional scene onto the three-dimensional model of the static digital surface of the terminal building. It includes dividing the texture area of ​​the video image based on the two-dimensional layout nodes, performing affine transformation and reconstruction on the image texture of each area, building the relationship between the two-dimensional initial layout and the scene layout, and projecting it into the three-dimensional model.

6. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1-2.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 2 is implemented.

Images