Camera calibration digital twinning method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of video positioning, and relates to a camera digital twinning calibration method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a video sequence through a camera; establishing a three-dimensional map of the located area, wherein the three-dimensional map carries world coordinates; n calibration objects Qi are selected on the three-dimensional map, and the world coordinates of the n calibration objects are Qi (X, Y, Z); detecting real corresponding calibration objects Qi'of the n calibration objects Qi in the video sequence, and obtaining pixel coordinates Qi '(x, y) of the real corresponding calibration objects Qi'; a conversion matrix M exists, so that Qi'is equal to MQi, and the conversion matrix M is calculated through n groups of virtual-real corresponding points; detecting a target object in the video sequence, obtaining the type S and the pixel coordinate P '(x, y) of the target object, and calculating the world coordinate of the target object S through P' = MP; and twinning the three-dimensional model of the target object at the corresponding position of the three-dimensional scene. The positioning precision of the target object is improved; the real-time interaction between the video sequence and the three-dimensional scene is realized; and powerful support is provided for the application of the digital twinning technology.

Description

Technical Field

[0001] The present invention relates to the fields of video positioning and 3D data visualization, and in particular to a camera calibration digital twin method, device, computer equipment and storage medium. Background Art

[0002] A 3D electronic map is a three-dimensional, abstract description of one or more aspects of the real world or a part of it in a certain proportion based on a 3D electronic map database. A digital twin is a virtual counterpart (mapping) of a physical entity in the real world.

[0003] There are many methods for camera calibration and digital twins in three-dimensional scenes. In the field of video positioning and 3D data visualization, they are mainly based on multi-target tracking and various positioning models. Multi-target tracking is an important issue in the field of computer vision. It is currently widely used in sports broadcasting, security monitoring, drones, unmanned vehicles, robots and other fields. However, there are problems such as insufficient physical accuracy between the virtual and real worlds and "misalignment" in the time dimension. The monocular positioning model is a model that uses the pixel coordinates of an object on a single camera to calculate the three-dimensional coordinates of the target based on spatial geometric relationships, but it often requires calibration points to be taken on site, resulting in large errors and unnecessary tedious work. Summary of the invention

[0004] In order to solve the above technical problems, on the one hand, the present invention provides a camera calibration digital twin method, which adopts the following technical solution, including:

[0005] Acquiring a video sequence by the camera;

[0006] Establishing a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates;

[0007] Select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X,Y,Z), i=1,2...,n;

[0008] Detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y);

[0009] There exists a 3×4 transformation matrix M such that Q i '=MQ i , calculate the transformation matrix M through n sets of virtual and real corresponding points;

[0010] Detecting a target object in the video sequence, obtaining the type S and pixel coordinates P'(x, y) of the target object, and calculating the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP;

[0011] The three-dimensional model of the twin target object S at the corresponding position P(X,Y,Z) of the three-dimensional scene.

[0012] Preferably, the step of establishing a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates, specifically comprises:

[0013] Collect satellite imagery and elevation data of the area;

[0014] Processing the satellite images and elevation data;

[0015] Modeling and texture mapping are performed on the processed satellite images and elevation data;

[0016] Import geographic data with world coordinates.

[0017] Preferably, n calibration objects Q1, Q2, ..., Q are selected on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X, Y, Z), i = 1, ... n, the steps specifically include:

[0018] Select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n ;

[0019] Get the world coordinates of n calibration objects as Q respectively i (X,Y,Z), i=1,2...,n;

[0020] Store the world coordinates of n calibration objects as Q i (X,Y,Z).

[0021] Preferably, the detecting n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i The steps of '(x,y) specifically include:

[0022] Extract n calibration objects Q from the three-dimensional map i The reference color feature of the corresponding reference image;

[0023] Comparing the sequence color feature of the video sequence with the reference color feature to determine similarity;

[0024] If they are similar, it means that n calibration objects Q are detected. i The actual corresponding calibration object Q i ', and obtain the corresponding calibration object Q i 'Pixel coordinates Q i '(x,y).

[0025] Preferably, there exists a 3×4 transformation matrix M such that Q i '=MQ i , through n sets of virtual and real corresponding points, the steps of calculating the transformation matrix M specifically include:

[0026] Set n = 6;

[0027] There exists a 3×4 transformation matrix M such that Q i '=MQ i , i=1,...,6, according to Q i and Q i 'Calculate the transformation matrix M.

[0028] Preferably, the step of detecting the target object in the video sequence, obtaining the type S and pixel coordinates P'(x, y) of the target object, and calculating the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP specifically includes:

[0029] Preprocessing the video sequence;

[0030] Detect the target object S and identify the position of the target object S, i.e., the pixel coordinates P'(x, y) and the type s;

[0031] The world coordinates P (X, Y, Z) of the target object S are calculated using the relationship P'=MP.

[0032] Preferably, the step of twinning the three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene specifically includes:

[0033] Construct a three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene;

[0034] Optimize and adjust the three-dimensional model of the target object S.

[0035] In order to solve the above technical problems, the present invention also provides a camera calibration digital twin device, which adopts the following technical solution, including:

[0036] An acquisition module, used for acquiring a video sequence by the camera;

[0037] A building module, used to build a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates;

[0038] A selection module is used to select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X, Y, Z), i = 1, ... n;

[0039] A detection module, used to detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y);

[0040] The first calculation module is used to calculate the existence of a 3×4 transformation matrix M such that Q i '=MQ i , calculate the transformation matrix M through n sets of virtual and real corresponding points;

[0041] A second calculation module is used to detect a target object in the video sequence, obtain the type S and pixel coordinates P'(x, y) of the target object, and calculate the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP;

[0042] The twin module is used to twin the three-dimensional model of the target object S at the corresponding position P (X, Y, Z) in the three-dimensional scene.

[0043] In order to solve the above technical problems, the present invention also provides a computer device, which adopts the technical solution described below, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned camera calibration digital twin method when executing the computer-readable instructions.

[0044] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, which adopts the technical solution described below, and stores computer-readable instructions on the computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned camera calibration digital twin method are implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects: dynamic environmental information can be obtained through the video sequence captured by the camera; a three-dimensional map containing world coordinates is then established, providing a reference framework for subsequent calibration work; carefully selecting n calibration objects on the three-dimensional map and obtaining their precise world coordinates is the key to ensuring calibration accuracy; then the real counterparts of these calibration objects are accurately detected in the video sequence, and their pixel coordinates are obtained, providing necessary data for calculating the transformation matrix M; using n groups of virtual and real corresponding points, a 3×4 transformation matrix M is solved, which is a bridge connecting the pixel coordinates in the video sequence and the three-dimensional world coordinates. Once the transformation matrix M is determined, the target object can be detected in the video sequence, and its coordinates in the three-dimensional world can be calculated through the transformation matrix. Finally, a three-dimensional model of the target object is twinned at the corresponding position of the three-dimensional scene, realizing accurate mapping from video to three-dimensional scene; improving the accuracy of target positioning; realizing real-time interaction between video sequences and three-dimensional scenes; providing strong support for the application of digital twin technology; at the same time, through precise three-dimensional modeling, a solid foundation is laid for subsequent simulation, analysis and optimization work. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 is a flow chart of an embodiment of a camera calibration digital twin method of the present invention;

[0048] Figure 2 It is a schematic diagram of the scene used in the camera calibration digital twin method of the present invention;

[0049] Figure 3 It is a flow chart of establishing a three-dimensional map used in the camera calibration digital twin method of the present invention;

[0050] Figure 4 It is a detailed flow chart of the video target digital twin to three-dimensional scene used in the camera calibration digital twin method of the present invention;

[0051] Figure 5 It is a structural schematic diagram of an embodiment of a camera calibration digital twin device of the present invention;

[0052] Figure 6 It is a schematic structural diagram of an embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] It should be noted that the camera calibration digital twin method provided in the embodiment of the present invention is generally executed by a server / terminal device, and accordingly, the camera calibration digital twin device is generally set in the server / terminal device.

[0057] It should be understood that the number of terminal devices, networks and servers is only illustrative. Depending on the implementation requirements, there can be any number of terminal devices, networks and servers.

[0058] Embodiment 1

[0059] Please refer to Figure 1 , Figure 2 , showing a flow chart of an embodiment of the camera calibration digital twin method of the present invention and a schematic diagram of its use scenario. The camera calibration digital twin method comprises the following steps:

[0060] Step S1: A video sequence is acquired by a camera.

[0061] In this embodiment, the electronic device (e.g., server / terminal device) on which the camera calibration digital twin method runs can receive the camera calibration digital twin request through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAXX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0062] Cameras include various imaging devices that can acquire RGB images.

[0063] In this embodiment, step S1, obtaining a video sequence by a camera, may further include the following steps:

[0064] S11, capturing an image sequence in a scene by a camera.

[0065] The image sequence in the scene captured by the camera constitutes a video sequence. The camera can be fixed or mobile, depending on the requirements of the application scenario. Video acquisition provides raw data for subsequent target detection. You can choose the appropriate camera and adjust the appropriate shooting parameters (such as exposure, focal length, etc.) according to actual needs.

[0066] S12, preprocessing the collected video sequence.

[0067] The purpose of preprocessing is to eliminate noise and improve image quality. This includes operations such as image enhancement, denoising, and grayscale. The preprocessing step can significantly improve the usability of video data, reduce the computational complexity of subsequent processing, and improve the accuracy of target recognition. For example, grayscale operation can reduce the amount of data and increase processing speed, while image enhancement can enhance the contrast between the target area and the background, making the target easier to detect.

[0068] Step S2, creating a three-dimensional map of the area, the three-dimensional map carrying world coordinates.

[0069] In this embodiment, step S2, establishing a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates, may further include the following steps:

[0070] S21, collect satellite images and elevation data of the area.

[0071] For example, you can use tools such as BIGEMAP map downloader to download satellite images and elevation data of the required area. Select Google Earth source to download high-definition satellite images and elevation data. For example, you can download 16-level or 17-level elevation data, as well as higher-level satellite images, to ensure that the generated 3D map is clear and detailed.

[0072] Satellite images provide surface texture information, and elevation data provides terrain undulation information. The two are combined to form three-dimensional terrain.

[0073] You can also use a 3D laser scanner to obtain high-precision terrain data. The 3D laser scanner scans the target area and obtains point cloud data. The point cloud data contains the 3D coordinate information of the target area with an accuracy of up to millimeter level. Laser scanning data provides high-precision terrain and building surface information, which is suitable for modeling complex terrain.

[0074] You can also use an oblique photography camera to photograph the target area. Plan the shooting route and take multiple oblique photos. The photos must include detailed information such as the building's facade and terrain. Oblique photography data provides a rich perspective and sense of perspective, which is used to generate high-resolution textures and three-dimensional effects.

[0075] S22, processes satellite images and elevation data.

[0076] Perform projection conversion and format conversion. Convert satellite images, elevation data, and laser scanning data into formats suitable for 3D modeling software (such as .obj, .fbx, etc.). Projection conversion ensures the consistency and accuracy of the data, and format conversion enables the data to be used in 3D modeling software.

[0077] Export elevation data to DEM format. Use GIS software (such as ArcGIS) to export elevation data to DEM format. DEM format files contain terrain height information and can be used in 3D scenes. DEM format files are the basis of terrain modeling and are used to generate three-dimensional terrain models.

[0078] Process satellite images. For example, use image processing software (such as Photoshop) to pre-process satellite images, such as denoising and contrast enhancement. Define the projection and save it in GeoTIFF format for easy use in 3D modeling software. The pre-processed satellite images are used as texture maps to increase the realism and details of the model.

[0079] S23, modeling and texture mapping are performed on the processed satellite images and elevation data.

[0080] The steps of 3D laser scanning modeling are: Import the point cloud data of laser scanning in 3D modeling software (such as 3ds Max, Maya). Use the height map tool to convert the point cloud data into a 3D terrain model. Use the modeling tool to create geographical features such as buildings and roads. Laser scanning modeling provides high-precision terrain and building models, which is suitable for modeling complex terrain.

[0081] The steps of oblique photography 3D modeling are: Import oblique photography photos into 3D modeling software. Use photo matching algorithm to generate 3D point cloud model. Use texture extraction tool to apply photos as texture map to the model. Oblique photography modeling provides rich texture and details, increasing the realism of the model.

[0082] The steps for texture mapping are: Open the satellite image in the 3D modeling software and apply it as a texture map to the terrain and building models. Adjust the texture coordinates and material properties to ensure that the texture is displayed correctly. Texture mapping provides rich surface information of the ground and buildings, making the model more realistic and detailed.

[0083] S24, import geographic data with world coordinates.

[0084] Import geographic data with world coordinates (such as GIS data) into the 3D modeling software. Use the coordinate conversion tool to align the model coordinates with the world coordinates. The purpose of this is to ensure that each point in the model carries accurate world coordinate information for subsequent application and analysis.

[0085] It should be noted that the presentation forms of three-dimensional maps include but are not limited to three-dimensional real-scene maps, three-dimensional virtual maps, and other forms of maps with coordinates.

[0086] All the objective objects in the 3D map have world coordinates (WGS84 coordinates and elevation coordinates). The pre-established 3D map becomes an efficient calibration field. The world coordinates of the calibration objects can be directly obtained on the 3D map, which is convenient for the subsequent selection of calibration objects and the solution of virtual-real mapping relationship. The higher the world coordinate accuracy, the more conducive it is to the calibration object Q i selection and subsequent calculations.

[0087] Figure 3 This is a flowchart of establishing a three-dimensional map used in the camera calibration digital twin method of the present invention. Figure 3 As shown, the process of establishing a three-dimensional map of the area is as follows:

[0088] ①UAV flight photos:

[0089] For scenes with lower precision requirements, Altizure is used for automatic aerial photography. Aerial photography is carried out at five angles: front, back, left, right, and top; the photo overlap is 70%; the tilt angle is 45° when flying front, back, left, and right; the flight altitude is modified according to the actual terrain conditions; and the size of the flight area is adjusted according to the performance of different drones.

[0090] For scenes with high precision requirements, manual aerial photography is performed using DJI GO. Roads and highways are photographed by tilting, and the drone is photographed along the left and right sides of the road and highway. The height of the drone is adjusted according to local conditions. The height is controlled between 20 meters and 40 meters during tilting. Generally, the shooting is done back and forth, and the tilt angles can be different. For orthophoto shooting, the camera angle is adjusted to face down, and the center of the road and highway is generally photographed along the road, and the height is controlled above 40 meters. For buildings with more regular shapes, when the building is less than 50 meters, tilt shooting is adopted, the height is higher than the building, and each side of the building is photographed at the same height. Change the height and tilt angle to shoot each side of the building separately; when the building is higher than 50 meters, the tilt shooting can be adopted in segmented mode, and the building is photographed in segments, using different tilt angles to shoot each side of the building; if the top surface of the building is flat, it can be photographed in the same way as roads and highways.

[0091] If the top surface is triangular or not flat, shoot it once in an orthographic manner, and then use different angles to shoot in a surround manner; when the building is lower than 50 meters, shoot from the front, and the drone will fly to the middle height of the building (either lower or higher), adjust the camera angle to look straight ahead, and move the distance between the drone and the building to ensure that the top and bottom of the building are captured, and shoot all sides of the drone; when the building is higher than 50 meters, shoot from the front in segmented mode, and shoot the building in segments.

[0092] The shooting method for irregular-shaped buildings is generally the same as that for regular-shaped buildings, with the difference being that you need to shoot from multiple angles and at different distances. For some details, you need to shoot from close up and at low altitude. For shooting of some concave and convex surfaces, shadowed surfaces or holes, adjust the camera angle and try to capture the internal details of these places. If you can shoot orthogonally, do so. Use fly-around shooting to shoot buildings at different heights, distances and angles.

[0093] ②Add photos:

[0094] Photos taken by different drones should be placed in different folders because of different camera parameters. After adding photos, you can check the images and remove damaged photos to reduce the amount of aerial triangulation calculations. Photos from different blocks should be placed separately and imported in batches. When adding control points, each BLACK block should have at least 3 control points and each control point should have at least three photo points. When adding control points, 84 defaults to ellipsoid height. If orthometric height is required, the coordinate system should be the wgs84 orthometric coordinate system. Connection points can be used for restrictions, such as direction and size. The 3D view can view the camera shooting position. If the coordinate system or view is obviously incorrect, check whether there are damaged photos or GPS-missing photos in the added photos. Remove them and then check whether the 3D view is normal.

[0095] ③Aerial triangulation calculation:

[0096] The settings when submitting aerial triangulation are: density setting: high, optical image setting: multi-step; the engine task allocation for aerial triangulation calculation is random, so before submitting aerial triangulation, turn off other engines first, leave a well-configured computer to run the engine, and then submit the aerial triangulation calculation.

[0097] ④Modeling:

[0098] After the aerial triangulation calculation is completed, a new reconstruction project is created and the area is cut into blocks. It is best to narrow the range in the view to the range of overlapping photos. When cutting blocks, control the memory space slightly smaller than the maximum memory according to the computer memory size to reduce the task blocks as much as possible.

[0099] ⑤Block merging:

[0100] To merge blocks, select several BLACK blocks that have control points and have completed triangulation at the same time, right-click and choose Merge Blocks. At this time, you can merge several blocks into one BLACK block with triangulation results. Before merging blocks, the overlapping parts of the blocks to be merged must have 3 to 4 common control points. Otherwise, if the blocks themselves have control points, they will be misaligned after merging.

[0101] Step S3, select n calibration objects Q1, Q2, ..., Q i , ..., Q n , the world coordinates of n calibration objects are Q i (X,Y,Z), i=1,2...,n.

[0102] In this embodiment, step S3, n calibration objects Q1, Q2, ..., Q are selected on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X, Y, Z), i = 1, 2..., n can also specifically include the steps of:

[0103] S31, select n calibration objects Q1, Q2, ..., Q on the three-dimensional map i , ..., Q n .

[0104] The selection of calibration objects directly affects the accuracy and stability of the calibration results. It should be noted that appropriate calibration objects should be selected according to the application scenario (such as indoor, outdoor, lighting conditions, etc.). Ensure that the calibration objects have distinct features and are easy to detect and identify.

[0105] For fixed-focus and fixed-attitude cameras, 5-6 red circular targets are generally selected as calibration objects, and the color selection depends on the environment.

[0106] S32, respectively obtain the world coordinates of n calibration objects as Q i (X,Y,Z), i=1,2...,n.

[0107] Open the 3D map platform and enter the coordinate picking tool. Slide the mouse on the map to display the map coordinates synchronously. Click the selected location on the map.

[0108] Repeat the above steps until n calibration objects are selected, and obtain the world coordinates of n calibration objects as Q i (X,Y,Z), i=1,2...,n.

[0109] S33, store the world coordinates of n calibration objects as Q i (X,Y,Z).

[0110] Synchronously store the world coordinates of n calibration objects as Q i (X,Y,Z).

[0111] Step S4: Detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y).

[0112] Q i 'and Q i is a set of corresponding virtual and real points.

[0113] Detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', the selected detection algorithms include but are not limited to: CNN neural network, RNN neural network, DNN neural network, GAN neural network, and other deep learning algorithms.

[0114] In this embodiment, step S4, n calibration objects Q are detected in the video sequence. i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y) can also specifically include the following steps:

[0115] S41, extract n calibration objects Q in the three-dimensional map i The reference color features of the corresponding reference image.

[0116] The color feature extraction algorithm is used to extract n calibration objects Q in the three-dimensional map. iThe corresponding reference image is processed. Color feature extraction methods such as color histogram, color set, color moment, etc. can be used. For the calibration object in the three-dimensional map, for example, the color histogram is used to extract the reference color feature. It can count the proportion of different colors in the image, thereby reflecting the color distribution characteristics of the calibration object.

[0117] In order to extract color features more accurately, clustering algorithms can be further used to quantify colors. For example, the K-means clustering algorithm is used to divide the colors in the image into several categories, and then the number of pixels in each category is counted. This method can reduce the redundancy of color features and improve the efficiency of feature extraction.

[0118] S42, comparing the sequence color feature of the video sequence with the reference color feature to determine the similarity.

[0119] The same method as step S41 is used to extract the sequence color features of the video sequence.

[0120] After extracting the sequence color features and reference color features, feature matching and verification are performed. The sequence color features of the video sequence are compared with the reference color features to determine the similarity to verify the identity or attributes of the calibrated object. The matching algorithm can be selected according to the specific application scenario, such as Euclidean distance, Manhattan distance, etc.

[0121] Finally, the extracted sequence color features and reference color features are stored and managed separately. The sequence color features and reference color features are saved in the database for subsequent analysis and processing. At the same time, an effective index and query mechanism is established to quickly retrieve and use these feature data.

[0122] S43, if similar, it means that n calibration objects Q are detected i The actual corresponding calibration object Q i ', and obtain the corresponding calibration object Q i 'Pixel coordinates Q i '(x,y).

[0123] If the sequence color features are similar to the reference color features, it means that n calibration objects Q are detected i The actual corresponding calibration object Q i '. Then record the corresponding calibration object Q i 'Pixel coordinates Q i '(x,y).

[0124] Step S5, there exists a 3×4 transformation matrix M such that Q i '=MQ i , calculate the transformation matrix M through n sets of virtual and real corresponding points.

[0125] In this embodiment, step S5, there exists a 3×4 transformation matrix M such that Q i '=MQ i , through n groups of virtual and real corresponding points, calculating the transformation matrix M may further include the following steps:

[0126] S51, set n=6.

[0127] In specific implementation, the larger n is, the more corresponding points are calibrated, which is more conducive to reducing errors and obtaining a more accurate conversion matrix M between pixel coordinates and world coordinates. The number of groups of corresponding points is ≥ 6, and n is selected as 6 in this embodiment.

[0128] S52, there exists a 3×4 transformation matrix M such that Q i '=MQ i , i=1,...,6, according to Q i and Q i 'Calculate the transformation matrix M.

[0129] From the above steps, we know Q i 'and Q i , then the transformation matrix M can be calculated.

[0130] Step S6, detecting the target object in the video sequence, obtaining the type S and pixel coordinates P'(x, y) of the target object, and calculating the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP.

[0131] In this embodiment, step S6, detecting the target object in the video sequence, obtaining the type S and pixel coordinates P'(x, y) of the target object, and calculating the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP may further include the following steps:

[0132] S61, preprocessing the video sequence.

[0133] Preprocessing video sequences includes extracting video frames. Video frame extraction is to split the video sequence into individual image frames for frame-by-frame analysis. Use a video processing library such as OpenCV to read the video sequence. Video sequences usually exist in the form of files. Iterate through the video frames, extract them frame by frame and save them as image files or process them directly in memory.

[0134] S62, detecting the target object S, identifying the position of the target object S, ie, the pixel coordinates P'(x, y) and the type s.

[0135] Apply target detection algorithms such as Haar Cascades, YOLO, SSD, etc. on the extracted video frames to identify the location (for example, it can be represented by a rectangular box) and type s of the target object S. Extract the pixel coordinates P'(x, y) of the target object S, that is, the coordinates of the center point or upper left corner of the rectangular box.

[0136] S63, calculating the world coordinates P (X, Y, Z) of the target object S through the relationship P'=MP.

[0137] The transformation matrix M and the pixel coordinates P'(x, y) have been obtained through the above steps, so according to the relationship P'=MP, the world coordinates P(X, Y, Z) of the target object S can be calculated.

[0138] A mapping relationship between a two-dimensional video sequence and a three-dimensional scene is established, and the three-dimensional world coordinates are calculated through two-dimensional pixel coordinates to locate the target.

[0139] Step S7, twinning the three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene.

[0140] In this embodiment, step S7, generating a three-dimensional model of the twin target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene may further include the following steps:

[0141] S71, construct a three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene.

[0142] Laser point cloud modeling uses point cloud data obtained through laser scanning to construct three-dimensional models through modeling software (such as AutoCAD Civil3D, SketchUp, etc.).

[0143] Oblique photography modeling uses the image data obtained by oblique photography technology to generate a 3D model through a 3D reconstruction algorithm (such as triangulation, voxel-based methods, etc.). This method can quickly generate a 3D model of a large area scene.

[0144] Texture mapping is an important step to enhance the realism of 3D models. By mapping pre-processed texture data (such as photos, patterns, etc.) onto the surface of the 3D model, it makes it look more realistic. The specific steps of texture mapping include:

[0145] Texture mapping maps texture data onto the surface of a 3D model to ensure that the correspondence between the texture and the model surface is correct.

[0146] Texture adjustment: adjust the texture's size, position, rotation angle and other parameters to make it fit the model surface perfectly.

[0147] Texture optimization, including detail enhancement, color correction and other processing, can improve the realism and visual effects of textures.

[0148] S72, optimizing and adjusting the three-dimensional model of the target object S.

[0149] In order to improve the accuracy of the 3D model, a series of optimization work can be performed. The specific steps include:

[0150] Model lightweighting: By building multi-level and multi-layered instantiated and non-instantiated caches to generate pyramid cache data after scene data scheduling optimization, model data lightweighting can be quickly achieved and model rendering efficiency can be improved.

[0151] Detail enhancement: Enhance the details of the model, such as adding texture details, optimizing geometric structures, etc., to make the model more refined.

[0152] Error correction: By comparing with actual data, the errors in the model are corrected to improve the accuracy of the model.

[0153] Position adjustment refers to placing the constructed 3D model at the corresponding position P (X, Y, Z) of the 3D scene. The specific steps include:

[0154] Coordinate transformation: transform the model's coordinate system into the scene coordinate system to ensure the model's position in the scene is correct.

[0155] Fine-tune the position of the model according to actual needs to make it consistent with other elements in the scene.

[0156] Rotation and scaling: Rotate and scale the model according to the needs of the scene to make it conform to the visual effect of the scene.

[0157] It can be foreseen that when the video sequence processing speed is fast enough, the detected target types are sufficient and correct, and the world coordinates of the target objects are solved accurately enough, the virtual and real worlds can be synchronized with digital twins.

[0158] Model digital twinning is to place the target at the location of the target in a three-dimensional scene and twin a three-dimensional model of the same type as the target.

[0159] In some optional implementations of this embodiment, the target object S includes but is not limited to: a person, a car, a ship, etc.

[0160] Through the above process, the digital twin of the people, vehicles, objects and other targets of interest in the two-dimensional video image can be synchronously copied in the high-precision three-dimensional scene based on the high-precision space and low-latency time dimensions. For example, after the digital twin of the person of interest from the video surveillance in the above way, the behavior, movement trajectory, etc. of one or more persons can be restored synchronously in the three-dimensional world, and the characteristics of the three-dimensional real-life map can be added to achieve corresponding monitoring and early warning of the persons of interest, such as people entering restricted areas, people violating traffic rules, etc.; various types of vehicles can also be synchronously digitally twinned in this way in scenes such as cities and highways, and speed analysis, direction analysis, trajectory prediction, etc. can be performed, and then illegal and irregular behaviors such as speeding / low-speed driving and illegal occupation of lanes can be recorded and predicted.

[0161] In some optional implementations of this embodiment, after step S7, the electronic device may further perform the following steps:

[0162] S8, interactive experience of the three-dimensional model of the twinned target object S.

[0163] In order to provide a user-friendly interactive experience, interactive design can be performed on the 3D model. The specific steps include:

[0164] Interaction logic design: design the interaction logic between users and 3D models, such as clicking, dragging, zooming and other operations.

[0165] Interface design, designing interface elements used to control 3D models, such as buttons, sliders, menus, etc.

[0166] Animation design, adding animation effects to 3D models, such as rotation, scaling, movement, etc., to enhance the user's sense of immersion.

[0167] According to the interactive experience design, realize the relevant functions of the 3D model. The specific steps include:

[0168] Interaction logic and interface elements can be implemented using programming languages ​​(such as C++, Java, Python, etc.) and 3D engines (such as Unity, Unreal Engine, etc.).

[0169] Test the interactive experience functions to ensure that they function correctly and perform well. Optimize based on the test results to improve the user experience.

[0170] Integration and deployment: Integrate the implemented functions into the 3D scene and deploy them to ensure that users can get a consistent interactive experience on any device.

[0171] Figure 4 This is a detailed flow chart of converting the video target digital twin to a three-dimensional scene used in the camera calibration digital twin method of the present invention. Figure 4 As shown in the figure, the detailed process of converting the video target digital twin to a three-dimensional scene includes the following steps:

[0172] S101, creating a dictionary.

[0173] The video frame queue dictionary (video_container) of the device is stored, the queue dictionary (detected_info_queue_dict) of the monitoring information is stored, and the dictionary (det obj list dict) of the device detection target list is stored.

[0174] S102, example initialization.

[0175] Read the configuration file to initialize the device list: getDevlnfoFromini().

[0176] S103, instantiate a device tree.

[0177] Instantiate the device tree and obtain the attribute parameters of the corresponding device: init Device Tree().

[0178] S104, initializing the model.

[0179] Initialize TrtYOLO and deepsort models.

[0180] S105, creating a video window.

[0181] Create a video window on the main page: Create Qlabel list().

[0182] Then you can choose to go to step S106 or step S108.

[0183] S106: Create a queue.

[0184] A queue for storing mot-processed video images (mot_frame_queue) and a queue for storing video images with people (person frame queue).

[0185] Proceed to step S107.

[0186] S107, initializing related threads.

[0187] Initialize the video interruption restart thread: videoInterruptRestrartThresd, initialize the personnel information sending thread: sendPersonidInfoThread, initialize the personnel information obtaining thread: getPersonIDTread, and initialize the tracking thread: mot thread.

[0188] Proceed to step S110.

[0189] S108, acquiring a video image.

[0190] The video image of each device is acquired, and then the process goes to step S109.

[0191] S109, initializing the target detection thread.

[0192] Initialize the target detection thread detectTRTFrameThread.

[0193] S110, start the thread.

[0194] Enable the previewAll thread function: enable the target detection thread (detectTRTFrameThread), enable the video interruption restart thread (videoInterruptRestartThread), enable the send person information thread (sendPersonidInfoThread), enable the get person information thread (getPersonIDTread), and enable the tracking thread (mot thread).

[0195] S111, start coordinate conversion.

[0196] Start the coordinate conversion module to convert pixel coordinates into corresponding world coordinates in real time.

[0197] The implementation of this embodiment has the following beneficial effects: dynamic environmental information can be obtained through the video sequence captured by the camera; a three-dimensional map containing world coordinates is then established, providing a reference framework for subsequent calibration work; carefully selecting n calibration objects on the three-dimensional map and obtaining their precise world coordinates is the key to ensuring calibration accuracy; then the real counterparts of these calibration objects are accurately detected in the video sequence, and their pixel coordinates are obtained, providing necessary data for calculating the transformation matrix M; using n groups of virtual and real corresponding points, a 3×4 transformation matrix M is solved, which is a bridge connecting the pixel coordinates in the video sequence and the three-dimensional world coordinates. Once the transformation matrix M is determined, the target object can be detected in the video sequence, and its coordinates in the three-dimensional world can be calculated through the transformation matrix. Finally, a three-dimensional model of the target object is twinned at the corresponding position of the three-dimensional scene, realizing accurate mapping from video to three-dimensional scene; improving the accuracy of target positioning; realizing real-time interaction between video sequences and three-dimensional scenes; providing strong support for the application of digital twin technology; at the same time, through precise three-dimensional modeling, a solid foundation is laid for subsequent simulation, analysis and optimization work.

[0198] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0199] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0200] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0201] Embodiment 2

[0202] Further references Figure 5 , as a response to the above Figure 1 The present invention provides an embodiment of a camera calibration digital twin device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0203] like Figure 5 As shown, the camera calibration digital twin device 80 of this embodiment includes: an acquisition module 81, a building module 82, a selection module 83, a detection module 84, a first calculation module 85, a second calculation module 86 and a twin module 87. Among them:

[0204] An acquisition module 81 is used to acquire a video sequence by a camera;

[0205] A building module 82 is used to build a three-dimensional map of the area, and the three-dimensional map carries world coordinates;

[0206] The selection module 83 is used to select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X, Y, Z), i = 1, ... n;

[0207] Detection module 84, used to detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y);

[0208] The first calculation module 85 is used to have a 3×4 transformation matrix M such that Q i '=MQ i , calculate the transformation matrix M through n sets of virtual and real corresponding points;

[0209] The second calculation module 86 is used to detect the target object in the video sequence, obtain the type S and pixel coordinates P'(x, y) of the target object, and calculate the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP;

[0210] The twinning module 87 is used to twin the three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene.

[0211] The beneficial effects of implementing this embodiment are: improving the accuracy of target positioning; realizing real-time interaction between video sequences and three-dimensional scenes; providing strong support for the application of digital twin technology; and at the same time, through precise three-dimensional modeling, laying a solid foundation for subsequent simulation, analysis and optimization work.

[0212] Embodiment 3

[0213] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0214] The above-mentioned computer device 9 includes a memory 91, a processor 92, and a network interface 93 which are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 9 having components of a memory 91, a processor 92, and a network interface 93, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0215] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0216] The memory 91 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 9. Of course, the memory 91 may also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 91 is generally used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions of the camera calibration digital twin method, etc. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or are to be output.

[0217] The processor 92 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 92 is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to run computer-readable instructions or process data stored in the memory 91, such as computer-readable instructions for running the camera calibration digital twin method.

[0218] The network interface 93 may include a wireless network interface or a wired network interface. The network interface 93 is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0219] The beneficial effects of implementing this embodiment are: improving the accuracy of target positioning; realizing real-time interaction between video sequences and three-dimensional scenes; providing strong support for the application of digital twin technology; and at the same time, through precise three-dimensional modeling, laying a solid foundation for subsequent simulation, analysis and optimization work.

[0220] Embodiment 4

[0221] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the camera calibration digital twin method as described above.

[0222] The beneficial effects of implementing this embodiment are: improving the accuracy of target positioning; realizing real-time interaction between video sequences and three-dimensional scenes; providing strong support for the application of digital twin technology; and at the same time, through precise three-dimensional modeling, laying a solid foundation for subsequent simulation, analysis and optimization work.

[0223] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the various embodiment methods of the present invention.

[0224] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific embodiments, or to perform equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A camera calibration digital twin method, characterized in that: The steps include: Acquiring a video sequence by the camera; Establishing a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates; Select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X,Y,Z), i=1,2...,n; Detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y); There exists a 3×4 transformation matrix M such that Q i '=MQ i , calculate the transformation matrix M through n sets of virtual and real corresponding points; Detecting a target object in the video sequence, obtaining the type S and pixel coordinates P'(x, y) of the target object, and calculating the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP; The three-dimensional model of the twin target object S at the corresponding position P(X,Y,Z) of the three-dimensional scene.

2. The camera calibration digital twin method according to claim 1, characterized in that: The step of establishing a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates, specifically comprises: Collect satellite imagery and elevation data of the area; Processing the satellite images and elevation data; Modeling and texture mapping are performed on the processed satellite images and elevation data; Import geographic data with world coordinates.

3. The camera calibration digital twin method according to claim 1, characterized in that: The method selects n calibration objects Q1, Q2, ..., Q i , ..., Q n , the world coordinates of n calibration objects are Q i (X, Y, Z), i = 1, ... n, the steps specifically include: Select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n ; Get the world coordinates of n calibration objects as Q respectively i (X,Y,Z), i=1,2...,n; Store the world coordinates of n calibration objects as Q i (X,Y,Z).

4. The camera calibration digital twin method according to claim 1, characterized in that: The n calibration objects Q are detected in the video sequence. i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i The steps of '(x,y) specifically include: Extract n calibration objects Q from the three-dimensional map i The reference color feature of the corresponding reference image; Comparing the sequence color feature of the video sequence with the reference color feature to determine similarity; If they are similar, it means that n calibration objects Q are detected i The actual corresponding calibration object Q i ', and obtain the corresponding calibration object Q i 'Pixel coordinates Q i '(x,y).

5. The camera calibration digital twin method according to claim 1, characterized in that: There exists a 3×4 transformation matrix M such that Q i '=MQ i , through n sets of virtual and real corresponding points, the steps of calculating the transformation matrix M specifically include: Set n = 6; There exists a 3×4 transformation matrix M such that Q i '=MQ i , According to Q i and Q i 'Calculate the transformation matrix M.

6. The camera calibration digital twin method according to claim 1, characterized in that: The steps of detecting the target object in the video sequence, obtaining the type S and pixel coordinates P'(x, y) of the target object, and calculating the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP specifically include: Preprocessing the video sequence; Detect the target object S and identify the position of the target object S, i.e., the pixel coordinates P'(x, y) and the type s; The world coordinates P (X, Y, Z) of the target object S are calculated using the relationship P'=MP.

7. The camera calibration digital twin method according to any one of claims 1 to 6, characterized in that: The step of twinning the three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene specifically includes: Construct a three-dimensional model of the target object S at the corresponding position P (X, Y, Z) of the three-dimensional scene; Optimize and adjust the three-dimensional model of the target object S.

8. A camera calibration digital twin device, characterized in that: include: An acquisition module, used for acquiring a video sequence by the camera; A building module, used to build a three-dimensional map of the area, wherein the three-dimensional map carries world coordinates; A selection module is used to select n calibration objects Q1, Q2, ..., Q on the three-dimensional map. i , ..., Q n , the world coordinates of n calibration objects are Q i (X, Y, Z), i = 1, ... n; A detection module, used to detect n calibration objects Q in the video sequence i The actual corresponding calibration object Q i ', get the corresponding calibration object Q i 'The pixel coordinate is Q i '(x,y); The first calculation module is used to calculate the existence of a 3×4 transformation matrix M such that Q i '=MQ i , calculate the transformation matrix M through n sets of virtual and real corresponding points; A second calculation module is used to detect a target object in the video sequence, obtain the type S and pixel coordinates P'(x, y) of the target object, and calculate the world coordinates P(X, Y, Z) of the target object S through the relationship P'=MP; The twin module is used to twin the three-dimensional model of the target object S at the corresponding position P (X, Y, Z) in the three-dimensional scene.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the camera calibration digital twin method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the camera calibration digital twin method according to any one of claims 1 to 7.

Images