Distributed real-time three-dimensional space reconstruction system and method

Abstract

The invention discloses a distributed real-time three-dimensional space reconstruction system and a distributed real-time three-dimensional space reconstruction method, which can acquire and reconstruct a large-range three-dimensional scene in real time, focus on real-time performance and large range, and are the main difference from the traditional three-dimensional reconstruction method. Coded and compressed three-dimensional data including spatial position data and color image data are transmitted on a network in real time in a high-frame-rate streaming mode, and only color data are transmitted in a traditional method, or the spatial position data acquired at a single time are transmitted according to frames; the installation position of the camera is flexible, the camera can be installed in an area where data needs to be collected according to requirements, and the calibration method supports rapid deployment of equipment; the data collected by the discrete cameras are combined in real time, and are subjected to time synchronization and three-dimensional space splicing to form a whole piece of large-range real-time changing three-dimensional scene data, while two-dimensional images collected by the traditional cameras are independent, and space splicing does not exist; the three-dimensional scene data supports real-time three-dimensional interaction and display and is translated, rotated and zoomed according to a user instruction; and the data which are aligned and spliced in real time in the three-dimensional space are still recorded, stored and played back according to the three-dimensional format.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a distributed real-time three-dimensional space reconstruction system and method. Background Art

[0002] In the prior art, one of the typical methods for three-dimensional space reconstruction is to solve extreme constraints on multiple color images. For example, the solution in Reference 1 can calculate a three-dimensional model from two-dimensional image information. However, this three-dimensional model obtained by reasoning rather than directly through three-dimensional space perception is often inefficient and the modeling accuracy is not high enough. A 3D camera can directly obtain spatial depth information, and use a color sensor and depth sensor calibration method similar to Reference 2 to spatially align the color image and the depth image, and quickly reconstruct a color three-dimensional model within the perceptible area.

[0003] However, the inventors found that in the process of real-time 3D space reconstruction, due to the limited field of view of a single camera, for large objects and large-scale scenes, it is often necessary to use multiple sets of 3D cameras at the same time to expand the camera coverage. In addition, since the camera can only reconstruct the front surface of the object in the field of view, the back and sides of the object are inevitably missing, and multiple sets of cameras must be used to reconstruct in real time at different positions and then splice them together.

[0004] After searching, reference 3 discloses a 3D panoramic shooting system, in which multiple 3D image acquisition units are arranged around the shooting area, and color images containing color image information and depth images containing depth image information are respectively obtained. Based on the overlapping areas of adjacent acquisition units, the data collected by the multiple 3D image acquisition units are spliced ​​to generate a 3D panoramic image.

[0005] The inventor found that the solution has the following shortcomings:

[0006] 1. All acquisition units are required to be arranged equidistantly toward the center point, which is not applicable when the observed scene is further expanded or multiple objects need to be observed simultaneously.

[0007] 2. This technology uses real-time camera positioning and data stitching based on image processing and feature point matching, which requires that adjacent cameras must have overlapping fields of view. However, in cases of terrain restrictions, scattered scenes, etc., such camera layout conditions may not be met, making this method ineffective.

[0008] 3. This technology uses a local processing architecture that first stitches and processes locally and then transmits and sends. However, when the distance between cameras is far or the distance between the camera and the local server is far, the direct connection between the camera and the server based on the data cable will fail, and transmission must be carried out with the help of network equipment. The working mode of processing first and then transmitting cannot be used, and the working mode of transmitting first and then processing must be adopted.

[0009] In addition, when using multiple 3D cameras for real-time reconstruction, the core technologies of real-time three-dimensional space reconstruction, such as synchronous acquisition between cameras, synchronous processing of 3D data, comprehensive management and configuration of cameras, and real-time adjustment of reconstruction frame rate, all have certain problems or shortcomings, and the number of cameras is difficult to increase on a large scale.

[0010] Reference 1, Li Wenjing. Research on multi-camera calibration and 3D reconstruction technology[D]. Xi'an University of Technology, 2022.DOI:10.27398 / d.cnki.gxalu.2022.000869.

[0011] Reference 2, Patent Publication Number, CN111311689B, Patent Name, A method and system for calibrating relative external parameters of laser radar and camera.

[0012] Reference 3, Patent Publication Number, CN108616742B, Patent Name, A 3D Panoramic Shooting System and Method. Summary of the invention

[0013] The present invention provides a distributed real-time 3D space reconstruction system and method, which can deploy and position multiple 3D cameras at any location for large-scale scenes or large-size objects, and reconstruct the physical space covered by real-time 3D reconstruction. The 3D cameras can be managed in real time through a network, and the reconstruction data can be transmitted and the 3D model can be presented, thus getting rid of the limitations of real-time 3D reconstruction on scene terrain, camera spacing, transmission distance, etc.

[0014] To achieve the above-mentioned object, in a first aspect, the present invention provides a distributed real-time three-dimensional space reconstruction system, the system comprising a server node and a plurality of edge computing nodes, each edge computing node comprising a three-dimensional camera and an edge computing device; wherein,

[0015] Each edge computing node is used to:

[0016] Collecting color depth information through a three-dimensional camera, where the color depth information includes color information and depth information;

[0017] The color depth information collected by the corresponding three-dimensional camera is spatially aligned and time-synchronized in real time through the edge computing device; and the color depth information after the spatial alignment and time synchronization is encoded and sent to the server node in real time;

[0018] Server nodes are used to:

[0019] After receiving the color depth information sent by each edge computing node, it is decoded in real time, and after decoding, point cloud data is generated based on the decoded color depth information; the point cloud data can be rendered by the server node or the terminal node to obtain the corresponding three-dimensional model.

[0020] On the other hand, the present invention provides a distributed real-time three-dimensional space reconstruction method, which is applied to the above system, and the method comprises:

[0021] Each edge computing node collects color depth information through a three-dimensional camera; the color depth information includes color information and depth information;

[0022] Each edge computing node performs spatial alignment and time synchronization processing on the color depth information collected by the corresponding 3D camera in real time through the edge computing device, and encodes the color depth information after the spatial alignment and time synchronization processing and sends it to the server node in real time;

[0023] After receiving the color depth information sent by each edge computing node, the server node decodes it in real time and generates point cloud data based on the decoded color depth information; the point cloud data can be rendered by the server node or the terminal node to obtain the corresponding three-dimensional model.

[0024] Beneficial effects of the present invention:

[0025] 1. Real-time acquisition and reconstruction of large-scale three-dimensional scenes, focusing on real-time and large-scale, is the main difference from traditional three-dimensional reconstruction methods.

[0026] 2. Real-time high frame rate streaming transmission of encoded and compressed 3D data on the network, including spatial position data and color image data. Traditional methods only transmit color data, or only transmit single-collected spatial position data frame by frame.

[0027] 3. The camera installation position is flexible. The camera can be installed in the area where data needs to be collected according to needs, and the calibration method supports the rapid deployment of equipment.

[0028] 4. The data collected by discrete cameras are combined in real time, and after time synchronization and three-dimensional spatial stitching, a whole block of three-dimensional scene data that changes in real time over a large range is formed. The two-dimensional images collected by traditional cameras are independent of each other and there is no three-dimensional spatial stitching.

[0029] 5. The data aligned and spliced ​​in real time in three-dimensional space are still recorded, stored and played back in three-dimensional format. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the installation assembly of a tracker, a tracker base, a rigid intermediate connector, and a camera in one embodiment;

[0031] Figure 2 A schematic diagram of an embodiment in which a tracker and its base are mechanically mounted on a rigid intermediate connector;

[0032] Figure 3 A schematic diagram of an embodiment in which a tracker and its base are mounted on a rigid intermediate connector by magnetic attraction;

[0033] Figure 4 A schematic diagram of a scene in which a camera and a tracker mounted thereon are calibrated with a coordinate system by means of a combination of a planar marker and the tracker in one embodiment;

[0034] Figure 5 A flowchart of coordinate system calibration of a camera and a tracker mounted thereon in one embodiment;

[0035] Figure 6 A schematic diagram of positioning a single local tracker in one embodiment;

[0036] Figure 7 A flowchart of a single local tracker positioning in one embodiment;

[0037] Figure 8 A flowchart of global tracker extension and camera coordinate system global unification in one embodiment;

[0038] Fig.9A A schematic diagram of a global tracker expansion in one embodiment;

[0039] Fig. 9B A schematic diagram of a global tracker expansion in one embodiment;

[0040] Fig. 9C A schematic diagram of a global tracker expansion in one embodiment;

[0041] Fig.9D A schematic diagram of a global tracker expansion in one embodiment;

[0042] Fig.9E A schematic diagram of a global tracker expansion in one embodiment;

[0043] Fig.10 It is a structural schematic diagram of a distributed real-time three-dimensional space reconstruction system in one embodiment;

[0044] Fig.11 It is a structural block diagram of a distributed real-time three-dimensional space reconstruction system in one embodiment;

[0045] Fig.12 A topological structure diagram of module 2 in one embodiment;

[0046] Fig.13 A topological structure diagram of module 3 in one embodiment;

[0047] Fig.14 A topological structure diagram of module 4 in one embodiment;

[0048] Fig.15 is a topological structure diagram of module 5 in one embodiment;

[0049] Fig.16 is a topological structure diagram of module 5 in one embodiment;

[0050] Fig.17 is a topological structure diagram of module 5 in one embodiment;

[0051] Fig.18 A topological structure diagram of module 6 in one embodiment;

[0052] Fig.19 is a topological structure diagram of module 6 in one embodiment;

[0053] Fig. 20 A topological structure diagram of module 6 in one embodiment;

[0054] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0055] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0056] The present invention involves two major inventive concepts, the first is a distributed real-time three-dimensional space reconstruction system and method, and the other is a multi-camera spatial positioning method and system with a large range and non-overlapping field of view.

[0057] First, the distributed real-time 3D space reconstruction system provided by the present invention is described. In one embodiment, a distributed real-time 3D space reconstruction system is provided, the system comprising a plurality of 3D cameras, an edge computing device respectively deployed at each 3D camera, a network communication device, a server node, and a terminal node; wherein,

[0058] A three-dimensional camera is used to collect color depth information of the covered scene, where the color depth information includes color information and depth information;

[0059] The edge computing device is located next to the 3D camera and directly establishes a data connection with the 3D camera. It has information acquisition, data processing, and network communication functions and is used for:

[0060] Establish a device management channel with the server node, receive instructions sent by the server node to adjust the operating parameters of the 3D camera, and send the camera configuration to the server node;

[0061] Perform spatial alignment and temporal synchronization processing on the color depth information collected by the corresponding 3D camera in real time;

[0062] Receive the instruction sent by the server node to adjust the processing method of the color depth information collected by the three-dimensional camera;

[0063] Establish a data transmission channel with the server node, encode the color depth information after spatial alignment and time synchronization, and send it to the server node in real time;

[0064] Network communication equipment used to establish network connections between edge computing devices, server nodes, and terminal devices.

[0065] Server nodes, which can be local or remote computing servers, are used to:

[0066] Establishing a network connection with the terminal node and the edge computing device, sending the control information received from the terminal node to the edge computing device, or sending the three-dimensional camera operating parameters and the edge computing device operating status received from the edge computing device to the terminal node;

[0067] After receiving the color depth information sent by each edge computing device, it is decoded in real time, and after decoding, three-dimensional color point cloud data is generated based on the restored color depth information;

[0068] Rendering is performed based on the point cloud data to generate a three-dimensional model and the three-dimensional model is sent to each terminal for display;

[0069] The 3D model file is stored when recording the 3D scene, or the 3D model is sent to each terminal for display during playback.

[0070] Endpoints for:

[0071] Receive the 3D model sent by the server node and render it for display;

[0072] Receive user operation instructions and send them to the server, thereby regulating edge computing devices and 3D cameras;

[0073] Receive user interaction instructions for the presented 3D model and respond.

[0074] like Fig.10 and Fig.11 As shown, in the distributed real-time three-dimensional space reconstruction system described in the present invention, one of the three-dimensional cameras and one of the edge computing devices next to it together constitute an edge computing node.

[0075] The edge computing node is the basic deployment unit of the acquisition unit. When deployed, the edge computing node is set around the area to be reconstructed. There is no need to set a field of view overlapping area for the 3D camera, and there is no need to consider the physical distance between the edge computing node and the server node and the terminal node.

[0076] The network equipment includes communication equipment such as switches and routers, which establish mutual connections between edge computing nodes, server nodes and terminal nodes. After the network connection is set up, the edge computing node does not need to obtain the specific network address information of the server node, but needs to obtain the network where the server node is located to register its own connection information; the terminal node needs to obtain the specific network address information of the server node to directly connect to the server node.

[0077] The terminal node can be independent of the server node or integrated into the server node, and the functional settings and data transmission logic of the two remain unchanged.

[0078] Next, the method corresponding to the distributed real-time three-dimensional space reconstruction system provided by the present invention is described. The method is implemented by dividing into 6 software modules, and each module will be introduced one by one below to facilitate those skilled in the art to understand the method corresponding to the distributed real-time three-dimensional space reconstruction system provided by the present invention.

[0079] Module 1: 3D camera spatial position calibration module

[0080] The following multi-camera spatial positioning method with a large range and non-overlapping field of view is used to obtain the relative position relationship of each camera relative to the reference camera. The reference camera is the camera that provides the basic reference coordinate system among the cameras. The relative position relationship is the 4×4 rotation and translation matrix of other cameras relative to the reference camera, which describes the azimuth distance and rotation posture of each camera relative to the reference camera. The specific content of the multi-camera spatial positioning method with a large range and non-overlapping field of view can be found below and will not be repeated here.

[0081] Module 2: Color depth information synchronous acquisition and synchronous reception module

[0082] like Fig.12 As shown, module 2 is used to synchronize the color depth information collected by multiple 3D cameras to the same time base.

[0083] The synchronous acquisition and reception of color depth information is to control multiple 3D cameras to acquire color depth information of the physical scene at the same time node or an approximate time node, the edge computing node sends the color depth information with timestamps, and the server node receives and rearranges the color depth information with timestamps to the same time reference.

[0084] When controlling multiple 3D cameras to collect color depth information of physical scenes at the same time node or similar time nodes, first select a network time server to provide a unified time reference accurate to milliseconds. Each edge computing node and server node are connected to the network time server to obtain a unified time reference and keep their system clock synchronized with the network server clock to ensure event synchronization and consistency of data collection. When the color depth information collection event is triggered, each edge computing node receives the collection signal and collects color depth information at the same time node, so that the color depth information collection time of each 3D camera is the same or approximately the same.

[0085] Before the edge computing node sends the color depth information, it adds the corresponding acquisition time as a timestamp to the collected color depth information, compresses the data as a whole using lossless coding, and then each edge computing node sends the encoded information to the server node.

[0086] After the data transmission channel is opened, the server node starts to receive the color depth information with timestamps sent by all edge computing nodes at the same time. After the color depth information of each 3D camera enters the server node from different communication threads, it is temporarily stored in the data queue corresponding to the edge computing node. The server node parses the timestamp corresponding to the color depth information from the received data, and extracts all the 3D camera color depth information with the same or approximately the same acquisition time point from all received data queues according to the acquisition time point, so that the color depth information of all 3D cameras to be processed is in the same time base, with the same or similar acquisition time, and then decodes the originally compressed color depth data for splicing and fusion of three-dimensional space data. The time server can be a remote standard time server, or it can be a server node or an edge computing node.

[0087] The trigger event for collecting color depth information can be an information collection signal set at a fixed time interval, corresponding to automatically collecting color depth information of the physical scene at a fixed frame rate, serving the needs of high-frequency real-time three-dimensional reconstruction, or it can be an instantaneous collection signal sent by the terminal device, corresponding to a single collection of color depth information of the physical scene, serving the needs of instantaneous real-time three-dimensional reconstruction.

[0088] Module 3: 3D spatial data splicing and fusion module

[0089] like Fig.13 As shown, module 3 is used to convert the color depth information of each camera into spatial 3D point cloud data in real time and stitch and fuse them under a unified coordinate reference.

[0090] On each camera data processing thread on the server node, the decoded color depth information is processed and converted respectively. After decoding the color depth information, the two-dimensional color and depth images are first preprocessed, including but not limited to using filtering algorithms to reduce measurement noise on depth images and using image processing algorithms to improve image quality on color data. Subsequently, a stereo matching algorithm is used to extract high-precision three-dimensional point clouds from the color and depth data. Each point has independent color and spatial position information, and 3D point cloud data corresponding to each 3D camera is generated. Under the same reference coordinate system, the external parameter matrix corresponding to each camera can be extracted from the external parameter matrix list of the camera calibration result and set to the corresponding processing thread. According to the external parameter matrix of each 3D camera, the spatial position coordinates of all point cloud data corresponding to each 3D camera are transformed respectively, so that the local physical scenes collected by all 3D cameras can be mapped to the corresponding positions in the virtual space, and the 3D point cloud data in the coverage area of ​​each camera can be spliced ​​to form a larger reconstruction space.

[0091] The splicing and fusion of the three-dimensional spatial data is executed cyclically at a certain frame rate to perform real-time three-dimensional reconstruction of the physical scene.

[0092] Although the 3D point cloud data corresponds to each 3D camera, they all use the same color space description method, and their color data values ​​remain consistent in the color data collected by different cameras.

[0093] When the 3D point cloud data is generated from the color depth data, the point cloud position coordinates are based on the respective camera coordinate system. After the camera extrinsic matrix transformation, the 3D point cloud data corresponding to all 3D cameras are transformed to the same coordinate space based on the coordinate system when the camera is calibrated.

[0094] The point cloud data generation and coordinate position conversion are distinguished by 3D cameras, and the color and depth information received from each 3D camera is processed in an independent thread.

[0095] Module 4 3D point cloud data synchronous rendering module

[0096] like Fig.14 As shown, module 4 is used to adjust the processing and rendering frame rate of the 3D point cloud data corresponding to each camera, so that each camera displays the point cloud data at the same or similar frame rate.

[0097] The data processing threads corresponding to each camera on the server node have different processing frame rates due to the impact of data volume, network latency, and uneven allocation of computing resources. That is, there are differences in the number of times per second the color depth data is received, decoded, 3D point cloud is generated, and coordinate changes are processed. This causes the overall performance of the point cloud to show inconsistent reconstruction speeds in various local areas.

[0098] The terminal node can set a specific processing frame rate setting for each camera thread on the server node through specific configuration, and adjust the processing and rendering frame rate of the 3D point cloud data corresponding to each camera according to the actual situation, so that the processing speed of each camera thread tends to be the same or similar.

[0099] After setting a specific frame rate for a camera processing thread, the overall processing time T0 corresponding to the frame rate can be calculated first, that is, when the thread keeps processing each frame of data at T0, the corresponding 3D point cloud data can be ensured to be rendered in real time according to the frame rate. After the processing of receiving color depth data, decoding, generating 3D point cloud and coordinate change corresponding to the current frame is completed, the time T1 consumed by the current frame for the above processing can be counted. When T0 is greater than T1, it indicates that the data processing speed of the current thread is too fast. Set T0-T1 as the waiting time interval of the current frame. The thread is paused at this time interval. After the waiting time is over, the corresponding point cloud data is submitted to the rendering device for screen generation. When T0 is less than T1, it indicates that the processing speed of the current thread is too slow and cannot reach the target frame rate. The target frame rate can be reduced and the frame rate can be synchronously updated to the data processing threads of all cameras. At the same time, the thread wait is no longer set, and the corresponding point cloud data is directly submitted to the rendering device for screen generation.

[0100] When calculating the waiting time interval according to the above process, if all processing threads need to wait, it indicates that the set target frame rate is too low and the target frame rate can be gradually increased.

[0101] Module 5 Network connection and data transmission module

[0102] like Fig.15 , 16 As shown in FIG. 17 , module 5 is used to manage the 3D camera and transmit color depth information through the network.

[0103] The edge computing node has communication and network connection capabilities. After starting the network connection, based on the network communication device, it registers its own network address, port and other device connection information to the network where the server node is located, and waits for the connection signal from the terminal node.

[0104] The terminal node has communication and network connection capabilities. After starting the network connection, it sends a connection signal to the server node based on the network communication device to establish a network connection between the terminal node and the server node. The server node then forwards the connection signal of the terminal device to each edge computing node to establish a network connection between the edge computing node and the terminal node.

[0105] Based on the network connection between the edge computing node and the terminal node, the edge computing node is managed through the terminal device to establish a device management channel. Specifically, the edge computing node sends the configuration information of the 3D camera and the edge computing device to the outside, and the information is forwarded to the terminal node via the server node, displayed to the user on the terminal device and receives the user's edits. After modifying the configuration information of the edge computing node, the terminal node sends the updated configuration information to the outside, and the information is forwarded to the edge computing node via the server node, and then the edge computing node updates the 3D camera and edge computing device configuration.

[0106] Based on the network connection between the edge computing node and the terminal node, the edge computing node sends the color depth information to the server node in real time and sends it to the terminal device for display after processing, so as to establish a data transmission channel. Specifically, after collecting the color depth information through the data synchronization acquisition module, each edge computing device aligns the color depth information to the same time reference, uses a lossless compression algorithm to encode and compress the color depth information, and sends the compressed data to the server node through the network. The server node decodes the original data to obtain the color depth information of each camera, and generates a 3D point cloud rendering image through the splicing and fusion module of the three-dimensional spatial data, and then sends the image to the terminal node. After receiving the 3D point cloud rendering image, the terminal node can interact with the point cloud data presented on the terminal node by collecting user interaction instructions, and can also collect the device configuration information changed by the user, and reset some edge computing node parameters that can be configured in the real-time data stream through the above-mentioned device management channel.

[0107] In the color depth data transmission, the two data streams of color and depth are also supported to be transmitted separately. The color data stream is compressed by the encoder, transmitted to the server node through the network, and then decoded and restored to the original color data at the server node. The depth data does not pass through the encoder, and the lossless data is directly transmitted to the server node through the network. The two data streams transmit data in parallel, but timestamps are added to the data streams. After receiving the separated color depth data, the server node extracts the color depth information collected at the same or adjacent time according to the timestamp, and merges them into one frame of data to achieve the true restoration of the reconstruction of the three-dimensional scene point cloud.

[0108] The data transmission access needs to be created after the device management channel is established, and is started when the user sends a start data transmission command from the terminal node to the edge computing node through the device management channel. Each transmission completed by this channel corresponds to the complete processing of one frame of data from the 3D camera. In real-time 3D reconstruction, this data transmission channel needs to remain normally and continuously running until it is closed when the user sends a stop data transmission command from the terminal node to the edge computing node through the device management channel.

[0109] When the server node is connected to the edge computing node, the device management channel and the data transmission channel both use multi-threaded parallel methods. Each edge computing node establishes an independent network connection with the server node, and sends data to the server node or receives data from the server node at the same time.

[0110] When the server node is connected to the terminal node, the device management channel and the data transmission channel both use a single connection method. The device configuration and status information are transmitted serially between the two in a sequential manner. The color depth information sent by each camera is spliced ​​and processed into a 3D point cloud rendering image and transmitted to the terminal node.

[0111] The network connection between the edge computing node and the cloud / local server, and the network transmission between the cloud / local server and the terminal node, can use a combination of wireless transmission and wired transmission according to the visual working environment, and cooperate with switches, routers and other devices to establish network connections. In a specific embodiment, the implementation technology of the network transmission includes but is not limited to one or more of the following combinations: Ethernet cable, optical communication, Wi-Fi, 4G / 5G, Bluetooth, ZigBee, etc.

[0112] In a specific embodiment, the network connection between the preferred edge computing node and the cloud / local server is implemented in two ways. First, each edge computing node is connected to the device-side network switch via a local Ethernet cable and / or Wi-Fi, and then connected to the cloud / local server via an Ethernet cable / optical communication. Second, each edge computing node is directly connected to the cloud server via a 4G / 5G network.

[0113] In a specific embodiment, the network transmission between the preferred cloud / local server and the terminal node is implemented in two ways. One is that each terminal node is connected to the terminal-side network switch via a local Ethernet cable and / or Wi-Fi, and then connected to the cloud / local server via an Ethernet cable / optical communication method. The other is that each terminal node is directly connected to the cloud server via a 4G / 5G network.

[0114] Module 6: Recording and playback module for three-dimensional spatial data

[0115] like Fig.18 , 19 As shown in FIG. 20 , module 6 is used to record real-time three-dimensional reconstruction data and play back the recorded three-dimensional reconstruction data.

[0116] In the real-time 3D space reconstruction system of the present invention, the 3D reconstruction data can be displayed in real time and can also be synchronously stored in the file system and the stored reconstruction data can be played back. The storage of the 3D reconstruction data is based on the recording event as the basic description unit, and each recording event includes a recording event information, the configuration file of each camera, and the color depth file of each camera.

[0117] The recording event information is a general description of the recording event, including at least the start time, end time, recording duration, encoding method, etc. of the recording event, as well as the number of cameras, the storage location of the camera configuration file, the storage location of the camera color depth file, etc. This information can be used as a search entry for each recording event and is stored in a searchable database in a table format with the recording start time point as the primary key.

[0118] The camera configuration file is the configuration and parameters of the 3D camera, including at least resolution, acquisition frame rate, internal parameter matrix, external parameter matrix, etc. The configuration information of each camera is unique. The file is stored in a dedicated file storage system, and the storage path is recorded in the corresponding recording event.

[0119] The camera color depth file is a video file composed of the camera color depth information in time series, corresponding to the three-dimensional information of the physical scene collected by the camera during operation. The color depth file of each camera is unique. The file is stored in a dedicated file storage system, and the storage path is recorded in the corresponding recording event.

[0120] When the main thread of the server node receives the start recording signal sent by the terminal node, it creates a recording event with the current time as the start time of the recording event. After receiving the color depth information sent by the edge computing node, each camera processing thread stores the color depth information received in the file storage system according to the file naming rules, and then performs subsequent processing on the color depth data.

[0121] When the main thread of the server node receives the stop recording signal sent by the terminal node, it obtains the configuration information of all cameras from the edge computing node and saves it to the corresponding location of the file system. The current time is used as the end time point of the recording event, and the recording event information such as recording duration and encoding method is completed. The entire path of the video file and camera configuration file is added, and the 3D reconstruction recording event is saved to the database. After receiving the color depth information sent by the edge computing node, each camera processing thread no longer stores the color depth information and directly performs subsequent processing.

[0122] The recording of the real-time 3D reconstruction data is based on the premise of opening the data transmission channel of the real-time 3D reconstruction system, and the completion of the recording does not affect the continued operation of the reconstruction system.

[0123] With respect to the recording of real-time three-dimensional reconstruction data, the real-time three-dimensional reconstruction system described in the present invention also supports the playback of recorded three-dimensional reconstruction data. After reading the recorded event sequence from the database, the terminal node can sort the recorded events according to the recording start time, allowing the user to select the event to be played back, and send the recorded event playback command and the event start time to the server node. After the server node receives the playback recording event command and the event start time, it queries the corresponding recording event information from the recording event database according to the event start time, and then reads the configuration information of each camera from the file system, starts the processing thread corresponding to each camera according to the number of cameras, and sets the camera configuration information to the processing thread corresponding to the camera. Each thread starts to read the color depth information sequence from the corresponding color depth file, and performs subsequent point cloud stitching processing and point cloud image rendering until the color depth file is read.

[0124] The recording event can be retrieved from the recording event database according to the event start time, or a specific recording event can be deleted from the recording event database according to the event start time, and the camera configuration file and the camera color depth file can also be deleted from the file system.

[0125] The recording event playback does not require the opening of edge computing nodes, the opening of device management channels, or the connection of terminal nodes and edge computing nodes. On the data transmission channel, the edge computing nodes do not need to be started, and the corresponding camera processing threads on the server nodes receive color depth information from the edge computing nodes, and instead read the color depth information sequence with timestamps from the corresponding camera files, and the subsequent processing remains unchanged.

[0126] Next, the multi-camera spatial positioning method with a large range and non-overlapping field of view according to the present invention is described.

[0127] In order to better understand the technical contribution of this method, it is necessary to describe the background of the inventor's invention:

[0128] Computer vision is based on the physical environment information obtained by the camera. It can identify, track and segment people and objects in the scene with high precision, serve the needs of environmental perception, scene understanding, intelligent interaction, etc., create huge economic and social benefits, and is a typical feature of the new generation of information technology. Among them, three-dimensional (3D) cameras have spatial perception capabilities and can obtain relative depth information of objects in the scene while perceiving color images. Therefore, they are widely used in three-dimensional scenes such as three-dimensional reconstruction, object segmentation, and spatial tracking. Its typical use method is to superimpose the 3D physical scene information obtained by multiple cameras according to their actual spatial positions and postures, supplement the field of view of a single camera from different directions, and expand the perceptible scene range through data splicing.

[0129] In this process, one of the key steps is to obtain the relative position and attitude information between cameras, that is, to obtain the position and attitude of the local coordinate system of all cameras in a unified global coordinate system. The more commonly used method is binocular positioning. By setting identifiable markers in the overlapping field of view between each two cameras, the relative position and attitude relationship between the camera and the marker is obtained based on the image processing method, thereby calculating the relative position and attitude relationship between each two cameras, and obtaining the relative position and attitude relationship between all cameras through coordinate system transfer. However, the premise of using this method is that there must be overlapping fields of view between cameras. It is not only inefficient, but also fails in scenes with large-scale non-overlapping fields of view, such as discrete distribution of cameras, long distances, and large viewing angle deviations.

[0130] After searching, the existing multi-camera calibration solutions without overlapping field of view can be roughly divided into the following three technical routes:

[0131] The first method is to put multiple calibration plates into the same coordinate system to obtain the transformation relationship between the coordinate systems and realize the transfer of coordinate systems between cameras with non-overlapping fields of view, as described in reference 4, using the calibration method of multiple small calibration plates. First, the Zhang Zhengyou method is used to calibrate the intrinsic parameters of each camera; then, the relative position relationship of the small calibration plates is measured using a total station, and the small calibration plates are unified into the same coordinate system, integrated into a large calibration plate, and the pose relationship of each camera relative to the integrated calibration plate is calculated respectively; finally, the coordinate system is unified to obtain the extrinsic parameters between multiple cameras.

[0132] The second method is to move the cameras so that they can all capture the same target to obtain the transformation relationship between the coordinate systems, thereby realizing the transfer of coordinate systems between cameras with non-overlapping fields of view. As described in Reference 5, a plane calibration plate and a two-axis turntable are fixed, and multiple cameras are fixed on the two-axis turntable. A world coordinate system is established with the upper left corner of the target as the origin. Multiple images are obtained at multiple positions using a single camera through the rotation of the turntable itself, and the relationship between the world coordinate system and the turntable coordinate system is determined; secondly, the turntable is rotated so that the target enters the field of view of each camera respectively to complete the external parameter calibration of the single camera; finally, the relative position relationship between each camera is solved based on the relationship between the world coordinate system and the turntable coordinate system, and the relationship between the world coordinate system and the coordinate system of each camera.

[0133] The third method is to introduce an auxiliary camera to shoot all calibration plates to obtain the transformation relationship between the coordinate systems and realize the transfer of coordinate systems between cameras with non-overlapping fields of view. The principle is as described in document 6, and the rotation matrix and translation vector between each camera to be calibrated and the auxiliary camera are obtained respectively.

[0134] Through the analysis of the non-overlapping field of view multi-camera calibration technology in the existing technical routes, the inventors found that the three routes in the existing technology require each camera to take a photo of the calibration plate or target separately, and then use the photos to determine the intrinsic and extrinsic parameters of each camera by means of, for example, the Zhang Zhengyou calibration method, and then use the internal and external parameters of each camera to construct a transformation matrix to obtain the coordinate system transformation relationship between each camera, that is, to achieve the transfer of coordinate systems between cameras with non-overlapping fields of view (for a detailed explanation of the specific formula principle, please refer to Chapter 3 of Reference 1 on multi-camera calibration). However, when the camera constructs the transformation matrix between the coordinate systems of multiple cameras using the internal and external parameters of each camera, there are the following deficiencies:

[0135] 1. Before performing multi-camera calibration, each camera must be calibrated separately to record the intrinsic and extrinsic parameters of each camera. Then these cameras can be combined into a new system to perform multi-camera calibration without a common field of view. During the calibration process, the position and posture of each camera must be consistent with that during individual calibration. When the camera position and posture change, the external parameters will change. It is necessary to recalibrate the camera separately to determine the new external parameters, which causes inconvenience.

[0136] 2. Due to processing reasons, lens distortion includes radial distortion, tangential distortion, centrifugal distortion and thin prism distortion. In the calibration process of a single camera, based on the complexity of the algorithm, only the radial distortion of the lens is generally considered, while other distortion parameters are ignored. For a single camera, the distortion parameters have little effect on subsequent 3D reconstruction, object segmentation, spatial tracking and other tasks. However, in multi-camera calibration, the distortion error of the camera farthest from the camera of the reference coordinate system will be accumulated and amplified in the coordinate system transfer operation. For example, if the camera numbered 1 is used as the reference camera and there are 40 cameras, the coordinate system conversion process of the 40th camera to the reference camera needs to rely on the parameters of 38 cameras from No. 2 to No. 39 for coordinate transfer, resulting in the distortion error of the 40th camera being accumulated and amplified. At the same time, if these 40 cameras use different calibration plates for separate calibration. It may also lead to poor consistency of parameters obtained from different camera calibrations. These poorly consistent parameters will bring uncertainty in the coordinate system transfer in multi-camera calibration. For example, if the calibration accuracy of camera No. 5 is poor, then the cameras that need to be converted into the reference coordinate system through the parameters of camera No. 5 will have the accuracy of the tutorial. In general, when the intrinsic and extrinsic parameters of the camera are used as the parameters of the coordinate transfer matrix, the errors carried in the calibration process may be propagated to the multi-camera calibration system.

[0137] Reference 4, Wang Anran, Hao Xiangyang, Cheng Chuanqi, Jia Kaikai. A multi-camera external parameter calibration method using multiple small calibration plates [J]. Surveying and Mapping and Spatial Geographic Information, 2019, 42(06): 222-225+229.

[0138] Reference 5, Lu Yanan, Wan Zijing, Wang Xiangjun. A method for solving the position relationship of cameras without a common field of view [J]. Applied Optics

[0139] Science, 2017, 38(03): 400-405.

[0140] Reference 6, Patent Publication Number, CN116188591A, Patent Name, Multi-camera global calibration method and device, electronic equipment.

[0141] To this end, the present invention provides a multi-camera calibration system and method with non-overlapping fields of view, which can sequentially obtain the position and attitude parameters of the local coordinate system of each camera in a common unified coordinate system for large-scale non-overlapping fields of view such as discrete distribution of multiple cameras, long distance, and large viewing angle deviation. It is also possible to eliminate the need to calibrate the internal and external parameters of all cameras separately before calibrating the multiple cameras, making it convenient to change the position and attitude of some cameras during the multi-camera calibration process and reducing the risk of propagation of internal and external parameter errors of some cameras in the multi-camera system.

[0142] This method consists of four steps, including the installation and positioning of the tracker on the camera, the calibration of the tracker coordinate system and the camera coordinate system, the local tracker positioning, and the global tracker expansion and global unification of the camera coordinate system.

[0143] It should be noted that since a 3D camera is an integration of a visible light image sensor (i.e., camera) and a depth sensor (such as a laser scanner or a structured light scanner), in the multi-camera spatial positioning method with a large range and non-overlapping field of view provided in the following embodiments, in order to facilitate the calibration between the cameras, each 3D camera is actually regarded as a camera with only a visible light image sensor (i.e., camera). The following calibration method does not involve the calibration between the visible light image sensor (i.e., camera) and the corresponding depth sensor on each 3D camera. The calibration of the two can refer to reference 2, which is not an improvement point of this application and will not be repeated here. Therefore, the following 3D camera taking pictures of the chessboard refers to taking pictures using a visible light image sensor.

[0144] The step one is to install and position the tracker on the three-dimensional camera.

[0145] The tracker is provided with a plurality of signal receiving devices in different directions on its outer surface. When used in conjunction with a base station that transmits signals outward, the position and attitude information of the local coordinate system of the tracker relative to the base station coordinate system can be obtained according to the distribution of the intensity of the received signal, thereby forming a corresponding position and attitude matrix. The bottom can be a regular plane structure such as a regular rectangle or circle, and can be provided with regular positioning structures such as square holes, round holes, V-shaped grooves, and threaded holes. The center position of the bottom is the origin of the coordinate system during spatial tracking. Two of the three coordinate axes are located on the bottom surface, and the third axis is perpendicular to the bottom surface, forming a local coordinate system on the tracker. The position and attitude information obtained during spatial tracking is the pose matrix of the coordinate system relative to the base station coordinate system.

[0146] The signal emitted by the base station is dispersed outward from one point, with a fixed coverage angle and range, and uses its own coordinate system as a reference to establish a real-time tracking relationship with the tracker in the observable area. After adjusting the observation position and observation angle, the base station re-establishes the reference coordinate system at its location and establishes a tracking relationship with the tracker in the new observable area.

[0147] The tracker is provided with a special base in a regular rectangular structure, which is precisely combined with the tracker through the positioning and connection structure at the bottom of the tracker. The rectangular edges of the base are parallel to the coordinate axis of the tracker. After the connection, the two are no longer separated and are connected to the outside as a whole.

[0148] The tracker base may be provided with positioning holes, positioning pins and other structures. Except for the parts connected to the tracker, the other structures are not restricted, and the positioning and connection structures may be specially designed according to the external connection method.

[0149] The overall structure of the tracker and the base can be positioned on the 3D camera with the help of a rigid intermediate connector with a special shape. The rigid intermediate connector is first positioned on the outer surface of the 3D camera, and then the tracker and the base are positioned and connected to the rigid intermediate connector as a whole, so that the tracker and the 3D camera have a fixed relative position relationship.

[0150] The rigid intermediate connecting piece has a structure adapted to the appearance of the three-dimensional camera and is pre-fixed on each three-dimensional camera by means of plane, end face contact positioning and hole axis constraint positioning.

[0151] The positioning of the tracker and the base as a whole on the rigid intermediate connector can be mechanically performed by end face positioning and screw tightening. The positioning method can also be magnetic, with positioning pins and magnets provided on the tracker base, positioning holes and opposite magnets provided at corresponding positions of the rigid intermediate connector, and the positioning is performed by hole-shaft matching and the connection is performed by magnet adsorption.

[0152] The tracker and the base are integrally connected and fixed on the rigid intermediate connector, and are a detachable structure, which can be installed when the three-dimensional camera position needs to be obtained, and can be disassembled after the position is obtained.

[0153] The second step is to calibrate the tracker coordinate system and the three-dimensional camera coordinate system.

[0154] The 3D camera coordinate system is a reference coordinate system when the 3D camera collects depth information (by first spatially aligning the depth image and the color image, the depth information collected by the depth sensor can be converted to the 3D camera coordinate system). It is also a local coordinate system when 3D physical scenes are stitched together. It is defined on the color image sensor (i.e., visible light image sensor) of the 3D camera.

[0155] The calibration of the tracker coordinate system and the three-dimensional camera coordinate system is to obtain the relative position and posture relationship between the tracker coordinate system and the three-dimensional camera coordinate system after the tracker and the base are positioned and installed on the rigid intermediate connector as a whole. The specific method is to set a combination of a plane marker and a tracker whose relative position and posture are known, and the base station simultaneously obtains the coordinates of the tracker on the combination and the tracker on the three-dimensional camera, and obtains the position and posture of the tracker installed on the three-dimensional camera relative to the local coordinate system of the plane marker through coordinate system conversion. After the three-dimensional camera obtains its position and posture relative to the local coordinate system of the plane marker through image recognition, the position and posture of the three-dimensional camera coordinate system relative to the local coordinate system of the tracker installed thereon can be obtained through coordinate system change.

[0156] The planar marker and tracker assembly comprises a rectangular planar marker that can be located and identified by color image and a tracker and base assembly placed at the center of the marker. The rectangular planar marker defines its own local coordinate system at its center, and the coordinate axes of the tracker coordinate system are parallel to the coordinate axes of the marker local coordinate system. The two differ only by the height of the tracker base in the vertical direction of the planar marker. The relative transformation matrix between the two coordinate systems can be obtained by measuring the height of the tracker base.

[0157] The plane marker may be a regular chessboard, an ellipse, or any image with a complex texture, and the relative position relationship between the plane marker and the three-dimensional camera may be obtained through a feature point recognition method.

[0158] The relative position and attitude relationship between the tracker coordinate system and the three-dimensional camera coordinate system is consistent and universal when the shapes of the tracker base, the rigid intermediate connector and the three-dimensional camera do not change. When the same model of tracker and its base, the rigid intermediate connector and the three-dimensional camera are used for connection, the relative position and attitude relationship between the tracker coordinate system and the three-dimensional camera coordinate system does not change, and the position and attitude parameters of one coordinate system can be used to calculate the position and attitude parameters of the other coordinate system.

[0159] The step three is local tracker positioning.

[0160] The local tracker positioning is to use the base station to locate the trackers within its observable area once and obtain the coordinates of all trackers. The specific method is to adjust the position and observation angle of the base station so that at least two three-dimensional cameras are within its observable range, install trackers for all three-dimensional cameras in the observable area, and directly obtain the position and attitude parameters of all trackers in the observable area, that is, the local coordinates relative to the base station coordinate system. Subsequently, the local coordinates of all observed trackers are converted into relative coordinates relative to the coordinate system of the known object in this observation, and further calculate the coordinates of all trackers observed in this observation in the global reference coordinate system based on the global coordinates of the known object, and all trackers observed in this observation are recorded as new known trackers.

[0161] The known object is an object whose global coordinates have been obtained before a local positioning. When the local positioning is performed for the first time, it is the base station during the observation. When the local observation is performed later, it is the first known tracker within the observation range. In each local positioning, the local coordinates of the known object in the observation are obtained. When the base station is used as the known object, the position and rotation parameters of its local coordinates and global coordinates are both 0, and the position and attitude matrices are unit matrices.

[0162] The coordinate system of the known object is a local coordinate system established by the base station or the tracker itself.

[0163] The global reference coordinate system is the coordinate system of the base station when the local positioning is performed for the first time. The position and attitude parameters of the tracker observed this time are the global coordinates of the tracker, and there is no need to perform relative coordinate transformation.

[0164] The step 4 is to globally unify the global tracker extension and the 3D camera coordinate system.

[0165] The global tracker expansion is to adjust the observation position and observation angle of the base station in turn to change its observable area, and gradually expand until all trackers are covered. The specific method is to repeat the local tracker positioning method multiple times, and adjust the position and observation angle of the base station each time, so that the new observable area covers at least one known tracker except for the first positioning, until the position and attitude parameters of all trackers are obtained.

[0166] The global tracker expansion process can be performed simultaneously using more than one base station, or the number of three-dimensional cameras that can be located by the base station at a time can be increased by increasing the number of trackers.

[0167] The local tracker positioning and global tracker expansion require at least one base station and at least two trackers regardless of the number of 3D cameras to be positioned.

[0168] After the tracker becomes a known tracker, the tracker and the base can be removed as a whole from the rigid intermediate connecting member.

[0169] After the base station changes the observation position and the observation angle, the removed tracker that is not in the observable area can be installed on the three-dimensional camera in the current observable area.

[0170] When the number of trackers is insufficient to be installed on all 3D cameras covered by one local positioning, the process can be split into multiple local positioning processes, with trackers installed on only a small number of 3D cameras for positioning each time.

[0171] When the distance between the three-dimensional cameras is too far so that the observable area of ​​the base station cannot cover the trackers of at least two three-dimensional cameras, one or more new relay trackers are set in the middle of the two three-dimensional cameras, so that the base station can change its position and observation angle so that its observable area covers the tracker and at least one relay tracker installed on the three-dimensional camera when it is near the location of the three-dimensional camera, and covers at least two relay trackers at the same time when it is in the middle of the two three-dimensional cameras. Through multiple position transmissions of the relay tracker, according to the local tracker positioning and global tracker expansion method, the coordinate relationship of the trackers on multiple long-distance three-dimensional cameras in the same coordinate system is established.

[0172] It can be found that the transfer between the coordinate systems of each 3D camera in the present invention is not the conversion matrix constructed by the internal and external parameters obtained by calibrating each 3D camera separately, but the conversion matrix constructed by the tracker and base station fixed on the 3D camera. Therefore, it is not necessary to calibrate the internal and external parameters of all 3D cameras separately before multi-camera calibration, which is convenient for changing the position and posture of some 3D cameras during the multi-camera calibration process, and reduces the risk of propagation of the internal and external parameter errors of some 3D cameras in the multi-camera system. In addition, in this method, each new tracker performs coordinate conversion with the known tracker, and the base station only provides a transfer function for the coordinate system conversion between the two, so there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the difficulty of implementation.

[0173] On the other hand, assuming that there are 40 3D cameras that need to be calibrated without field of view, when the tracker is fixed to the 30th 3D camera, the positions and postures of the following 10 3D cameras No. 31-40 can be adjusted. There is no need to re-calibrate individually after adjustment before continuing the multi-camera calibration as in the prior art. Therefore, the solution of the present invention is more efficient in multi-camera calibration.

[0174] When the number of 3D cameras is small or the distance is close, the positioning of the trackers on all 3D cameras can be completed through a local positioning of the base station. The local tracker positioning process becomes a global positioning process, and the global tracker expansion is no longer performed.

[0175] According to the calibration results of the tracker coordinate system and the three-dimensional camera coordinate system, the position and attitude of the three-dimensional camera coordinate system relative to the local coordinate system of the tracker installed thereon is a fixed constant, and the position and attitude information of each three-dimensional camera in the global coordinate system can be obtained through the global coordinates of the tracker installed on the three-dimensional camera.

[0176] The method can further set the local coordinate system of one of the three-dimensional cameras as a reference, and through coordinate transformation, change the position and attitude parameters of all three-dimensional cameras into relative coordinates relative to the local coordinate system of the three-dimensional camera, and the position and angle values ​​of the three-dimensional camera become 0, and the position and attitude matrices are unit matrices.

[0177] The method may also introduce an additional reference coordinate system, change the positions and postures of all three-dimensional cameras into relative values ​​under the reference coordinate system, and perform an overall transformation on the collected spatial position data.

[0178] This embodiment is aimed at the spatial positioning of multiple 3D space three-dimensional cameras in a large range without overlapping fields of view, but is also applicable to the spatial positioning of more than two ordinary 2D planar three-dimensional cameras, and the spatial positioning of more than two 3D cameras (i.e., three-dimensional cameras) at close range with overlapping fields of view.

[0179] Before implementing this embodiment, the installation positions of multiple 3D cameras have been determined according to the requirements, and there is no need to set overlapping field of view areas for the three-dimensional cameras in advance when designing the installation positions. According to this embodiment, the intervals between the three-dimensional cameras can be set to be infinite and face different observation angles.

[0180] When implementing this embodiment, the required operations are generally divided into four stages, namely, installation and positioning of the tracker on the 3D camera, calibration of the tracker coordinate system and the 3D camera coordinate system, local tracker positioning, and global tracker expansion and global unification of the 3D camera coordinate system.

[0181] The first implementation phase is the installation and positioning of the tracker on the 3D camera.

[0182] refer to Figure 1 The tracker 102 and the tracker base 104 are installed and combined into a tracker and base whole 106, the rigid intermediate connecting piece 108 and the three-dimensional camera 110 are installed and combined into a three-dimensional camera and connecting piece whole 112, which is further combined into an overall structure 114 including the tracker and the three-dimensional camera. The whole is used in conjunction with the tracking base station 116 to obtain the spatial position and posture of the overall structure 114 relative to the base station 116.

[0183] Tracker 102 has multiple signal receiving devices in different directions on its outer surface. Base station 116 actively transmits specific signals to a certain area. When tracker 102 is within the observable range of base station 116, each signal receiving device calculates the position and attitude information of the local coordinate system of tracker 102 relative to the coordinate system of base station 116 in real time based on the distribution of signal perception strength, forming a corresponding position and attitude matrix. After adjusting the observation position and observation angle, base station 116 will re-establish the reference coordinate system at the new position and establish a tracking relationship with the tracker in the new observable area.

[0184] In some embodiments, the tracker 102 can also actively transmit signals in multiple directions and the base station 116 can sense the signal strength and distribution to obtain the relative positions of the two. The signal receiving device on the tracker 102 can also be set to multiple markers that can independently track the XYZ spatial coordinates. The base station 116 sequentially obtains the XYZ spatial coordinates of at least three markers on the tracker 102, and then obtains the spatial position and posture of the tracker 102 after combining. The electromagnetic tracking technology composed of the tracker and the base station belongs to the conventional technical means in this field and will not be repeated here. The specific implementation can refer to the following patent documents: CN107452036A-A global optimal optical tracker posture calculation method, CN102374847B-Workspace six-degree-of-freedom posture dynamic measurement device and method. It should be noted that the present invention uses optical positioning technology, which has higher accuracy than UWB (ultra-wideband technology) and can reach sub-millimeter level. Among them, in the prior art, such as CN102374847B-Workspace Six-DOF Position and Attitude Dynamic Measurement Equipment and Method, it is necessary to calculate the rotation angle of the transmitter (i.e., the base station) in order to settle the position and attitude of each tracker. In the present invention, each tracker that newly enters the current observation range is subjected to coordinate transformation with the known tracker, and the base station only provides a transit role for the coordinate system transformation between the two. Therefore, there is no need to set up a special module to control and record the rotation angle of the base station, which can reduce the difficulty of implementation.

[0185] In this embodiment, the bottom of the tracker 102 is a regular rectangular structure, the center of the bottom is the origin of the coordinate system during space tracking, two of the three coordinate axes are located on the bottom surface, and the third axis is perpendicular to the bottom surface, forming the local coordinate system of the tracker 102. The position and posture information obtained during space tracking is the pose matrix of the coordinate system relative to the coordinate system of the base station 116. The dedicated base 104 corresponding to the bottom structure of the tracker 102 is a regular rectangular parallelepiped structure, and the edges are parallel to the coordinate axes of the tracker 102. The two are accurately combined on the contact surface through positioning and connection structures to form a tracker and base as a whole 106, which will not be split thereafter and connected to the outside through the base 104 as a whole.

[0186] In this embodiment, a rigid intermediate connector 108 is used to position the overall structure 106 of the tracker 102 and the base 104 on the 3D camera 110. The rigid intermediate connector 108 has a structure that matches the appearance of the 3D camera 110, and is pre-fixed on the 3D camera 1120 by means of side plane and end face contact positioning, forming a 3D camera and connector overall 112, which does not affect the installation and normal use of the 3D camera 110 on the external support structure.

[0187] The tracking device and the base assembly 106 are positioned on the rigid intermediate connection 108 by using Figure 2The mechanical method of end face positioning and screw tightening is shown. In this method, the tracker and base assembly 106 and the rigid intermediate connector 108 are provided with four pairs of corresponding positioning holes, and the positioning holes of the rigid intermediate connector 108 are provided with threads. After the two are aligned, the four bolts 202 are tightened. This method can realize the rapid installation and removal of the tracker and base assembly 106 on the 3D camera and connector assembly 112. This positioning can also be adopted Figure 3 The magnetic attraction method shown in the figure. In this method, three positioning pins 302 and an annular magnet 306 are set on the tracker base, and three positioning holes 304 and an annular magnet 308 are correspondingly set on the rigid intermediate connecting member 108. The positioning holes 304 and the positioning pins 302 are used to position the two, and the adsorption force between the magnets 306 and 308 is used to connect and fix the two.

[0188] In this embodiment, the tracker and the base assembly 106 may be positioned on the rigid intermediate connector 108 using either of two methods, both of which have the same positioning effect and do not affect the three-dimensional camera positioning method and the final effect.

[0189] The tracker and base assembly 106 is only installed on the 3D camera and connector assembly 112 when the position of the 3D camera 110 needs to be acquired, and is disassembled after the position of the 3D camera 110 is acquired. The tracker base 106 and the rigid intermediate connector 108 need to have high processing accuracy and can be freely combined between different instances.

[0190] In some embodiments, the structural design of the tracker base 104 and the rigid intermediate connector 108 is not restricted except for the areas that are in direct contact with the tracker 102 and the three-dimensional camera 110. The matching appearance and positioning structures such as square holes, round holes, V-shaped grooves, and connection and force-bearing structures such as threaded holes can be designed according to needs.

[0191] The second implementation phase is the calibration of the tracker coordinate system and the 3D camera coordinate system.

[0192] After the tracker and the 3D camera are assembled into the overall structure 114, the tracker 102 and the 3D camera 110 have a fixed relative position relationship. Figure 4 The overall structure 114 of the tracker 102-1 and the 3D camera 110, and the combination of a planar marker 402 and the tracker 106 with a base are placed in the observable area of ​​the same base station 116. The overall structure 114 of the tracker and the 3D camera needs to adjust the corresponding posture so that the 3D camera 110 can collect the image of the planar marker 402 at a close distance and try to be located in the center of the image.

[0193] refer to Figure 5, the calibration of the coordinate system of the tracker 102-1 and the coordinate system of the 3D camera 110 may include steps 502 to 520. Steps 502 to 508 generate a coordinate transformation relationship between the tracker 102-2 and the planar marker 402, steps 510 to 512 generate a coordinate transformation relationship between the tracker 102-1 and the tracker 102-2 on the overall structure 114 of the tracker and the 3D camera, and steps 514 to 516 generate a coordinate transformation relationship between the 3D camera 110 and the planar marker 402.

[0194] Step 502 is to place the assembly 106 of the base 104B and the tracker 102-2 at the center of the planar marker 402, so that it is within the observable range of the base station 116, and the planar marker 402 is at the center of the 2D image of the 3D camera 110 and occupies most of the image area. In step 504, the height of the tracker base 104B is measured using a high-precision measuring tool, that is, the distance between the connection surface of the tracker base 104B and the tracker 102-2 to the contact surface of the tracker base 104B and the planar marker 402. In step 506, the position of the base 104B on the planar marker 402 is adjusted so that the coordinate axes of the coordinate system of the tracker 102-2 and the coordinate axes of the local coordinate system of the marker 402 are parallel to each other, and the difference between the two is only the height distance of the tracker base 104B in the vertical direction of the planar marker 402, which is the origin distance between the coordinate system of the tracker 102-2 and the coordinate system of the planar marker 402. In step 508, since the coordinate system of tracker 102-2 and the coordinate system of planar marker 402 differ only in the height distance of tracker base 104B in one coordinate axis direction, the transformation matrix between the two coordinate systems can directly change a corresponding translation coordinate into the height distance of tracker base 104B based on the unit matrix.

[0195] Step 510 is to simultaneously observe the tracker 102-1 and the tracker 102-2 on the overall structure 114 of the tracker and the three-dimensional camera through the base station 116 to obtain their respective spatial positions, and then in step 512, obtain the relative transformation matrix between the two trackers 102-1 and 102-2 through coordinate system transformation.

[0196] In another embodiment, in order to avoid the mechanical error of measuring the height of the tracker base 104B from affecting the conversion matrix between the plane marker coordinate system and the tracker coordinate system thereon, a 3D camera calibration method can be used. Specifically, the coordinates of each signal receiving device on the tracker in the tracker coordinate system are known, and the plane marker and the tracker thereon can be photographed by a 3D camera at the same time, and then the conversion relationship between the plane marker coordinate system and the plane marker coordinate system can be obtained by using the image feature point matching technology, and then the conversion matrix between the plane marker coordinate system and the tracker coordinate system thereon can be obtained by transferring the coordinate system.

[0197] In step 514, the 2D plane 3D camera of the 3D camera 110 captures the corresponding image, and simultaneously records the image resolution and its own internal parameters, and saves them as an electronic file. In step 516, the 3D camera 110 extracts the pixel coordinates of the marker point on the image of the plane marker 402 through image recognition, and obtains its position and posture relative to the coordinate system of the plane marker 402 through the mapping relationship between the 2D and 3D positions, that is, the relative transformation matrix between the coordinate system of the 3D camera 110 and the coordinate system of the plane marker 402.

[0198] In step 518, the coordinate transformation matrix between the tracker 102-2 and the plane marker 402, the coordinate transformation matrix between the tracker 102-1 and the tracker 102-2, and the coordinate transformation matrix between the 3D camera 110 and the plane marker 402 are integrated to calculate the coordinate transformation matrix between the tracker 102-1 and the 3D camera 110 on the overall structure 114 of the tracker and the 3D camera. In step 520, the obtained coordinate transformation matrix between the tracker 102-1 and the 3D camera 110 is stored as a known constant. When the coordinate position of any object between the tracker 102-1 and the 3D camera 110 is obtained, the coordinate position of the other object can be calculated.

[0199] In this embodiment, the plane marker 402 is composed of black and white squares of equal size arranged according to a certain rule, but in some embodiments, the plane marker 402 can also be a regular circle, ellipse, or any image with complex texture, and the relative position relationship with the three-dimensional camera can be obtained through the feature point recognition method.

[0200] In this embodiment, the relative position and posture relationship between the coordinate system of the tracker 102 and the coordinate system of the three-dimensional camera 110 is consistent and universal when the shapes of the tracker base 104, the rigid intermediate connector 108 and the three-dimensional camera 110 do not change. That is, after multiple trackers 102 and multiple three-dimensional cameras 110 are combined through the tracker base 104 and the rigid intermediate connector 108, the relative position between the selected tracker 102 and the three-dimensional camera 110 remains unchanged, and the coordinate transformation matrix remains unchanged.

[0201] In some embodiments, when the same model of tracker 102 and its base 104, rigid intermediate connector 108 and three-dimensional camera 110 are connected, the relative position and posture relationship between the tracker 102 coordinate system and the three-dimensional camera 110 coordinate system does not change, and the position and posture parameters of one coordinate system can be calculated using the position and posture parameters of the other coordinate system.

[0202] Implementation phase three is local tracker positioning.

[0203] In this embodiment, each time the base station 116 is placed, the tracker 102 installed on the 3D camera 110 in the observable area is observed and positioned once, which is a local tracker positioning process. The local tracker positioning process only locates and obtains the position of the tracker 102, and does not calculate the position of the 3D camera 110. The calculation is only performed after the corresponding trackers on all 3D cameras have completed positioning.

[0204] refer to Figure 6 , adjust the position and observation angle of the base station 116 so that at least two 3D cameras 110 are within its observable range, install the tracker and the base assembly 106 for all the 3D cameras 110 within the observable area to obtain the overall structure 114 including the tracker and the 3D camera, and directly obtain the position and posture parameters of all the trackers 102 in the observable area, that is, the local coordinates relative to the base station 116 coordinate system. The base station 116 then sends these position coordinates to the computer for storage and processing.

[0205] refer to Figure 7 , the operation and algorithm flow of a single local tracker positioning may include steps 702 to 722. Step 702 is to adjust the position and observation angle of the base station 116 so that at least two three-dimensional cameras 110 are within its observable range, and to install the tracker and the base unit 106 for all three-dimensional cameras 110. Step 704 is to simultaneously obtain the coordinates of each tracker 102 relative to the base station 116 coordinate system, and in step 706, send the coordinate data to the computer in real time for storage and processing. Step 708 is to determine the positioning progress. Steps 710 to 714 are performed when the local positioning is performed for the first time, and steps 716 to 720 are performed when the local positioning is not performed for the first time.

[0206] At step 710, the first local positioning is started. No tracker 102 has obtained the global coordinates yet, and the global coordinate system has not been defined. The base station 116 can be set as a known object and become the reference coordinate system for this observation. At step 712, the coordinate system of the base station 116 is defined as the global coordinate system, and at step 714, the coordinates of all trackers 102 in this observation are set as global coordinates.

[0207] In step 716, at least one local positioning has been performed and a global coordinate system has been established. Some trackers 102 have obtained global coordinates, and the coordinate system of the first known tracker 102 is selected as the reference coordinate system for this observation. In step 718, the relative position and posture of all other trackers 102 observed in this observation relative to the reference coordinate system can be obtained through coordinate transformation, and multiple coordinate transformation matrices can be obtained. In step 720, based on the obtained coordinates of other trackers 102 relative to the known tracker 102 coordinate system and the global coordinates of the known tracker 102, the global coordinates of all trackers 102 observed in this observation in the global reference coordinate system can be calculated.

[0208] Step 722 is to obtain the global coordinates of all trackers 102 observed by the base station 116 during the current positioning, record all trackers 102 observed this time as new known trackers 102, record the data and complete the local tracker positioning process.

[0209] In some embodiments, the base station 116 may simultaneously obtain the coordinates of multiple known trackers 102 during one observation, and may select one of the trackers 102 with known coordinates as a reference object for this positioning.

[0210] In some embodiments, a relay tracker is arranged between three-dimensional cameras that are far apart. At this time, the position of the base station 116 can be adjusted so that its observation range covers the relay tracker. The relay tracker only serves as a common positioning point for two local positioning.

[0211] The fourth implementation phase is the global tracker extension and global unification of the 3D camera coordinate system.

[0212] In this embodiment, the observation position and observation angle of the base station 116 are adjusted in sequence to change its observable area. In addition to the first positioning, the new observable area covers at least one known tracker. After multiple local tracker positioning processes, the observation area can be gradually expanded until it covers all trackers 102 on the three-dimensional camera. This process is a global tracker expansion process.

[0213] refer to Figure 8, the global tracker extension and the global unification of the 3D camera coordinate system include steps 802 to 810. After the base station position is updated according to step 702 so that its observation range covers at least two 3D cameras, the tracker outside the current observation area is removed in step 802 and installed on the 3D camera in the current observation area or as a relay tracker according to step 804.

[0214] After performing a local tracker positioning according to the process at 700, it is determined in step 806 whether the positioning of the trackers on all 3D cameras has been completed. When the positioning of the trackers on all 3D cameras has not been completed, steps 802 to 806 and 700 are repeated. When the positioning of the trackers on all 3D cameras has been completed, the coordinates of all 3D cameras in a unified global coordinate system are calculated in step 808 based on the relative conversion relationship between the 3D camera 110 and the tracker 102 installed thereon stored in step 520. In step 810, a certain 3D camera or an additional coordinate system can be set as a reference, and the coordinates of all 3D cameras can be converted to the new reference coordinate system, and the relative position relationship between each 3D camera remains unchanged.

[0215] Reference FIG. 9A to FIG. 9E , 3D cameras 110A, 110B, 110C, 110D, 110E, and 110F are the 3D cameras to be positioned. The 3D cameras are far apart and have no overlapping fields of view. Three trackers 102A, 102B, and 102C are used to perform multiple local tracker positioning through the global tracker expansion process, and the position coordinates of the corresponding trackers on all 3D cameras can be obtained.

[0216] refer to Fig.9A , the first local tracker positioning is performed on the 3D cameras 110A, 110B, and 110C. The base station 116 is located at position 1, and the observation range covers the 3D cameras 110A, 110B, and 110C. The trackers 102A, 102B, and 102C are installed on the 3D cameras accordingly, and the position coordinates of the trackers 102A, 102B, and 102C can be obtained at one time. In this local tracker positioning, the coordinate system of the base station 116 is the global coordinate system, and each tracker does not need to perform a global coordinate system conversion, and the trackers on the 3D cameras 110A, 110B, and 110C are marked as known objects.

[0217] refer to Fig. 9B, a local tracker positioning is performed on the 3D cameras 110C and 110D. The base station 116 is located at position 2, and the observation range covers the 3D cameras 110C and 110D. The tracker 102A on the 3D camera 110A is disassembled and reinstalled on the 3D camera 110D, and the position coordinates of the trackers 102C and 102A can be obtained at one time. In this local tracker positioning, the tracker 102C is a known object. After calculating the relative position coordinates of the tracker 102A relative to the tracker 102C, the global coordinates of the tracker 102A are calculated based on the global coordinates of the tracker 102C, and the tracker on the 3D camera 110D is marked as a known object.

[0218] refer to Fig. 9C Since the area of ​​3D camera 110D is far away from the area of ​​3D cameras 110E and 110F, the tracker 102B on 3D camera 110B is disassembled and placed near 3D camera 110D, and the tracker 102C on 3D camera 110C is disassembled and placed near 3D cameras 110E and 110F. Trackers 102B and 102C are set as relay trackers, and local positioning is performed on trackers 102A and 102B. Base station 116 is located at position 3, and its observation range covers tracker 102A on 3D camera 110D and relay tracker 102B, so the position coordinates of trackers 102A and 102B can be obtained at one time. In this local tracker positioning, tracker 102A is a known object. After calculating the relative position coordinates of tracker relay 102B relative to tracker 102A, the global coordinates of relay tracker 102B are calculated based on the global coordinates of tracker 102A, and relay tracker 102B is marked as a known object.

[0219] refer to Fig.9D , the first local positioning of the relay trackers 102B and 102C is performed. The base station 116 is located at position 4, and the observation range covers the relay trackers 102B and 102C, and their position coordinates can be obtained at one time. In this local tracker positioning, the relay tracker 102B is a known object. After calculating the relative position coordinates of the relay tracker 102C relative to the relay tracker 102B, the global coordinates of the relay tracker 102C are calculated based on the global coordinates of the relay tracker 102B, and the relay tracker 102C is marked as a known object.

[0220] refer to Fig.9E, a local tracker positioning is performed on the relay tracker 102C, the 3D cameras 110E, and 110F. The base station 116 is located at position 5, and the observation range covers the 3D cameras 110E, 110F, and the relay tracker 102C. The relay tracker 102B is disassembled and reinstalled on the 3D camera 110E, and the relay tracker 102A is disassembled and reinstalled on the 3D camera 110F, and the position coordinates of the trackers 102C, 102B, and 102A can be obtained at one time. In this local tracker positioning, the relay tracker 102C is a known object. After calculating the relative position coordinates of the trackers 102B and 102A relative to the relay tracker 102C, the global coordinates of the trackers 102B and 102A are calculated based on the global coordinates of the relay tracker 102C, and the trackers on the 3D cameras 110E and 110F are marked as known objects.

[0221] After calculating the positions of the trackers 102 installed on the three-dimensional cameras 110A, 110B, 110C, 110D, 110E, and 110F, based on the relative position conversion matrix between the three-dimensional camera 110 and the tracker 102 installed thereon, the coordinates of the three-dimensional cameras 110A, 110B, 110C, 110D, 110E, and 110F relative to the coordinate system of the base station at position 1 can be calculated, thereby completing the spatial positioning of multiple three-dimensional cameras at long distances and without overlapping fields of view.

[0222] In some embodiments, the local coordinate system of one of the three-dimensional cameras can be further set as the reference, and through coordinate transformation, the position and attitude parameters of all three-dimensional cameras are changed to relative coordinates relative to the local coordinate system of the three-dimensional camera, and the position and angle values ​​of the three-dimensional camera are all 0, and the position and attitude matrix is ​​the unit matrix. It is also possible to introduce an additional reference coordinate system, change the position and attitude of all three-dimensional cameras to relative values ​​under the reference coordinate system, and perform an overall transformation on the collected spatial position data.

[0223] In this embodiment, a base station 116 and three trackers 102 are used for 3D camera positioning. After the tracker 102 becomes a known tracker, the tracker and base unit 106 can be removed from the rigid intermediate connector and installed on a new 3D camera 110 to be observed or at a relay position. In some embodiments, regardless of the number of 3D cameras to be positioned, at least one base station and at least two trackers are required. The global tracker expansion process can be performed using more than one base station at the same time, or by increasing the number of trackers to increase the number of 3D cameras that can be positioned by the base station each time, thereby speeding up the positioning process.

[0224] In some embodiments, when the number of trackers is insufficient to be installed on all three-dimensional cameras covered by one local positioning, it can be split into multiple local positioning processes, with trackers installed on only a small number of three-dimensional cameras for positioning each time.

[0225] In this embodiment, when the distance between the 3D cameras 110 is relatively far, so that the observable area of ​​the base station 116 cannot cover the trackers 102 of at least two 3D cameras 110, one or more new relay trackers 102 are set in the middle of the two 3D cameras 110, so that the base station 116 can change its position and observation angle so that its observable area covers the tracker 102 and at least one relay tracker 102 installed on the 3D camera when it is near the position of the 3D camera 110, and covers at least two relay trackers 102 at the same time when it is in the middle of the two 3D cameras 110. Through multiple position transmissions of the relay trackers 102, the coordinate relationship of the trackers 102 on multiple long-distance 3D cameras 110 in the same coordinate system can be established.

[0226] In some embodiments, when the number of 3D cameras is small or the distance is close, the positioning of the trackers on all 3D cameras can be completed through a local positioning of the base station. The local tracker positioning process becomes a global positioning process, and the global tracker expansion and coordinate system are no longer performed.

[0227] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A distributed real-time three-dimensional space reconstruction system, characterized in that: The system includes a server node and a plurality of edge computing nodes, each edge computing node includes a three-dimensional camera and an edge computing device; wherein, Each edge computing node is used to: Collecting color depth information through a three-dimensional camera, where the color depth information includes color information and depth information; The color depth information collected by the corresponding 3D camera is spatially aligned and time-synchronized in real time through the edge computing device; the color depth information after spatial alignment and time synchronization is encoded and sent to the server node in real time; the server node is used to: After receiving the color depth information sent by each edge computing node, it is decoded in real time, and after decoding, point cloud data is generated based on the decoded color depth information; the point cloud data can be rendered by the server node or the terminal node to obtain the corresponding three-dimensional model.

2. The distributed real-time three-dimensional space reconstruction system according to claim 1, characterized in that: The system further includes a terminal node, wherein the terminal node is configured to: Rendering the received point cloud data to obtain a corresponding three-dimensional model and displaying the three-dimensional model; Or receive user operation instructions and send them to the server node, thereby regulating the edge computing device and 3D camera; Or receive user interaction instructions for the presented three-dimensional model and respond.

3. The distributed real-time three-dimensional space reconstruction system according to claim 1, characterized in that: The server node is also used for: Sending control information received from the terminal node to the edge computing device; wherein the control information is used to adjust the operating parameters of the three-dimensional camera; Or the three-dimensional camera operating parameters and the operating status of the edge computing device received from the edge computing device may be sent to the terminal node.

4. A distributed real-time three-dimensional space reconstruction method, characterized in that: Applied to the system according to any one of claims 1 to 3, the method comprising: Each edge computing node collects color depth information through a three-dimensional camera; the color depth information includes color information and depth information; Each edge computing node performs spatial alignment and time synchronization processing on the color depth information collected by the corresponding 3D camera in real time through the edge computing device, and encodes the color depth information after the spatial alignment and time synchronization processing and sends it to the server node in real time; After receiving the color depth information sent by each edge computing node, the server node decodes it in real time and generates point cloud data based on the decoded color depth information; the point cloud data can be rendered by the server node or the terminal node to obtain the corresponding three-dimensional model.

5. A distributed real-time three-dimensional space reconstruction method according to claim 4, characterized in that: Allowing the three-dimensional cameras to have no overlapping fields of view, the method further comprising: Based on a preset multi-camera calibration process with no overlapping fields of view, the relative position relationship of each 3D camera with respect to the reference camera is obtained; The reference camera is a 3D camera that provides a basic reference coordinate system among the 3D cameras; the relative position relationship is a transformation relationship between other 3D cameras and the reference camera, which describes the azimuth distance and rotation posture of other 3D cameras relative to the reference camera.

6. A distributed real-time three-dimensional space reconstruction method according to claim 4, characterized in that: The collecting of color depth information by a three-dimensional camera specifically includes: Control each 3D camera to collect color depth information at the same time node or a similar time node; The encoding of the color depth information after the spatial alignment processing and the temporal synchronization processing specifically includes: After adding a corresponding timestamp to the color depth information after the spatial alignment and time synchronization processing, the color depth information is encoded; the timestamp is the time point when the color depth information is collected; After the server node receives the color depth information sent by each edge computing node and before decoding, the method further includes: The server node rearranges each color depth information to the same time base based on the timestamp, so as to decode the color depth information at the same time base in the same batch; When color and depth data are transmitted separately, the color and depth data separated by each edge computing node must be merged.

7. A distributed real-time three-dimensional space reconstruction method according to claim 6, characterized in that: The server node rearranges each color depth information to the same time reference based on the timestamp, specifically including: The server node temporarily stores the color depth information received from each edge computing node in a data cache corresponding to each edge computing node; The server node parses the timestamp corresponding to each color depth information; The server node extracts all color depth information with the same or approximately the same acquisition time point from all data caches according to the timestamp, so that all color depth information that needs to be decoded in the same batch is at the same time reference.

8. A distributed real-time three-dimensional space reconstruction method according to claim 7, characterized in that: The generating point cloud data based on the decoded color depth information specifically includes: Create a corresponding camera processing thread for each 3D camera on the server node; On each camera processing thread, the decoded color depth information corresponding to each 3D camera is processed and converted respectively to generate point cloud data corresponding to the color depth information of each 3D camera in a unified standard system.

9. A distributed real-time three-dimensional space reconstruction method according to claim 8, characterized in that: The point cloud data corresponding to the color depth information collected by each 3D camera at the same time node or an approximate time node is used as the point cloud data belonging to the same frame. The method also includes a process of controlling each camera processing thread to submit the point cloud data belonging to the same frame to the main thread: Each camera processing thread determines the target processing time corresponding to each frame based on the target frame rate; wherein the target frame rate is the number of frames corresponding to the point cloud data that each camera processing thread needs to submit to the main thread within one cycle; and the target processing time is the estimated time required for each camera processing thread to generate the point cloud data corresponding to the frame; Real-time monitoring of the actual time taken by each camera processing thread to generate point cloud data corresponding to the same frame; If the actual processing time of a camera processing thread is less than the target processing time, a waiting time is set for the camera processing thread, and after the waiting time has passed, the camera processing thread submits the point cloud data to the main thread; the waiting time is the difference between the target time and the actual time; If the actual processing time of a camera processing thread is equal to the target processing time, the camera processing thread submits the point cloud data to the main thread; If the actual processing time of a camera processing thread is longer than the target processing time, the camera processing thread reduces the target frame rate so that the target processing time increases accordingly, and synchronizes the reduced target frame rate to other camera processing threads.

10. A distributed real-time three-dimensional space reconstruction method according to claim 4, characterized in that: The method further comprises: When the main thread of the server node receives the start recording signal sent by the terminal node, it uses the current time as the start time point of the recording event and saves the three-dimensional space reconstruction data starting from the start time point in the camera color depth file; While storing the camera color depth file, the camera configuration file of each 3D camera is stored at the corresponding position with the start time point as a unique marker; When the main thread of the server node receives the stop recording signal sent by the terminal node, it takes the current time as the end time point of the recording event and stores the camera color depth file recorded from the start time point to the end time point in the corresponding position of the database.

11. A distributed real-time three-dimensional space reconstruction method according to claim 4, characterized in that: The method further includes playing back the recorded three-dimensional reconstruction data, specifically including: The user selects the recording event to be replayed from the terminal node based on the recording start time, recording duration and other information, and sends the recording event playback command and the event start time to the server node; The server node queries the corresponding recording event information from the recording event database according to the event start time, and then reads the configuration information of each camera from the file system, starts the processing thread corresponding to each camera according to the number of cameras, and sets the camera configuration information to the processing thread corresponding to the camera; Each thread starts to read the color depth information sequence from the corresponding color depth file, and performs subsequent point cloud stitching processing and point cloud image rendering until the color depth file is read.

Images