Operation system, method, device and storage medium of TOF camera simulation system

Abstract

The application relates to a data simulation technology and discloses a TOF camera simulation system which comprises a transient rendering module, a TOF imaging simulation module and a TOF depth calculation module, wherein the transient rendering module is used for acquiring a scene file, camera pose and camera parameters and generating a transient illumination map according to the acquired data; the TOF imaging simulation module is used for obtaining a received modulation signal by using time information of the transient illumination map and a preset modulation signal function and calculating the received modulation signal to obtain an original phase map; and the TOF depth calculation module is used for analyzing the original phase map to obtain simulation real data, wherein the simulation real data comprises at least one of a simulation depth real value, simulation real point cloud and simulation real distance. The application can improve the efficiency of data simulation.

Description

Technical Field

[0001] This invention relates to the field of data simulation technology, and in particular to an operating system, method, electronic device and storage medium for a TOF camera simulation system. Background Technology

[0002] In fields such as 3D reconstruction, robot navigation, and motion-sensing games, it is often necessary to build complex deep learning networks to accurately measure the distance or depth data of a scene. However, deep learning methods require a large amount of data for training. Currently, data is mainly obtained by generating data through simulation systems. For example, 3D modeling software such as Blender is used to render the scene, and then the corresponding data is simulated based on the actual time-of-flight camera parameters.

[0003] However, the workflow of 3D modeling software such as Blender is relatively complex and the simulation efficiency is low. In addition, general simulation systems usually perform simulation by superimposing light signals for rendering, which results in a huge consumption of computing resources when computing imaging. Furthermore, the system is integrated and cannot decouple the system functions. Every time the rendering parameters are modified, re-rendering is required, which leads to low efficiency of data simulation. Summary of the Invention

[0004] This invention provides a TOF camera simulation system, method, and computer-readable storage medium, the main purpose of which is to solve the problem of low data simulation efficiency.

[0005] To achieve the above objectives, the present invention provides a TOF camera simulation system, comprising a transient rendering module, a TOF imaging simulation module, and a TOF depth calculation module, wherein: the transient rendering module is used to acquire scene files, camera pose, and camera parameters, and generate a transient illumination map based on the acquired data; the TOF imaging simulation module is used to obtain a received modulation signal using the time information of the transient illumination map and a preset modulation signal function, and to calculate an original phase map from the received modulation signal; the TOF depth calculation module is used to parse the original phase map to obtain simulated real data, wherein the simulated real data includes at least one of simulated depth real value, simulated real point cloud, and simulated real distance.

[0006] To address the aforementioned issues, this invention also provides a method for operating a TOF camera simulation system. The method includes: acquiring scene files, camera pose, and camera parameters, and generating a transient illumination map based on the acquired data; obtaining a received modulation signal using the time information of the transient illumination map and a preset modulation signal function, and calculating an original phase map from the received modulation signal; and parsing the original phase map to obtain simulated real data, wherein the simulated real data includes at least one of simulated depth, simulated real point cloud, and simulated real distance.

[0007] To address the aforementioned issues, this invention also provides a Time-of-Flight (TOF) depth camera, comprising a transmitter, a receiver, and a processor, wherein: the transmitter is used to transmit a modulated signal to the measured scene; the receiver is used to acquire the modulated signal reflected back from the measured scene and generate an original phase map, which is then transmitted to the processor; the processor carries a trained neural network model, used to process the original phase map through the trained neural network model to obtain a depth image of the measured scene; wherein the training data of the trained neural network model includes the corresponding original phase map and the ground truth value of the simulated depth map obtained from the aforementioned TOF simulation system.

[0008] To address the aforementioned problems, the present invention also provides an electronic device comprising: a processor for carrying the aforementioned TOF camera simulation system to acquire simulation data; and a memory storing a computer program executable by the processor, wherein the computer program, when executed by the processor, performs the aforementioned simulation method applied to the TOF camera simulation system.

[0009] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the simulation method described above for a TOF camera simulation system.

[0010] Compared with existing technologies, the embodiments of the present invention generate transient illumination maps through transient rendering, avoiding the rendering of scene files, camera poses and camera parameters by overlaying light, thereby improving the efficiency of rendering simulation. At the same time, the simulation system is divided into multiple functional modules. When the parameters in different functional modules need to be modified, there is no need to re-render the data. Rendering can be started from the functional module with the modified parameters, further improving the rendering efficiency. Attached Figure Description

[0011] Figure 1 This is a system architecture diagram of the operating system of a TOF camera simulation system provided in an embodiment of the present invention; Figures 2(a) to 2(c) are three transient illuminance maps at different times for a corner scene provided in one embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the principle of distance-to-depth conversion according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating a simulation method for a TOF camera simulation system according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a TOF depth camera according to an embodiment of the present invention; Figure 6This is a schematic diagram of the structure of an electronic device that implements a simulation method for a TOF camera simulation system according to an embodiment of the present invention.

[0012] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0014] This application provides a TOF camera simulation system, which can be mounted on electronic devices including but not limited to servers and terminals to acquire simulation data from a TOF camera. In other words, the TOF camera simulation system can be implemented on software or hardware mounted on terminal devices or server devices, and there are no limitations on this.

[0015] A Time-of-Flight (TOF) camera mainly consists of a transmitter, a receiver, and a processor. The transmitter transmits a modulated signal to the measured scene. The receiver acquires the modulated signal reflected back from the scene and generates a raw phase map, which is then transmitted to the processor. The raw phase map is the original data obtained by the receiver from the reflected modulated signal, i.e., the raw phase map. The processor calculates the depth map of the measured scene from the raw phase map. However, TOF cameras generally suffer from multipath effects and in-lens scattering, among other systematic errors. Traditional calculation methods struggle to eliminate or reduce these errors, and existing solutions utilize deep learning. Deep learning methods require a large amount of data for training. Currently, there are two main methods for acquiring data: building complex real-world data acquisition systems or generating data through simulation systems. Building a real-world data acquisition system is complex and time-consuming. Existing simulation systems rely on 3D modeling software such as Blender, which has a complex process and low simulation efficiency. Therefore, this application provides a TOF camera simulation system and method to address these problems.

[0016] Figure 1This is a system architecture diagram of a TOF camera simulation system provided in an embodiment of the present invention. In this embodiment, the TOF camera simulation system includes a transient rendering module 101, a TOF imaging simulation module 102, and a TOF depth calculation module 103. The transient rendering module 101 is used to acquire scene files, camera poses, and camera parameters, and generate a transient illumination map based on the acquired data. The TOF imaging simulation module 102 is used to obtain a received modulation signal using the time information of the transient illumination map and a preset modulation signal function, and to calculate the original phase map from the received modulation signal. The TOF depth calculation module 103 is used to parse the original phase map to obtain simulated real data, wherein the simulated real data includes at least one of simulated depth real value, simulated real point cloud, and simulated real distance.

[0017] In some embodiments, the transient rendering module 101 includes an input parser unit 1011 and a transient rendering unit 1012, wherein: the input parser unit 1011 is used to acquire and parse scene files, camera poses and camera parameters to generate a rendered view image; the transient rendering unit 1012 is used to use a path tracing algorithm to track the multipath reflection of light within the rendered view image, and simultaneously perform transient rendering to obtain a transient illumination map.

[0018] In one embodiment, the scene file acquired by the input parser unit 1011 includes data such as a 3D scene model, materials, textures, and light sources; camera pose refers to the current pose of the camera when it moves. Preferably, in this embodiment of the invention, the parameters of a pre-built pinhole camera model can be used as camera parameters. In other embodiments of the invention, the parameters of a pre-built lens group model can also be used as camera parameters.

[0019] In one embodiment, the transient rendering unit 1012 may employ a path tracing algorithm to achieve global illumination of the rendered view image, thereby simultaneously capturing multipath reflections within the rendered view image for rendering and generating a transient illumination map of the rendered view image.

[0020] In one practical application scenario of this invention, since the path tracing algorithm can emit a ray of light from the viewpoint (equivalent to the receiving end of the TOF camera), when the ray intersects with the surface of the object, it is reflected or refracted in another direction according to the surface material properties. This process is repeated until the light source (equivalent to the transmitting end of the TOF camera) is hit or the light escapes from the scene, thus forming a complete closed light path. Therefore, global illumination can be achieved through this path tracing algorithm, thereby simulating the multipath effect.

[0021] Specifically, path tracing algorithms can track light rays that are reflected multiple times within a rendered viewpoint image, thereby accurately simulating the multipath effect. In Time-of-Flight (TOF) technology, the multipath effect refers to the phenomenon where multiple emitted light rays travel through different reflection paths, or a single light ray undergoes multiple reflections before being received by the same pixel at the receiving end, leading to errors in distance calculation. Therefore, by using path tracing algorithms to track the multipath reflections of light rays within a rendered viewpoint image, accurate tracking of light ray paths can be achieved, thus improving the accuracy of subsequently generated simulation data.

[0022] In one embodiment, the path tracing algorithm can pre-set the tracing depth, i.e., the stage where the maximum number of light reflections occurs (i.e., tracing stops). In one practical application scenario of this invention, each time a ray hits the object surface in the rendered viewpoint image, it connects with the light source to form a complete light path. The tracing depth can be set to 1, at which point an ideal multipath-free effect can be simulated. Furthermore, the tracing depth is adjustable; the greater the tracing depth, the closer it approximates the real multipath effect.

[0023] In one embodiment, the transient rendering unit 1012 is used to perform transient rendering to obtain a transient illumination map, including: obtaining the optical path length of different optical paths in the rendered view image based on the propagation distance of multi-path reflection of light within the rendered view image, and calculating time information according to the optical path length of different optical paths, wherein the time information represents the time when the light emitted by the transmitter in the TOF camera reaches the receiver; calculating the distance truncation using the total length of a preset time window, wherein the total length of the time window is determined by the number of time windows, and each time window is used to record time information; performing transient rendering on the rendered view image according to the time information and the distance truncation, and simultaneously performing continuous exposure to obtain a sequence of continuously exposed images within the total length of the time window, thereby determining that the sequence of continuously exposed images is multiple transient illumination maps, wherein one time window records the time information of one frame of transient illumination map.

[0024] Specifically, the transient rendering unit 1012 in this embodiment adds a time window to record time information, enabling the rendering of multiple extremely short exposure image sequences (i.e., multiple transient illumination maps) at once. This time information is equivalent to the time it takes for light emitted from the transmitter in a TOF camera to reach the receiver, which can be derived by inversely calculating the optical path: Time = Optical Path / Speed ​​of Light. Here, the speed of light is the speed of light in ordinary air, and the optical path is the propagation distance obtained by the complete optical path formed by multi-path reflection simulated by a path tracing algorithm. For example, by setting 1000 time windows, each time window being 1 nanosecond (ns), a total of 1000 image sequences (i.e., transient illumination maps) with continuous exposure within a time range of 0-1000ns can be obtained; where the exposure time of each image is 1ns, and the image number corresponds to the time it takes for the light signal to travel from the transmitter to the receiver.

[0025] However, in practical applications, the total length of the time window is limited by the TOF camera's transmitter. A maximum distance truncation needs to be set to stop tracking the light. The distance truncation is calculated as: distance truncation = speed of light * maximum time / 2, where the maximum time is determined by the total length and number of time windows. It should be noted that the total length and number of time windows can be set according to the actual working distance and modulation frequency of the TOF camera. It is sufficient to ensure that the distance truncation is slightly greater than the camera's maximum working distance, and that the total length of the time window is an order of magnitude smaller than the period of the modulation signal emitted by the camera.

[0026] Compared to the use of a steady-state renderer in existing technologies, which superimposes all light signals over a longer exposure time and renders only one image at a time, the transient renderer used in this embodiment adds a time window to record time information, and can render multiple image sequences of extremely short exposures (i.e., transient illumination maps) at once, thus greatly improving rendering efficiency.

[0027] Figures 2(a) to 2(c) show three transient illuminance maps of a corner scene at different times according to one embodiment of the present invention. The corner scene consists of three orthogonal planes, with time increasing sequentially from left to right (different optical path lengths). The brightest white light band represents the direct reflection signal, while the darker bands represent contributions from multiple reflections. This is because the direct reflection signal attenuates the least, allowing a bright band to be seen propagating forward.

[0028] In one embodiment, the transient rendering module 101 further includes a ground truth rendering unit 1013, which is used to identify the ground truth value of the distance map of the rendered view image and calculate the ground truth value of the point cloud and / or the ground truth value of the depth map based on the ground truth value of the distance map. Specifically, the ground truth rendering unit 1013 also uses a path tracing algorithm to identify the ground truth value of the distance map of the rendered view image. Here, the distance map refers to the distance from the light emitted from the camera to the first reflecting surface of the scene. During the identification process, the distance values ​​of light rays with distances greater than the distance cutoff are set to zero in order to measure accurate distance values ​​and obtain the ground truth value of the distance map.

[0029] In one embodiment, the ground truth rendering unit 1013 calculates the point cloud ground truth and / or depth map ground truth based on the distance map ground truth by: performing distance-to-depth calculation based on the distance map ground truth and an equivalent pinhole camera model (i.e., triangulation) and using camera parameters (such as baseline) to obtain the depth map ground truth. Figure 3The conversion relationship between distance and depth is shown. Further, after obtaining the ground truth depth map, it can be converted into a point cloud based on camera intrinsic and extrinsic parameters to obtain the point cloud ground truth. It should be noted that the ground truth values ​​provided in this embodiment can be used for supervision in deep learning, providing a sample image set for the corresponding neural network training. The point cloud is mainly used to evaluate the algorithm's performance; differences that are difficult to distinguish in images can be visually observed through the point cloud. In addition, the ground truth distance map, point cloud ground truth, and / or depth map ground truth can be collectively referred to as simulation ground truth data.

[0030] In some embodiments, the TOF imaging simulation module 102 includes a signal modulation unit 1021 and a raw phase map generation unit 1022. The signal modulation unit 1021 is used to generate a received modulation signal based on the time information of each transient illumination map and a preset modulation signal function. The received modulation signal is equivalent to the modulation signal of the reflected light collected by the receiver of the TOF camera. The phase map generation unit 1022 is used to integrate (also known as cross-correlation) the received modulation signal and a preset reference signal to complete the phase difference calculation and obtain the raw phase map of each transient illumination map. The preset reference signal may be the modulation signal of the light signal emitted by the transmitter in the TOF camera or the exposure function of the receiver.

[0031] In some embodiments, the TOF imaging simulation module 102 may further include at least one of a phase compensation unit 1023, a scattering unit 1024, and a post-processing unit 1025, wherein: the phase compensation unit 1023 is used to add a preset phase offset to the received modulation signal; the scattering unit 1024 is used to simulate the scattering effect of the original phase image; and the post-processing unit 1025 is used to remove rendering noise from the original phase image and add noise to the original phase image.

[0032] It should be noted that, under normal circumstances, the phase offset information can be obtained by calibrating each TOF camera separately in advance. The phase compensation unit 1023 adds the preset phase offset to the received modulation signal, which can realize the simulation of the output of a specific camera, thereby making the simulation results closer to reality and improving the simulation effect.

[0033] In one embodiment, the scattering unit 1024 can simulate the effects caused by lens scattering, defocusing, and pixel crosstalk in a real camera, collectively referred to here as scattering effects. The final effect of scattering is that the energy of a pixel is dispersed to the surrounding pixels, which can be simulated using the point spread function (PSF). Obtaining the point spread function requires measurement for a real camera. Therefore, the scattering unit can simulate the data from different perspectives of scattering effects to improve the quality of the final simulation data.

[0034] In one embodiment, the post-processing unit 1025 primarily processes image noise. In one practical application scenario of this invention, to efficiently perform transient rendering, the number of samples is generally not very large, resulting in a large amount of noise in the final image. To improve the quality of the final simulation data, it is necessary to denoise the noise in the original phase map generated by rendering. Preferably, the original phase map can be denoised using median filtering and bilateral filtering. By combining these two methods, the noise removal effect on the original phase map can be improved.

[0035] Furthermore, in this embodiment of the invention, after removing noise generated during the rendering process, noise can be added to the original phase map. Specifically, since cameras themselves contain some noise in practical applications, if the simulated image has extremely low noise due to noise removal, the simulation results will not match reality. Therefore, to address the noise present in the actual camera, background noise and Gaussian noise can be added to the original phase map to simulate the output of the actual camera, making the final simulation data more realistic.

[0036] In some embodiments, the TOF depth calculation module 103 can calculate simulated real data from the original phase map; wherein, the simulated real data includes at least one of simulated true depth value, simulated true point cloud, and simulated true distance. Specifically, since traditional time-of-flight calculation methods assume that light undergoes only one direct reflection, they can have significant errors in scenarios with strong multipath effects. To address this issue, existing technologies have designed many different deep learning-based algorithms. Different deep learning algorithms require different inputs. To provide corresponding inputs for different types of deep learning algorithms, this system adds a TOF depth calculation module 103 to simulate outputs close to those of a real camera.

[0037] Preferably, the TOF depth calculation module 103 can directly learn the depth from the original phase map input to an end-to-end neural network model to obtain the simulated true depth value, or it can calculate the depth from the original phase map based on a traditional algorithm and then use deep learning to correct for multipath effects to simulate a true depth value close to the output of a real camera. After obtaining the simulated true depth value, it can be further converted into a simulated real point cloud to obtain the simulated real data of this embodiment.

[0038] In summary, the TOF camera simulation system provided in this application can obtain the output of an error-free ideal camera through the ground truth rendering unit. By adjusting the settings of the TOF imaging simulation module, it can simulate the output of a near-real camera and the output of a camera that only considers multipath effects. It provides flexible and targeted training data for different deep learning schemes, such as original phase maps, distance maps (ground truth / real), and depth maps (ground truth / real), etc., and only requires one set of transient rendering, which improves the generalization ability of the data.

[0039] Figure 4 This is an embodiment of the present invention provided for application in... Figure 1 The provided flowchart illustrates the simulation method for the TOF camera simulation system. For details not covered herein, please refer to the aforementioned embodiments; they will not be repeated here. Specifically, the simulation method includes: S1. Obtain scene files, camera poses, and camera parameters, and generate transient illumination maps based on the acquired data; S2. Obtain the received modulated signal using the time information of the transient illuminance map and the preset modulation signal function, and calculate the original phase map from the received modulated signal; S3. Analyze the original phase map to obtain the simulated real data, wherein the simulated real data includes at least one of the simulated depth real value, simulated real point cloud, and simulated real distance.

[0040] Figure 5 This is a schematic diagram of a Time-of-Flight (TOF) depth camera according to an embodiment of the present invention. The TOF depth camera includes a transmitter, a receiver, and a processor. The transmitter transmits a modulated signal to the measured scene. The receiver acquires the modulated signal reflected back from the measured scene and generates an original phase map, which is then transmitted to the processor. The processor processes the original phase map using a trained neural network model to obtain a depth image of the measured scene. The training data for the trained neural network model includes the corresponding original phase map and the ground truth of the simulated depth map obtained through a TOF simulation system. Specifically, the neural network model is trained using the original phase map as input and the ground truth of the simulated depth map as output samples to obtain the trained neural network model.

[0041] Figure 6 This is a schematic diagram of the structure of an electronic device for implementing a simulation method for a TOF camera simulation system according to an embodiment of the present invention. The electronic device 1 may include a processor 10 and a memory 11, wherein the processor 10 is used to carry the TOF camera simulation system provided by the present invention to acquire simulation data, and the memory 11 is used to store computer programs that can run on the processor 10, such as simulation programs applied to the TOF camera simulation system.

[0042] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, neural network chips, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing simulation programs for a TOF camera simulation system) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0043] In one embodiment, the memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive of the electronic device. In other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. Further, the memory 11 can include both internal storage units and external storage devices of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of the running program of the TOF camera simulation system, but also to temporarily store data that has been output or will be output.

[0044] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0045] It should be understood that the embodiments are for illustrative purposes only and are not limited to this structure in the scope of the patent application. Furthermore, if the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0046] The present invention also provides a computer-readable storage medium storing a computer program. When executed by a processor of an electronic device, the computer program can: acquire scene files, camera poses, and camera parameters, and generate a transient illumination map based on the acquired data; obtain a received modulation signal using the time information of the transient illumination map and a preset modulation signal function, and calculate an original phase map from the received modulation signal; and parse the original phase map to obtain simulated real data, wherein the simulated real data includes at least one of simulated depth real value, simulated real point cloud, and simulated real distance.

[0047] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0048] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0049] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0050] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0051] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0052] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A TOF camera simulation system, characterized in that, The system includes a transient rendering module, a TOF imaging simulation module, and a TOF depth calculation module, wherein: The transient rendering module is used to acquire scene files, camera poses and camera parameters and generate a rendered view image, and to perform transient rendering on the rendered view image to generate a transient illumination map. The TOF imaging simulation module is used to obtain the received modulation signal using the time information of the transient illumination map and a preset modulation signal function, and to calculate the original phase map from the received modulation signal. The TOF depth calculation module is used to parse the original phase map to obtain simulated real data, wherein the simulated real data includes at least one of simulated depth real value, simulated real point cloud, and simulated real distance; The transient rendering module includes an input parser unit and a transient rendering unit, wherein: The input parser unit is used to parse the scene file, camera pose, and camera parameters to generate a rendered viewpoint image; The transient rendering unit is used to track the multipath reflection of light within the rendered viewpoint image using a path tracing algorithm, and simultaneously perform transient rendering to obtain the transient illumination map.

2. The TOF camera simulation system as described in claim 1, characterized in that, The transient rendering unit uses a path tracing algorithm to track the multipath reflections of light within the rendered viewpoint image, and simultaneously performs transient rendering to obtain a transient illumination map, including: The optical path length of different optical paths in the rendered view image is obtained based on the propagation distance of multi-path reflection of light within the rendered view image, and time information is calculated based on the optical path length of the different optical paths, wherein the time information represents the time it takes for the light emitted by the transmitter in the TOF camera to reach the receiver. Distance cutoff is calculated using a pre-set total time window length, whereby the distance cutoff represents the cessation of ray tracking. The total time window length is determined by the number of time windows, and each time window is used to record the time information. Based on the time information and the distance truncation, the rendering view image is transiently rendered, and a continuous exposure image sequence within the total length of the time window is obtained simultaneously. The continuous exposure image sequence is determined to correspond to a multi-frame transient illumination map, wherein one time window records the time information of one frame of transient illumination map.

3. The TOF camera simulation system as described in claim 1, characterized in that, The transient rendering module also includes a ground truth rendering unit, which is used to identify the ground truth value of the distance map of the rendered view image, and calculate the ground truth value of the point cloud and / or the ground truth value of the depth map based on the ground truth value of the distance map.

4. The TOF camera simulation system as described in claim 1, characterized in that, The TOF imaging simulation module includes a signal modulation unit and a raw phase map generation unit, wherein... The signal modulation unit is used to generate a received modulation signal for each transient illuminance map based on the time information of each transient illuminance map and a preset modulation signal function; The original phase map generation unit is used to integrate the received modulation signal and the preset transmission reference signal to calculate the phase difference and obtain the original phase map of each transient illumination map.

5. The TOF camera simulation system as described in claim 4, characterized in that, The TOF imaging simulation module may further include at least one of a phase compensation unit, a scattering unit, and a post-processing unit, wherein... The phase compensation unit is used to add a preset phase offset to the received modulation signal; The scattering unit is used to simulate the scattering effect of the original phase image; The post-processing unit is used to remove rendering noise from the original phase map and to add noise to the original phase map.

6. The TOF camera simulation system as described in claim 5, characterized in that, When the post-processing unit removes rendering noise from the original phase map, it includes: denoising the original phase map by median filtering and bilateral filtering.

7. The TOF camera simulation system as described in claim 5, characterized in that, When the post-processing unit adds noise to the original phase map, it includes adding background noise and Gaussian noise to the original phase map.

8. A simulation method applied to a TOF camera simulation system, characterized in that, The method includes: Acquire scene files, camera poses, and camera parameters, and generate transient illumination maps based on the acquired data; The received modulation signal is obtained using the time information of the transient illuminance map and a preset modulation signal function, and the original phase map is obtained by calculating the received modulation signal. The original phase map is analyzed to obtain simulated real data, wherein the simulated real data includes at least one of simulated depth real value, simulated real point cloud, and simulated real distance; Acquire scene files, camera poses, and camera parameters, and generate transient illumination maps based on the acquired data, including: The scene file, camera pose, and camera parameters are parsed to generate a rendered viewpoint image; The path tracing algorithm is used to track the multipath reflection of light within the rendered viewpoint image, and transient rendering is performed simultaneously to obtain the transient illumination map.

9. A TOF depth camera, characterized in that, Includes the transmitter, receiver, and processor, among which: The transmitting end is used to transmit modulated signals to the measured scene; The receiving end is used to acquire the modulated signal reflected back by the measured object and generate an original phase map, which is then transmitted to the processor. The processor is equipped with a trained neural network model, which is used to process the original phase map through the trained neural network model to obtain a depth image of the measured scene; wherein, the training data of the trained neural network model includes the corresponding original phase map and simulated depth map ground values ​​obtained by the TOF camera simulation system according to any one of claims 1-7.

10. An electronic device, characterized in that, Includes a memory communicatively connected to the processor, wherein: The processor is used to carry the TOF camera simulation system according to any one of claims 1-7 to acquire simulation data; The memory stores a computer program that can be executed by the processor. When the computer program is executed by the processor, it performs the simulation method for the TOF camera simulation system as described in claim 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the simulation method for a TOF camera simulation system as described in claim 8.

Images