Method for correcting point cloud data of ITOF camera, ITOF camera, device, terminal and storage medium

Abstract

The invention belongs to the technical field of cameras, and provides a method for correcting point cloud data of an ITOF camera, the ITOF camera, a device, a terminal and a storage medium, and the method comprises the steps: firstly controlling the ITOF camera to be in an off state, and obtaining sparse point cloud detected by a laser radar in an on state; meanwhile, the laser radar is controlled to be in a closed state, and dense point clouds collected by the ITOF camera in the open state are obtained; and then a target model is constructed based on the sparse point cloud and the dense point cloud, and the finally obtained target model is used for correcting multi-path interference point cloud data received by the ITOF camera. According to the method, the dense point cloud, subjected to multi-path interference, of the ITOF camera is guided based on the sparse point cloud data of the laser radar without multi-path interference, the point cloud data generated in the follow-up ITOF camera operation process is corrected through the target model, and the multi-path interference phenomenon existing when the ITOF camera shoots the measured object can be effectively solved.

Description

Technical Field

[0001] The present application belongs to the field of camera technology, and in particular relates to a method for correcting ITOF camera point cloud data, an ITOF camera, a device, a terminal and a storage medium. Background Art

[0002] ITOF camera, the full name is indirect time of flight (ITOF) camera, it indirectly measures the flight time of light by measuring phase shift, so as to obtain the depth information of the target object. ITOF measurement technology is a non-contact optical 3D measurement technology. Compared with other optical 3D measurement technologies, ITOF has many advantages such as low cost, long measurement distance and compact structure. It is widely used in many fields such as consumer electronics and industrial measurement.

[0003] The ITOF camera mainly consists of three core components: the signal transmission module, the signal receiving module and the signal processing module. The signal transmission module is responsible for emitting periodically modulated light wave signals. After these light waves encounter objects in the scene and are reflected, they are captured by the lens in the signal receiving module and directed to the ITOF image sensor. Depending on the different distances between the object and the sensor, the light wave signals will show different phase shifts when they arrive at the sensor. Next, the image sensor accurately measures the phase shift of these returned light wave signals, and these data are then sent to the signal processing module for calculation and analysis and converted into corresponding phase information. Finally, these phase information will be further decoded to determine the depth information of the object.

[0004] The measurement theory of ITOF cameras is based on the following assumption: "Each pixel of the image sensor only captures a beam of light directly reflected from the target object" to perform relevant distance measurement calculations. However, in the actual application of terminal devices using ITOF technology, the actual situation is much more complicated than the ideal situation assumed above. Due to the irregularity of the surface of the object being measured, the sensor actually receives multiple rays of light reflected back through different paths. This phenomenon is called multipath interference. Due to the existence of multipath interference, it will cause deviations in the measurement data, thereby affecting the accuracy of depth calculation. Summary of the invention

[0005] The embodiments of the present application provide a method for correcting ITOF camera point cloud data, an ITOF camera, a device, a terminal and a storage medium, aiming to solve the technical problem that due to the irregularity and scattering properties of the surface of the object being measured, the point cloud data obtained when the ITOF camera side shoots the object being measured has multi-path interference.

[0006] In a first aspect, an embodiment of the present application provides a method for correcting ITOF camera point cloud data, the method comprising:

[0007] Controlling the ITOF camera to be in an off state, and obtaining a sparse point cloud detected by the laser radar in an on state, wherein the sparse point cloud represents data scanned by the laser radar;

[0008] Control the laser radar to be in an off state, and obtain a dense point cloud collected by the ITOF camera in an on state, wherein the dense point cloud represents a reflection signal received by the ITOF camera;

[0009] A target model is constructed based on the sparse point cloud and the dense point cloud, and the target model is used to correct the point cloud data received by the ITOF camera that is affected by multipath interference.

[0010] In one embodiment, the dense point cloud is composed of a camera depth map generated when the ITOF camera captures an image;

[0011] The constructing the target model based on the sparse point cloud and the dense point cloud comprises:

[0012] Transforming the sparse point cloud into a camera coordinate system of the ITOF camera to obtain a reference depth map aligned with the camera depth map;

[0013] Determine a camera grayscale image generated when the ITOF camera captures an image;

[0014] The camera grayscale image and the camera depth image are used as inputs of a training set of a neural network, and the reference depth image is used as a true value. Training is performed based on the input and the true value to construct a target model.

[0015] In one embodiment, transforming the sparse point cloud into a camera coordinate system of the ITOF camera to obtain a reference depth map aligned with the camera depth map comprises:

[0016] Registering the sparse point cloud with the dense point cloud to obtain a spatial transformation matrix;

[0017] The sparse point cloud is transformed into a camera coordinate system of the ITOF camera based on the spatial transformation matrix to obtain a reference depth map aligned with the camera depth map.

[0018] In one embodiment, after completing the construction of the target model, the method further includes:

[0019] Acquire a dense point cloud to be corrected that is collected by the ITOF camera when photographing the object to be measured at the kth moment, wherein the dense point cloud to be corrected represents a reflection signal that is subject to multipath interference and is received by the ITOF camera at the kth moment;

[0020] Determine the grayscale image to be measured and the depth image to be measured generated when the ITOF camera photographs the object to be measured at the kth moment;

[0021] Inputting the grayscale image to be measured and the depth image to be measured into a target model, and outputting a target depth image that is not affected by multipath interference by the target model;

[0022] A target dense point cloud free from multipath interference is determined based on the target depth map.

[0023] In one embodiment, determining a target dense point cloud free from multipath interference based on the target depth map includes:

[0024] Determining the internal reference information of the ITOF camera;

[0025] Based on the intrinsic reference information, the target depth map is converted into a target dense point cloud that is not affected by multipath interference.

[0026] In one embodiment, the target model includes an encoder, a decoder, and a spatial propagation network;

[0027] The step of inputting the grayscale image to be measured and the depth image to be measured into a target model, and outputting a target depth image that is not interfered by multipath by the target model, comprises:

[0028] Inputting the grayscale image to be tested and the depth image to be tested as input data into the encoder, so that the encoder encodes the input data to obtain an encoding result;

[0029] Inputting the encoding result to the decoder, so that the decoder decodes the encoding result to obtain initial depth map prediction data;

[0030] The spatial propagation network is called to determine the image data to be eliminated from the initial depth map prediction data, so that the target model outputs a target depth map that is not interfered by multipath, wherein the image data to be eliminated represents the edge position of the object under test.

[0031] In a second aspect, an embodiment of the present application provides an ITOF camera, wherein the ITOF camera includes a signal transmitting module, a signal receiving module, a signal processing module, and the target model as described in the first aspect above;

[0032] The signal transmitting module is used to transmit a light wave signal, wherein the light wave signal forms a reflection signal after being emitted to the object to be measured;

[0033] The signal receiving module is used to receive the reflected signal and transmit the reflected signal to the signal processing module;

[0034] The signal processing module is used to determine the corresponding dense point cloud to be corrected based on the reflected signal, and call the target model to correct the dense point cloud to be corrected, so as to output a target dense point cloud that is not affected by multipath interference.

[0035] In a third aspect, the present invention further proposes a device for constructing a target model for correcting ITOF camera point cloud data, the device comprising:

[0036] A first acquisition unit, used to control the ITOF camera to be in an off state, and to acquire a sparse point cloud detected by the laser radar in an on state, wherein the sparse point cloud represents data scanned by the laser radar;

[0037] A second acquisition unit is used to control the laser radar to be in an off state, and acquire a dense point cloud collected by the ITOF camera in an on state, wherein the dense point cloud represents a reflection signal received by the ITOF camera;

[0038] A construction unit is used to construct a target model based on the sparse point cloud and the dense point cloud, and the target model is used to correct the point cloud data received by the ITOF camera that is affected by multipath interference.

[0039] In a fourth aspect, the present invention further proposes a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the method for correcting ITOF camera point cloud data as described in the first aspect above are implemented.

[0040] In a fifth aspect, the present invention further proposes a storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for correcting ITOF camera point cloud data as described in the first aspect above are implemented.

[0041] The beneficial effect of the present application is that, first, the ITOF camera is controlled to be in an off state, and the sparse point cloud detected by the laser radar in an on state is obtained; at the same time, the laser radar is controlled to be in an off state, and the dense point cloud collected by the ITOF camera in an on state is obtained; then a target model is constructed based on the sparse point cloud and the dense point cloud, and the target model finally obtained is used to correct the point cloud data with multipath interference received by the ITOF camera. The present application guides the dense point cloud with multipath interference of the ITOF camera based on the sparse point cloud data of the laser radar without multipath interference, obtains a target model for correcting the multipath interference phenomenon, and corrects the point cloud data generated during the subsequent operation of the ITOF camera through the target model, which can effectively solve the multipath interference phenomenon when the ITOF camera shoots the measured object. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 is a schematic diagram of the structure of a terminal provided by an embodiment of the present application;

[0044] Figure 2 It is a schematic diagram of an embodiment flow chart corresponding to the method for correcting ITOF camera point cloud data provided by the present application;

[0045] Figure 3 A structural block diagram of an ITOF camera provided in an embodiment of the present application;

[0046] Figure 4 It is a flowchart of another embodiment corresponding to the method for correcting ITOF camera point cloud data provided by the present application;

[0047] Figure 5 is a schematic diagram of the structure of a target model involved in an embodiment of the present application;

[0048] Figure 6 A structural block diagram of a device for constructing a target model for correcting ITOF camera point cloud data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0050] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0051] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0052] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0053] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0054] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0055] An embodiment of the present invention provides a terminal, such as Figure 1 As shown, Figure 1 The terminal 1 of this embodiment includes: at least one processor 1001 ( Figure 1 Only one is shown), a memory 1005 and a computer program 1006 stored in the memory 1005 and executable on the at least one processor 1001, wherein the processor 1001 implements the steps in the method embodiment of the present application for correcting ITOF camera point cloud data when executing the computer program 1006.

[0056] The terminal 1 also includes a laser radar interface module 1003 for integrating or connecting a laser radar device, the laser radar device is used to detect the object under test, and the terminal 1 can obtain a sparse point cloud detected by the laser radar in an on state through the laser radar interface module 1003, and the sparse point cloud represents the data scanned by the laser radar.

[0057] The terminal 1 further comprises a camera interface module 1004 for capturing image feature data of the object under test.

[0058] In a specific application, the terminal 1 provided in the embodiment of the present application may be an ITOF camera integrated with a laser radar, or may be other terminal devices equipped with a laser radar and an ITOF camera.

[0059] Those skilled in the art will understand that Figure 1 It is only an example of the terminal 1 and does not constitute a limitation on the terminal 1. The terminal 1 may include more or less components than those shown in the figure, or a combination of certain components, or different components.

[0060] The processor 1001 may be a central processing unit (CPU), and the processor 10 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0061] In some embodiments, the memory 1005 may be an internal storage unit of the terminal 1, such as a hard disk or memory of the terminal 1. In other embodiments, the memory 1005 may also be an external storage device of the terminal 1, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal 1. Further, the memory 1005 may also include both an internal storage unit of the terminal 1 and an external storage device. The memory 1005 is used to store an operating system, a computer execution program of the laser radar calibration method, etc., and the computer execution program may at least include an image processing program and a motion control program.

[0062] It is understandable that the conventional solution to the multipath interference problem of ITOF cameras mentioned in the background technology is that ITOF cameras often use dual-frequency measurement to mitigate multipath interference, but the actual number of superimposed paths is unknown and the dual-frequency measurement solution is sensitive to noise. As for multi-frequency measurement, most ITOF cameras still do not support it, and the time resolution will decrease as the frequency increases.

[0063] refer to Figure 2In order to solve the technical problem that the point cloud data obtained when the ITOF camera shoots the object under test has multipath interference due to the irregularity and scattering properties of the surface of the object under test, the embodiment of the present application provides a method for correcting the point cloud data of the ITOF camera, and the method includes steps S10 to S30:

[0064] Step S10: controlling the ITOF camera to be in an off state, and obtaining a sparse point cloud detected by the laser radar in an on state, wherein the sparse point cloud represents data scanned by the laser radar;

[0065] Step S20: controlling the laser radar to be in an off state, and obtaining a dense point cloud collected by the ITOF camera in an on state, wherein the dense point cloud represents a reflection signal received by the ITOF camera;

[0066] It should be noted that the execution subject of the embodiment of the present application is the above Figure 1 The controller shown in the figure guides the dense point cloud of the ITOF camera with multipath interference through the sparse point cloud of the lidar without multipath interference, so as to correct the multipath interference.

[0067] In the specific implementation, the ITOF camera can be calibrated using a calibration board to obtain the internal parameter information of the ITOF camera (light coordinates, focal length, lens distortion type, distortion coefficient, image height and width). These internal parameter information are used for the subsequent conversion between the depth map and point cloud generated by the ITOF camera.

[0068] Secondly, the embodiment of the present application can customize a precise base to fix the laser radar and the ITOF camera. Since the operation of the laser radar and the operation of the ITOF camera interfere with each other, the ITOF camera should be turned off when collecting sparse point clouds of the laser radar without multipath interference. Similarly, when taking dense point clouds of the ITOF camera with multipath interference, the laser radar should be kept turned off.

[0069] Step S30: constructing a target model based on the sparse point cloud and the dense point cloud, wherein the target model is used to correct the point cloud data received by the ITOF camera that is affected by multipath interference.

[0070] This application guides the dense point cloud of the ITOF camera with multipath interference based on the sparse point cloud data of the laser radar without multipath interference, and obtains a target model for correcting the multipath interference phenomenon. The target model is used to correct the point cloud data generated during the subsequent operation of the ITOF camera, which can effectively solve the multipath interference phenomenon when the ITOF camera shoots the object under test.

[0071] Furthermore, the step S30 includes the following sub-steps (sub-steps S31 to S33):

[0072] Sub-step S31: transforming the sparse point cloud into the camera coordinate system of the ITOF camera to obtain a reference depth map aligned with the camera depth map;

[0073] Specifically, the sparse point cloud is registered with the dense point cloud to obtain a spatial transformation matrix; based on the spatial transformation matrix, the sparse point cloud is transformed into a camera coordinate system of the ITOF camera to obtain a reference depth map aligned with the camera depth map.

[0074] In a specific implementation, the embodiment of the present application can use the ICP algorithm to register the laser radar sparse point cloud without multipath interference and the ITOF camera dense point cloud with multipath interference to obtain the spatial transformation matrix RT. Let P represent the coordinates of the laser radar sparse point cloud without multipath interference, R represent the rotation matrix, T represent the translation vector, and the corresponding coordinates of the laser radar sparse point cloud without multipath interference transformed to the ITOF camera coordinate system are expressed as P`=R*P+T;

[0075] Then, the embodiment of the present application can use the RT matrix obtained by the ICP algorithm to transform the lidar dense point cloud without multipath interference into the ITOF camera coordinate system, thereby obtaining a lidar depth map without multipath interference (i.e., the reference depth map) that is aligned with the ITOF camera depth map subject to multipath interference.

[0076] Sub-step S32: determining a camera grayscale image generated when the ITOF camera captures an image;

[0077] Sub-step S33: using the camera grayscale image and the camera depth image as inputs of a training set of a neural network, and using the reference depth image as a true value, and performing training based on the input and the true value to construct a target model.

[0078] Specifically, the embodiment of the present application uses the camera grayscale image generated when the ITOF camera takes an image and the camera depth map with multipath interference as the input of the neural network, and uses the laser radar depth map without multipath interference (the reference depth map) obtained in substep S31 as the true value. In a specific implementation, the embodiment of the present application can use the Adam optimizer to dynamically adjust the learning rate to train the neural network to obtain the ITOF camera target depth map without multipath interference. For example, the embodiment can convert the ITOF camera target depth map without multipath interference into a dense point cloud of the ITOF camera target without multipath interference through the internal parameter information of the ITOF camera (new light coordinates, focal length, lens distortion type, distortion coefficient, image height and width).

[0079] In addition, since the input of the neural network only includes the depth map and grayscale map of the ITOF camera with multipath interference, it does not include the LiDAR depth map without multipath interference. Therefore, when the constructed target model is used to correct the point cloud data received by the ITOF camera with multipath interference, it is no longer necessary to use the LiDAR data without multipath interference. Therefore, the embodiment of the present application can use network distillation and pruning algorithms to compress the trained neural network model to obtain the final target model, which can reduce the complexity of the neural network, reduce computing costs and storage requirements, while maintaining similar performance of the neural network. In this way, the final target model can be deployed on the ITOF camera to achieve real-time correction of the multipath interference of the ITOF camera.

[0080] refer to Figure 3 , Figure 3 This is a structural block diagram of an ITOF camera provided in an embodiment of the present application, wherein the ITOF camera includes a signal transmitting module 01, a signal receiving module 02, a signal processing module 03, and the target model as described above;

[0081] The signal transmitting module 01 is used to transmit a light wave signal, wherein the light wave signal forms a reflection signal after being emitted to the object to be measured;

[0082] The signal receiving module 02 is used to receive the reflected signal and transmit the reflected signal to the signal processing module;

[0083] The signal processing module 03 is used to determine the corresponding dense point cloud to be corrected based on the reflected signal, and call the target model to correct the dense point cloud to be corrected, so as to output a target dense point cloud that is not affected by multipath interference.

[0084] Further, refer to Figure 4 After the target model is constructed, the embodiment of the present application can use the constructed target model to correct the point cloud data received by the ITOF camera that is interfered by multipath. The correction method mainly includes the following steps A1 to A4:

[0085] The method further comprises:

[0086] A1: obtaining a dense point cloud to be corrected collected by the ITOF camera when photographing the object to be measured at the kth moment, wherein the dense point cloud to be corrected represents a reflection signal subject to multipath interference received by the ITOF camera at the kth moment;

[0087] A2: Determine the grayscale image and depth image to be measured generated by the ITOF camera when photographing the object to be measured at the kth moment;

[0088] A3: inputting the grayscale image to be measured and the depth image to be measured into a target model, and the target model outputs a target depth image that is not affected by multipath interference;

[0089] A4: Determine a target dense point cloud that is not affected by multipath based on the target depth map.

[0090] In the present application examples, reference Figure 5 , and finally the target model includes an encoder, a decoder, and a spatial propagation network; after the target model is deployed to the ITOF camera, the target model can be called at any time to correct the point cloud data received by the ITOF camera that is interfered by multipath (the kth moment mentioned in the above description can be any moment in the camera's working process, k is a positive number). It is mainly to correct the dense point cloud to be corrected that is interfered by multipath when the ITOF camera shoots the object under test at the kth moment.

[0091] In a specific implementation, both the encoder and decoder parts can be composed of ResNet, and their weights are initialized using a model pre-trained on the ImageNet dataset. The grayscale image to be tested and the depth image to be tested taken by the ITOF camera are input to the encoder, and the encoder encodes the input data to obtain the encoding result. The encoding result is used as the input of the decoder, and the decoder decodes the encoding result data. The encoder and decoder adopt a step-by-step connection architecture to effectively extract the spatial information of the depth map to be tested and the grayscale image to be tested at different scales. The depth map to be tested and the grayscale image to be tested generated by the ITOF camera can generate initial depth map prediction data through the encoder and decoder architecture.

[0092] Afterwards, the spatial propagation network is called to determine the image data to be eliminated from the initial depth map prediction data, so that the target model outputs a target depth map that is not interfered by multipath, wherein the image data to be eliminated represents the edge position of the object under test.

[0093] It is understandable that, through experiments, the embodiments of the present application have found that multipath interference mostly exists at the edge of the object being measured. Therefore, the embodiments of the present application use an advanced dynamic spatial propagation network to refine the initial depth map prediction data to eliminate multipath interference.

[0094] Specifically, the propagation process of the dynamic space propagation network can be expressed by the following formula:

[0095]

[0096] in,

[0097]

[0098]

[0099] Indicates h t The value at (i, j), Indicates the distance Neighbor, w i，j (a, b) are the affinity matrix weights between (i, j) and its neighbors. The depth map output by the dynamic spatial propagation network is the target depth map after the ITOF camera eliminates multipath interference.

[0100] Finally, the ITOF target depth map without multipath interference is converted into a dense point cloud of the ITOF camera target without multipath interference through the internal reference information of the ITOF camera.

[0101] Further, refer to Figure 6 , Figure 6 A device for constructing a target model for correcting ITOF camera point cloud data is provided in an embodiment of the present application, and the device includes:

[0102] A first acquisition unit 10 is used to control the ITOF camera to be in an off state, and to acquire a sparse point cloud detected by the laser radar in an on state, wherein the sparse point cloud represents data scanned by the laser radar;

[0103] A second acquisition unit 20 is used to control the laser radar to be in an off state, and acquire a dense point cloud collected by the ITOF camera in an on state, wherein the dense point cloud represents a reflection signal received by the ITOF camera;

[0104] The construction unit 30 is used to construct a target model based on the sparse point cloud and the dense point cloud, and the target model is used to correct the point cloud data received by the ITOF camera that is affected by multipath interference.

[0105] It should be noted that the above device can be understood as Figure 1 The virtual device in terminal 1, terminal 1 executes the functions of each module in the device for constructing a target model for correcting ITOF camera point cloud data provided in the embodiment of the present application by running processor 1001; the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application, and their specific functions and technical effects brought about can be specifically referred to the method embodiment part, which will not be repeated here.

[0106] In addition, an embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for correcting ITOF camera point cloud data, characterized in that: The method comprises: Controlling the ITOF camera to be in an off state, and obtaining a sparse point cloud detected by the laser radar in an on state, wherein the sparse point cloud represents data scanned by the laser radar; Control the laser radar to be in an off state, and obtain a dense point cloud collected by the ITOF camera in an on state, wherein the dense point cloud represents a reflection signal received by the ITOF camera; A target model is constructed based on the sparse point cloud and the dense point cloud, and the target model is used to correct the point cloud data received by the ITOF camera that is affected by multipath interference.

2. The method according to claim 1, characterized in that The dense point cloud is composed of a camera depth map generated when the ITOF camera captures an image; The constructing the target model based on the sparse point cloud and the dense point cloud comprises: Transforming the sparse point cloud into a camera coordinate system of the ITOF camera to obtain a reference depth map aligned with the camera depth map; Determine a camera grayscale image generated when the ITOF camera captures an image; The camera grayscale image and the camera depth image are used as inputs of a training set of a neural network, and the reference depth image is used as a true value. Training is performed based on the input and the true value to construct a target model.

3. The method according to claim 2, characterized in that The step of transforming the sparse point cloud into a camera coordinate system of the ITOF camera to obtain a reference depth map aligned with the camera depth map comprises: Registering the sparse point cloud with the dense point cloud to obtain a spatial transformation matrix; The sparse point cloud is transformed into a camera coordinate system of the ITOF camera based on the spatial transformation matrix to obtain a reference depth map aligned with the camera depth map.

4. The method according to claim 2 or 3, characterized in that After completing the construction of the target model, the method further includes: Acquire a dense point cloud to be corrected that is collected by the ITOF camera when photographing the object to be measured at the kth moment, wherein the dense point cloud to be corrected represents a reflection signal that is subject to multipath interference and is received by the ITOF camera at the kth moment; Determine the grayscale image to be measured and the depth image to be measured generated when the ITOF camera photographs the object to be measured at the kth moment; Inputting the grayscale image to be measured and the depth image to be measured into a target model, and outputting a target depth image that is not affected by multipath interference by the target model; A target dense point cloud free from multipath interference is determined based on the target depth map.

5. The method according to claim 4, characterized in that The determining, based on the target depth map, a target dense point cloud that is not interfered by multipath includes: Determining the internal reference information of the ITOF camera; Based on the intrinsic reference information, the target depth map is converted into a target dense point cloud that is not affected by multipath interference.

6. The method according to claim 4, characterized in that The target model includes an encoder, a decoder and a spatial propagation network; The step of inputting the grayscale image to be measured and the depth image to be measured into a target model, and outputting a target depth image that is not interfered by multipath by the target model, comprises: Inputting the grayscale image to be tested and the depth image to be tested as input data into the encoder, so that the encoder encodes the input data to obtain an encoding result; Inputting the encoding result to the decoder, so that the decoder decodes the encoding result to obtain initial depth map prediction data; The spatial propagation network is called to determine the image data to be eliminated from the initial depth map prediction data, so that the target model outputs a target depth map that is not interfered by multipath, wherein the image data to be eliminated represents the edge position of the object under test.

7. An ITOF camera, characterized in that: The ITOF camera comprises a signal transmitting module, a signal receiving module, a signal processing module, and a target model as claimed in any one of claims 1 to 6; The signal transmitting module is used to transmit a light wave signal, wherein the light wave signal forms a reflection signal after being emitted to the object to be measured; The signal receiving module is used to receive the reflected signal and transmit the reflected signal to the signal processing module; The signal processing module is used to determine the corresponding dense point cloud to be corrected based on the reflected signal, and call the target model to correct the dense point cloud to be corrected, so as to output a target dense point cloud that is not affected by multipath interference.

8. A device for constructing a target model for correcting ITOF camera point cloud data, characterized in that: The device comprises: A first acquisition unit, used to control the ITOF camera to be in an off state, and to acquire a sparse point cloud detected by the laser radar in an on state, wherein the sparse point cloud represents data scanned by the laser radar; A second acquisition unit is used to control the laser radar to be in an off state, and acquire a dense point cloud collected by the ITOF camera in an on state, wherein the dense point cloud represents a reflection signal received by the ITOF camera; A construction unit is used to construct a target model based on the sparse point cloud and the dense point cloud, and the target model is used to correct the point cloud data received by the ITOF camera that is affected by multipath interference.

9. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the steps of the method for correcting ITOF camera point cloud data described in any one of claims 1 to 6 are implemented.

10. A storage medium, wherein the storage medium is a computer-readable storage medium, which stores a computer program, and when the processor executes the computer program, the steps of the method for correcting ITOF camera point cloud data according to any one of claims 1 to 6 are implemented.

Images