Compression method and apparatus for neural network models based on eliminating Time-of-Flight (TOF) multipath.

Abstract

This application relates to a compression method for neural network models based on eliminating Time-of-Flight (TOF) multipath propagation. The method includes: obtaining TOF data of a target acquired by a TOF device, the TOF data including raw data and confidence map data; preprocessing the raw data to obtain raw data after removing systematic errors; training a preset neural network model using the target's ground truth data, the raw data after removing systematic errors, and the confidence map data to obtain a first neural network model; and compressing the first neural network model using a knowledge distillation compression algorithm based on the ground truth data, the raw data after removing systematic errors, and the confidence map data to obtain a second neural network model. According to the solution provided in this application, the impact of multipath propagation by TOF cameras can be reduced or eliminated, which is beneficial to improving the accuracy of TOF cameras in acquiring depth data or point cloud data, and can meet the needs of mobile deployment.

Description

Technical Field

[0001] This application relates to the field of three-dimensional imaging technology, and in particular to a compression method and apparatus based on a neural network model that eliminates TOF multipath. Background Technology

[0002] Time-of-flight (TOF) technology calculates the distance to a target object by measuring the time difference or phase difference between the emission of a light beam to the target area, its reflection from the target object, and its reception, in order to obtain depth data or point cloud data information of the measured object.

[0003] Time-of-Flight (TOF) devices, such as TOF cameras, have begun to be used in fields such as 3D measurement, pose estimation, robot navigation, security, and surveillance. However, current TOF cameras suffer from multipath propagation, especially in narrow scenarios such as corners, leading to low accuracy in the acquired depth or point cloud data. Furthermore, the computational demands of existing deep learning-based solutions on the neural network models cannot be effectively deployed on mobile devices. Summary of the Invention

[0004] Based on this, this application provides a deep learning-based TOF multipath correction scheme and its mobile deployment solution. This solution can reduce or eliminate the impact of multipath phenomena caused by TOF cameras, while also reducing noise in point cloud data, which is beneficial to improving the accuracy of TOF cameras in acquiring depth data or point cloud data. Furthermore, it features a lightweight design and compresses the neural network model to meet the needs of mobile deployment.

[0005] According to a first aspect of this application, a compression method for a neural network model based on eliminating Time-of-Flight (TOF) multipath is provided, comprising:

[0006] Obtain TOF data of the target collected by the TOF device, wherein the TOF data includes raw data and confidence map data;

[0007] The original data is preprocessed to obtain the original data after removing system errors;

[0008] The preset neural network model is trained based on the target's true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model.

[0009] Based on the truth data, the original data after removing systematic errors, and the confidence graph data, the first neural network model is compressed using a knowledge distillation compression algorithm to obtain the second neural network model.

[0010] According to a second aspect of this application, a compression device for a neural network model based on eliminating TOF multipath is provided, comprising:

[0011] The first acquisition unit is used to acquire TOF data of the target collected by the TOF device, wherein the TOF data includes raw data and confidence map data;

[0012] The preprocessing unit is used to preprocess the raw data to obtain the raw data after removing system errors;

[0013] The second obtaining unit is used to train a preset neural network model based on the target's true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model.

[0014] The first compression unit is used to compress the first neural network model using a knowledge distillation compression algorithm based on the truth data, the original data after removing systematic errors, and the confidence graph data to obtain the second neural network model.

[0015] According to a third aspect of the present invention, an electronic device is provided, comprising:

[0016] Processor; and

[0017] A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in the first aspect.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having computer-readable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method as described in the first aspect.

[0019] According to the compression scheme for neural network models based on eliminating TOF multipath provided in this application, on the one hand, the original data and confidence map data in the target's TOF data are used as inputs to train the neural network model, and the ground truth data of the target is used as supervision data in the neural network training process. This can obtain corrected ray distance data, improve the accuracy of depth data or point cloud data, and reduce or eliminate the impact of multipath phenomenon caused by the TOF device. On the other hand, based on the original data, confidence map data, and ground truth data in the target's TOF data, a knowledge distillation algorithm is used to compress the neural network model after training. This can reduce the amount of computation and the size of the trained neural network model while ensuring the accuracy of the neural network model. Furthermore, based on the knowledge distillation algorithm, lightweight processing, neural architecture search algorithm, and quantization algorithm can be used to further compress the neural network model to meet the needs of mobile deployment. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0021] Figure 1 This is a flowchart of a compression method for a neural network model based on eliminating TOF multipath according to an embodiment of this application.

[0022] Figure 2 This is a schematic diagram illustrating the training or testing of a preset neural network model based on the original data after removing systematic errors, confidence graph data, and ground truth data, according to an embodiment of this application.

[0023] Figure 3 The image shows the effect of using GAN to solve the multipath phenomenon in TOF according to the embodiments of this application.

[0024] Figure 4 This is a schematic diagram of a compression device based on a neural network model that eliminates TOF multipath according to an embodiment of this application.

[0025] Figure 5 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] This application relates to a compression scheme for a neural network model based on eliminating TOF multipath. The scheme mainly includes: first, training a preset neural network model based on the original data, confidence map data and ground truth data collected from the target; then, compressing the trained neural network model based on the original data, confidence map data and ground truth data of the target using a knowledge distillation algorithm.

[0028] According to one aspect of this application, a compression method for neural network models based on eliminating TOF multipath is provided. Figure 1 This is a flowchart of a compression method based on a neural network model that eliminates Time-of-Flight (TOF) multipath according to an embodiment of this application. Figure 1As shown, the method includes the following steps.

[0029] Step S101: Obtain the TOF data of the target collected by the TOF device. The TOF data includes raw data and confidence map data.

[0030] In one embodiment, data is acquired using a Time-of-Flight (TOF) device (e.g., a TOF camera). The acquired data is TOF data, including raw data and confidence map data. The resolution of the confidence map data is consistent with the resolution of the TOF device. The TOF data includes data with and without multipath interference.

[0031] Step S102: Perform data preprocessing on the original data to obtain the original data after removing system errors.

[0032] The raw RAW data acquired by the TOF device is preprocessed to remove systematic errors, resulting in the original data Correct_RAW after systematic error removal. Systematic error removal includes temperature drift correction, non-sinusoidal signal correction, fixed phase shift, dark noise removal, and stray light removal.

[0033] Taking stray light removal as an example, the formula for stray light removal is Smeasure=S+S*h, where S is the ideal correct signal, Smeasure is the acquired signal with stray light, and h is the signal noise. The data after stray light removal can be obtained through calibration algorithm.

[0034] Experiments have shown that removing systematic errors from the original data can improve the generalization ability of neural network models, giving them better generalization capabilities.

[0035] Step S103: Train the preset neural network model using the target's true data, the original data after removing systematic errors, and the confidence map data to obtain the first neural network model.

[0036] According to one embodiment, a truth-based system is built. While the Time-of-Flight (TOF) device acquires target data, the truth-based system simultaneously acquires target data to obtain a truth dataset. The truth-based system includes hardware capable of acquiring 3D data, such as LiDAR, a 3D scanner, and a depth camera. The TOF device and the truth-based system can be integrated into the same device or system. The target can be a region or an object.

[0037] Step S103 includes:

[0038] Sub-step S1031: Input the original data after removing systematic errors and the confidence map into a preset neural network model, use the ground truth data as supervision data, and obtain one or more candidate neural network models corresponding to one or more iterations.

[0039] In one embodiment, the preset neural network model can be any neural network model, such as a GAN (Generative Adversarial Network) model. Multiple sets of raw data (after removing systematic errors), confidence map data, and real data are used to form a test set to train the preset neural network model. Specifically, the raw data (after removing systematic errors) and the confidence map data are used as inputs to the preset neural network model. Parameters such as the number of iterations and the learning rate are set to obtain the output of the preset neural network model, which can be the corrected ray distance. Ray distance and ground truth data are of the same type of data. By comparing the ground truth data as supervision data with the corrected ray distance, the accuracy of the preset neural network can be obtained.

[0040] The number of iterations for a neural network model can be selected based on needs or experience, and multiple iterations can be selected. Multiple iterations correspond to multiple trained candidate neural network models, which have different network parameters.

[0041] Figure 2 This is a schematic diagram illustrating the training or testing of a preset neural network model based on the original data after removing systematic errors, confidence plot data, and ground truth data, according to an embodiment of this application. Figure 2 The diagram shown provides a more intuitive illustration of the process of training or testing a preset neural network model.

[0042] In the process of training a preset neural network model, the objective function used is based on the true data, the original data after removing systematic errors, and the confidence graph data. This objective function is referred to as the first objective function.

[0043] In one specific embodiment, taking a pre-defined neural network model such as a GAN network as an example, the expression of the first objective function is:

[0044] L total =L cgan +λ L1 L L1 +λ TV L TV (1)

[0045] in,

[0046] A GAN network consists of a generator network G and a discriminator network D;

[0047] L total It is the overall objective function of the GAN network.

[0048] L cgan It is the objective function of the D network;

[0049]

[0050] L L1 =E x，y [||yG(x)||1]

[0051] L TV =F(IR, confidence)

[0052] E represents the dataset.

[0053] E x This means that the dataset is composed of x; x represents the TOF data, including the original data after removing systematic errors and the confidence plot data; x represents the input data of the generator network G.

[0054] E y This means that the dataset consists of y; y includes truth data;

[0055] E x，y This means that the dataset consists of x and y;

[0056] L L1 represents the first-order norm, and ||||1 indicates taking the first-order norm;

[0057] L TV The smoothing term is obtained by nonlinear transformation of the IR plot and the confidence plot, where F represents the nonlinear transformation; the IR plot is calculated from the original data Correct_RAW after removing systematic errors.

[0058] Sub-step S1032: If there is more than one candidate neural network model, the candidate neural network models are tested using a validation set, and the candidate neural network model with the highest test accuracy is determined as the first neural network model.

[0059] When multiple candidate neural network models are available, they are tested using a validation set. The neural network model with the highest test accuracy is selected as the first neural network model. The validation set consists of the original data of the target after removing systematic errors, confidence plot data, and real data. It may be data separately collected by the truth system and the Time-of-Flight (TOF) device, and is different from the test set mentioned above.

[0060] The accuracy of multiple candidate neural network models is compared, and the candidate neural network model with the highest accuracy is determined as the first neural network model for subsequent compression.

[0061] Step S104: Based on the truth data, the original data after removing systematic errors, and the confidence graph data, the first neural network model is compressed using a knowledge distillation compression algorithm to obtain the second neural network model.

[0062] After obtaining the first neural network model with the highest accuracy, it needs to be compressed to facilitate deployment on mobile devices and improve computing speed.

[0063] Generally speaking, knowledge distillation algorithms can reduce computation and model size while preserving the accuracy of neural network models. However, when the compression ratio exceeds 4 times, the accuracy of the compressed neural network model will decrease. That is, if conventional knowledge distillation algorithms are used directly to compress neural network models, the accuracy of the compressed model cannot be guaranteed.

[0064] In order to compress the neural network model while ensuring accuracy, in this application, the objective function used in the process of compressing the first neural network model by the knowledge distillation compression algorithm is an objective function based on the true data of the target, the original data after removing systematic errors, and the confidence map data. This objective function is referred to as the second objective function.

[0065] In one specific embodiment, taking a pre-defined neural network model such as a GAN network as an example, the expression of the second objective function is:

[0066] L total =L cgan +λ L1 L L1 +λ TV L TV +λ distill L distill (2)

[0067] in,

[0068] A GAN network consists of a generator network G and a discriminator network D;

[0069] L total It is the overall objective function of the GAN network.

[0070] L cgan It is the objective function of the D network;

[0071]

[0072] LL1 =E x，y [||yG(x)||1]

[0073] L TV =F(IR, confidence)

[0074] E represents the dataset.

[0075] E x This means that the dataset is composed of x; x represents the TOF data, including the original data after removing systematic errors and the confidence plot data; x represents the input data of the generator network G.

[0076] E y This means that the dataset consists of y; y includes truth data;

[0077] E x，y This means that the dataset consists of x and y; L L1 represents the first-order norm, and ||||1 indicates taking the first-order norm;

[0078] L TV The smoothing term is obtained by nonlinear transformation of the IR plot and the confidence plot, where F represents the nonlinear transformation; the IR plot is calculated from the original data Correct_RAW after removing systematic errors.

[0079] L distill This represents the error in the knowledge distillation term.

[0080] By employing the second objective function proposed in this application in the knowledge distillation compression algorithm, compression of neural network models can be achieved while ensuring accuracy.

[0081] The compression method for neural network models based on eliminating TOF multipath proposed in this application further includes the following steps:

[0082] Step S105: Before compressing the first neural network model using the knowledge distillation compression algorithm, the first neural network model is lightened to obtain a lightened first neural network model.

[0083] In one embodiment, lightweighting the first neural network model can compress the first neural network model while ensuring accuracy. After lightweighting, the knowledge distillation algorithm can be applied to the first neural network model, which can further compress the size of the second neural network model after the knowledge distillation algorithm.

[0084] In one embodiment, lightweighting includes replacing the original convolution in the first neural network model with a depthwise separable convolution.

[0085] Thus, step S104 includes: compressing the lightweight first neural network model using a knowledge distillation compression algorithm.

[0086] Step S106: The second neural network model is compressed using a neural architecture search algorithm to obtain the third neural network model.

[0087] The second neural network model, compressed using the knowledge distillation algorithm, is then compressed a second time using neural architecture search to obtain the third neural network model. This allows for further compression of the neural network model while maintaining accuracy.

[0088] Step S107: The third neural network model is processed by a quantization algorithm to obtain the fourth neural network model.

[0089] The third neural network model, compressed using the neural architecture search algorithm, is then processed using a quantization algorithm to obtain the fourth neural network model. In one embodiment, the quantization algorithm includes converting the original float data into data formats such as int16 and uint8. This makes it easier to deploy the neural network model on mobile devices and improves the computation speed of the neural network model.

[0090] Figure 3 The image shows the effect of using GAN to solve the multipath phenomenon in TOF according to the embodiments of this application.

[0091] exist Figure 3 In this context, the neural network model is a GAN network, a generative model that uses a generator network G (Generator) and a discriminator network D (Discriminator) to continuously play a game, enabling the generator network G to learn the distribution of data and use the GAN network to solve the problem.

[0092] GANs are effective in addressing the multipath phenomenon in Time-of-Flight (TOF) scenarios. Figure 3 As shown, taking a scene of shooting a corner as an example, 1 is the original point cloud, and 2 is the point cloud after correction by the GAN network. It can be seen that the effect is good.

[0093] Experiments show that the original network model (the first neural network model) is compressed by about 100 times for mobile deployment after lightweight network design, knowledge distillation algorithm, neural architecture search, and quantization algorithm. At the same time, the computing speed is improved by about 8 times.

[0094] According to the compression method for neural network models based on eliminating TOF multipath provided in this application, on the one hand, the original data and confidence map data in the target's TOF data are used as inputs to train the neural network model, and the ground truth data of the target is used as supervision data in the neural network training process. This can obtain corrected ray distance data, improve the accuracy of depth data or point cloud data, and reduce or eliminate the impact of multipath phenomenon caused by the TOF device. On the other hand, based on the original data, confidence map data, and ground truth data in the target's TOF data, a knowledge distillation algorithm is used to compress the neural network model after training. This can reduce the amount of computation and the size of the trained neural network model while ensuring the accuracy of the neural network model. Furthermore, based on the knowledge distillation algorithm, lightweight processing, neural architecture search algorithm, and quantization algorithm can be used to further compress the neural network model to meet the needs of mobile deployment.

[0095] According to another aspect of this application, a compression device based on a neural network model that eliminates TOF multipath is provided. Figure 4 This is a schematic diagram of a compression device based on a neural network model that eliminates Time-of-Flight (TOF) multipath according to an embodiment of this application. Figure 4 As shown, the device includes the following units.

[0096] The first acquisition unit 401 is used to acquire TOF data of the target collected by the TOF device, wherein the TOF data includes raw data and confidence map data.

[0097] In one embodiment, data is acquired using a Time-of-Flight (TOF) device (e.g., a TOF camera). The acquired data is TOF data, including raw data and confidence map data. The resolution of the confidence map data is consistent with the resolution of the TOF device. The TOF data includes data with and without multipath interference.

[0098] The preprocessing unit 402 is used to preprocess the raw data to obtain the raw data after removing system errors.

[0099] The raw RAW data acquired by the TOF device is preprocessed to remove systematic errors, resulting in the original data Correct_RAW after systematic error removal. Systematic error removal includes temperature drift correction, non-sinusoidal signal correction, fixed phase shift, dark noise removal, and stray light removal.

[0100] Taking stray light removal as an example, the formula for stray light removal is Smeasure=S+S*h, where S is the ideal correct signal, Smeasure is the acquired signal with stray light, and h is the signal noise. The data after stray light removal can be obtained through calibration algorithm.

[0101] Experiments have shown that removing systematic errors from the original data can improve the generalization ability of neural network models, giving them better generalization capabilities.

[0102] The second obtaining unit 403 is used to train a preset neural network model based on the true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model.

[0103] According to one embodiment, a truth-based system is built. While the Time-of-Flight (TOF) device acquires target data, the truth-based system simultaneously acquires target data to obtain a truth dataset. The truth-based system includes hardware capable of acquiring 3D data, such as LiDAR, a 3D scanner, and a depth camera. The TOF device and the truth-based system can be integrated into the same device or system. The target can be a region or an object.

[0104] The second obtaining unit 403 includes:

[0105] Subunit 4031 is used to input the original data after removing systematic errors and the confidence map into a preset neural network model, and use the true data as supervision data to obtain one or more candidate neural network models corresponding to one or more iterations.

[0106] In one embodiment, the preset neural network model can be any neural network model, such as a GAN (Generative Adversarial Network) model. Multiple sets of raw data (after removing systematic errors), confidence map data, and real data are used to form a test set to train the preset neural network model. Specifically, the raw data (after removing systematic errors) and the confidence map data are used as inputs to the preset neural network model. Parameters such as the number of iterations and the learning rate are set to obtain the output of the preset neural network model, which can be the corrected ray distance. Ray distance and ground truth data are of the same type of data. By comparing the ground truth data as supervision data with the corrected ray distance, the accuracy of the preset neural network can be obtained.

[0107] The number of iterations for a neural network model can be selected based on needs or experience, and multiple iterations can be selected. Multiple iterations correspond to multiple trained candidate neural network models, which have different network parameters.

[0108] Figure 2 This is a schematic diagram illustrating the training or testing of a preset neural network model based on the original data after removing systematic errors, confidence plot data, and ground truth data, according to an embodiment of this application. Figure 2 The diagram shown provides a more intuitive illustration of the process of training or testing a preset neural network model.

[0109] In the process of training a preset neural network model, the objective function used is based on the true data, the original data after removing systematic errors, and the confidence graph data. This objective function is referred to as the first objective function.

[0110] In one specific embodiment, the expression of the first objective function is as shown in expression (1) above.

[0111] The determination subunit 4032 is used to test the candidate neural network models through a validation set when there are more than one candidate neural network model, and determine the candidate neural network model with the highest test accuracy as the first neural network model.

[0112] When multiple candidate neural network models are available, they are tested using a validation set. The neural network model with the highest test accuracy is selected as the first neural network model. The validation set consists of the original data of the target after removing systematic errors, confidence plot data, and real data. It may be data separately collected by the truth system and the Time-of-Flight (TOF) device, and is different from the test set mentioned above.

[0113] The accuracy of multiple candidate neural network models is compared, and the candidate neural network model with the highest accuracy is determined as the first neural network model for subsequent compression.

[0114] The first compression unit 404 is used to compress the first neural network model using a knowledge distillation compression algorithm based on the truth data, the original data after removing systematic errors, and the confidence map data to obtain the second neural network model.

[0115] After obtaining the first neural network model with the highest accuracy, it needs to be compressed to facilitate deployment on mobile devices and improve computing speed.

[0116] Generally speaking, knowledge distillation algorithms can reduce computation and model size while preserving the accuracy of neural network models. However, when the compression ratio exceeds 4 times, the accuracy of the compressed neural network model will decrease. That is, if conventional knowledge distillation algorithms are used directly to compress neural network models, the accuracy of the compressed model cannot be guaranteed.

[0117] In order to compress the neural network model while ensuring accuracy, in this application, the objective function used in the process of compressing the first neural network model by the knowledge distillation compression algorithm is an objective function based on the true data of the target, the original data after removing systematic errors, and the confidence map data. This objective function is referred to as the second objective function.

[0118] In one specific embodiment, the expression of the second objective function is as shown in expression (2) above.

[0119] By employing the second objective function proposed in this application in the knowledge distillation compression algorithm, compression of neural network models can be achieved while ensuring accuracy.

[0120] The compression device based on the neural network model that eliminates TOF multipath proposed in this application also includes the following units:

[0121] The lightweight processing unit 405 is used to perform lightweight processing on the first neural network model before compressing it using the knowledge distillation compression algorithm, so as to obtain a lightweight first neural network model.

[0122] In one embodiment, lightweighting the first neural network model can compress the first neural network model while ensuring accuracy. After lightweighting, the knowledge distillation algorithm can be applied to the first neural network model, which can further compress the size of the second neural network model after the knowledge distillation algorithm.

[0123] In one embodiment, lightweighting includes replacing the original convolution in the first neural network model with a depthwise separable convolution.

[0124] Thus, the first compression unit 404 is used to compress the lightweight first neural network model using a knowledge distillation compression algorithm.

[0125] The second compression unit 406 is used to compress the second neural network model using a neural architecture search algorithm to obtain a third neural network model.

[0126] The second neural network model, compressed using the knowledge distillation algorithm, is then compressed a second time using neural architecture search to obtain the third neural network model. This allows for further compression of the neural network model while maintaining accuracy.

[0127] The quantization processing unit 407 is used to process the third neural network model through a quantization algorithm to obtain the fourth neural network model.

[0128] The third neural network model, compressed using the neural architecture search algorithm, is then processed using a quantization algorithm to obtain the fourth neural network model. In one embodiment, the quantization algorithm includes converting the original float data into data formats such as int16 and uint8. This makes it easier to deploy the neural network model on mobile devices and improves the computation speed of the neural network model.

[0129] According to the compression device for a neural network model based on eliminating TOF multipath provided in this application, on the one hand, the original data and confidence map data in the target's TOF data are used as inputs to train the neural network model, and the ground truth data of the target is used as supervision data in the neural network training process. This can obtain corrected ray distance data, improve the accuracy of depth data or point cloud data, and reduce or eliminate the impact of multipath phenomenon caused by the TOF device. On the other hand, based on the original data, confidence map data, and ground truth data in the target's TOF data, a knowledge distillation algorithm is used to compress the neural network model after training. This can reduce the amount of computation and the size of the trained neural network model while ensuring the computational accuracy of the neural network model. Furthermore, based on the knowledge distillation algorithm, lightweight processing, neural architecture search algorithm, and quantization algorithm can be used to further compress the neural network model to meet the needs of mobile deployment.

[0130] See Figure 5 , Figure 5 An electronic device is provided, including a processor; and a memory storing computer instructions that, when executed by the processor, cause the processor to perform the computer instructions as follows: Figure 1 The method and its detailed scheme are shown.

[0131] It should be understood that the above-described device embodiments are merely illustrative, and the device disclosed in this invention can be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, integrated into another system, or some features may be ignored or not executed.

[0132] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of the present invention can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0133] If the integrated unit / module is implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor or chip can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the on-chip cache, off-chip memory, and storage can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0134] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when executed by multiple processors, causes the processors to perform actions such as... Figure 1 The method and its detailed scheme are shown.

[0136] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A compression method for a neural network model based on eliminating Time-of-Flight (TOF) multipath propagation, comprising: Obtain TOF data of the target collected by the TOF device, wherein the TOF data includes raw data and confidence map data; The original data is preprocessed to obtain the original data after removing system errors; The preset neural network model is trained based on the target's true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model. Based on the true data, the original data after removing systematic errors, and the confidence graph data, the first neural network model is compressed using a knowledge distillation compression algorithm to obtain a second neural network model; The first objective function used for training the preset neural network model is based on the ground truth data, the original data after removing systematic errors, and the confidence graph data. When the preset neural network model is a GAN network, the first objective function is: L total =L cgan +λ L1 L L1 +λ TV L TV in, A GAN network consists of a generator network G and a discriminator network D; L total It is the overall objective function of the GAN network. L cgan It is the objective function of the D network; L L1 =E x,y [||y-G(x)||1] L TV =F(IR,confidence) E represents the dataset. E x This means that the dataset is composed of x; x represents the TOF data, including the original data after removing systematic errors and the confidence plot data; x represents the input data of the generator network G. E y This means that the dataset consists of y; y includes truth data; E x,y This means that the dataset consists of x and y. L L1 represents the first-order norm, and || ||1 indicates taking the first-order norm; L TV The smoothing term is obtained by nonlinear transformation of the IR plot and the confidence plot, where F represents the nonlinear transformation; the IR plot is calculated from the original data Correct_RAW after removing systematic errors.

2. The compression method as described in claim 1, wherein, The step of training a preset neural network model using the target's true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model includes: The original data after removing systematic errors and the confidence map data are input into a preset neural network model, and the ground truth data is used as supervision data to obtain one or more candidate neural network models corresponding to one or more iterations.

3. The compression method as described in claim 2, wherein, The step of training a preset neural network model using the target's true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model further includes: When there is more than one candidate neural network model, the candidate neural network models are tested using a validation set, and the candidate neural network model with the highest test accuracy is determined as the first neural network model.

4. The compression method as described in claim 1, wherein, The second objective function used in the process of compressing the first neural network model using the knowledge distillation compression algorithm is an objective function based on the ground truth data of the objective, the original data after removing systematic errors, and the confidence plot data.

5. The compression method as described in claim 4, wherein, When the preset neural network model is a GAN network, the second objective function is: L total =L cgan +λ L1 L L1 +λ TV L TV +λ distill L distill in, A GAN network consists of a generator network G and a discriminator network D; L total It is the overall objective function of the GAN network. L cgan It is the objective function of the D network; L L1 =E x,y [||y-G(x)||1] L TV =F(IR,confidence) E represents the dataset. E x This means that the dataset is composed of x; x represents the TOF data, including the original data after removing systematic errors and the confidence plot data; x represents the input data of the generator network G. E y This means that the dataset consists of y; y includes truth data; E x,y This means that the dataset consists of x and y. L L1 represents the first-order norm, and || ||1 indicates taking the first-order norm; L TV The smoothing term is obtained by nonlinear transformation of the IR plot and the confidence plot, where F represents the nonlinear transformation; the IR plot is calculated from the original data Correct_RAW after removing systematic errors. L distill This represents the error in the knowledge distillation term.

6. The compression method as described in claim 1, further comprising: Before compressing the first neural network model using the knowledge distillation compression algorithm, the first neural network model is lightweighted to obtain a lightweight first neural network model. The compression of the first neural network model using the knowledge distillation compression algorithm includes: compressing the lightweight first neural network model using the knowledge distillation compression algorithm.

7. The compression method as described in claim 1, further comprising: The second neural network model is compressed using a neural architecture search algorithm to obtain the third neural network model.

8. The compression method as described in claim 7, further comprising: The third neural network model is processed by a quantization algorithm to obtain the fourth neural network model.

9. A compression device for a neural network model based on eliminating TOF multipath, comprising: The first acquisition unit is used to acquire TOF data of the target collected by the TOF device, wherein the TOF data includes raw data and confidence map data; The preprocessing unit is used to preprocess the raw data to obtain the raw data after removing system errors; The second obtaining unit is used to train a preset neural network model based on the target's true data, the original data after removing systematic errors, and the confidence map data to obtain a first neural network model. The first compression unit is used to compress the first neural network model using a knowledge distillation compression algorithm based on the truth data, the original data after removing systematic errors, and the confidence graph data to obtain the second neural network model. The first objective function used for training the preset neural network model is based on the ground truth data, the original data after removing systematic errors, and the confidence graph data. When the preset neural network model is a GAN network, the first objective function is: L total =L cgan +λ L1 L L1 +λ TV L TV in, A GAN network consists of a generator network G and a discriminator network D; L total It is the overall objective function of the GAN network. L cgan It is the objective function of the D network; L L1 =E x,y [||y-G(x)||1] L TV =F(IR,confidence) E represents the dataset. E x This means that the dataset is composed of x; x represents the TOF data, including the original data after removing systematic errors and the confidence plot data; x represents the input data of the generator network G. E y This means that the dataset consists of y; y includes truth data; E x,y This means that the dataset consists of x and y. L L1 represents the first-order norm, and || ||1 indicates taking the first-order norm; L TV The smoothing term is obtained by nonlinear transformation of the IR plot and the confidence plot, where F represents the nonlinear transformation; the IR plot is calculated from the original data Correct_RAW after removing systematic errors.

10. An electronic device, comprising: processor; as well as A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8.

Images