Millimeter wave image reconstruction method, image reconstruction model training method and device

Abstract

Provided are a millimeter wave image reconstruction method, apparatus, device, medium, and program product, relating to the field of millimeter wave imaging. The method comprises: using a single-frequency millimeter wave to acquire a first echo signal of a target under inspection; obtaining first input data based on the first echo signal; processing the first input data using a pre-trained image reconstruction model to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the target under inspection and first range-direction data of at least one pixel in the first two-dimensional image; and reconstructing a first three-dimensional image of the target under inspection based on the first prediction result. This method can at least partially address the problem of current single-frequency systems lacking range resolution, and achieve, to a certain extent, the imaging performance of a broadband system using a single-frequency system.

Description

Technical Field

[0001] The present disclosure relates to the field of millimeter wave imaging, and more specifically, to a millimeter wave image reconstruction method, an image reconstruction model training method, an apparatus, a device, a medium, and a program product. Background Art

[0002] In traditional millimeter-wave image reconstruction applications, three-dimensional images are obtained using broadband systems with range resolution. However, broadband systems suffer from limitations such as excessive bandwidth usage, high hardware costs, slow speeds, and significant data transmission delays. Single-frequency systems offer advantages such as reduced bandwidth usage and simpler hardware, but they lack range resolution. Therefore, how to ensure range resolution in single-frequency systems for image reconstruction remains a pressing challenge. Summary of the Invention

[0003] In view of the above problems, the present disclosure provides a millimeter wave image reconstruction method, an image reconstruction model training method, an apparatus, a device, a medium and a program product that enable a single-frequency system to have range resolution in image reconstruction.

[0004] One aspect of an embodiment of the present disclosure provides a millimeter wave image reconstruction method, comprising: using a single-frequency millimeter wave to collect a first echo signal of a detected target; obtaining first input data based on the first echo signal; using a pre-trained image reconstruction model to process the first input data to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the detected target and first range data of at least one pixel in the first two-dimensional image; and reconstructing a first three-dimensional image of the detected target based on the first prediction result.

[0005] According to an embodiment of the present disclosure, the image reconstruction model is trained according to the following operations: based on the signal characteristics of the single-frequency millimeter wave acquisition, second input data is obtained through a training target; the second input data is input into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training target and second range data of at least one pixel in the second two-dimensional image; the image reconstruction model is trained according to the contrast loss between the second prediction result and the label data, wherein the label data includes a third two-dimensional image of the training target and third range data of at least one pixel in the third two-dimensional image.

[0006] According to an embodiment of the present disclosure, it also includes: reconstructing a second three-dimensional image of the training target based on the second two-dimensional image and the second range data; wherein the second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model based on the contrast loss between the second prediction result and the label data includes: training the image reconstruction model based on the contrast loss between the second three-dimensional image and the third three-dimensional image.

[0007] According to an embodiment of the present disclosure, the signal characteristics based on the single-frequency millimeter wave acquisition are used to obtain the second input data through the training target, including: using a wide-band millimeter wave to acquire a second echo signal of the training target; obtaining amplitude data and phase data of a first frequency point based on the second echo signal, wherein the wide-band millimeter wave includes the first frequency point; and obtaining the second input data based on the amplitude data and phase data of the first frequency point.

[0008] According to an embodiment of the present disclosure, obtaining the second input data according to the amplitude data and phase data of the first frequency point includes: using the amplitude data and phase data of the first frequency point as the second input data.

[0009] According to an embodiment of the present disclosure, obtaining the second input data based on the amplitude data and phase data of the first frequency point includes: obtaining an unfocused image of each of M positions based on the amplitude data and phase data of the first frequency point, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the unfocused image of each position as the second input data.

[0010] According to an embodiment of the present disclosure, the signal characteristics based on the single-frequency millimeter wave acquisition are used to obtain the second input data through a training target, including: obtaining an unfocused image of each of M positions based on the third two-dimensional image, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the unfocused image of each position as the second input data.

[0011] According to an embodiment of the present disclosure, obtaining the first input data according to the first echo signal includes: using the amplitude data and phase data of the first echo signal as the first input data.

[0012] According to an embodiment of the present disclosure, obtaining the first input data based on the first echo signal includes: obtaining an unfocused image of each of M positions based on the amplitude data and phase data of the first echo signal, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the unfocused image of each position as the first input data.

[0013] Another aspect of an embodiment of the present disclosure provides a millimeter wave image reconstruction method, including: using N single-frequency millimeter waves to collect N first echo signals of a detected target, wherein the center frequencies of any two single-frequency millimeter waves among the N single-frequency millimeter waves are different, and the N single-frequency millimeter waves correspond one-to-one to the N first echo signals, and N is an integer greater than or equal to 2; obtaining N first input data based on the N first echo signals; using a pre-trained image reconstruction model to process the N first input data to obtain N first prediction results, wherein each of the N first prediction results includes a first two-dimensional image of the detected target; performing image fusion based on the N first prediction results to reconstruct a fourth two-dimensional image of the detected target.

[0014] According to an embodiment of the present disclosure, each of the first prediction results also includes first distance data of at least one pixel in the first two-dimensional image, and the method also includes: performing image fusion based on the fourth two-dimensional image and the first distance data in at least one first prediction result to reconstruct a fourth three-dimensional image of the inspected target.

[0015] According to an embodiment of the present disclosure, each of the N single-frequency millimeter waves has a corresponding image reconstruction model, and the image reconstruction model corresponding to each single-frequency millimeter wave is trained according to the following operations: based on the signal characteristics collected by each single-frequency millimeter wave, second input data is obtained through a training target; the second input data is input into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training target and second distance data of at least one pixel in the second two-dimensional image; the image reconstruction model is trained according to the contrast loss between the second prediction result and the label data, wherein the label data includes a third two-dimensional image of the training target and third distance data of at least one pixel in the third two-dimensional image.

[0016] According to an embodiment of the present disclosure, it also includes: reconstructing a second three-dimensional image of the training target based on the second two-dimensional image and the second range data; wherein the second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model based on the contrast loss between the second prediction result and the label data includes: training the image reconstruction model based on the contrast loss between the second three-dimensional image and the third three-dimensional image.

[0017] According to an embodiment of the present disclosure, the second input data is obtained through a training target based on the signal characteristics of each single-frequency millimeter wave acquisition, including: using a wideband millimeter wave to acquire a second echo signal of the training target; obtaining amplitude data and phase data of a first frequency point based on the second echo signal, wherein the wideband millimeter wave includes the first frequency point, and the first frequency point is the same as the center frequency of the single-frequency millimeter wave; and obtaining the second input data based on the amplitude data and phase data of the first frequency point.

[0018] According to an embodiment of the present disclosure, obtaining the second input data according to the amplitude data and phase data of the first frequency point includes: using the amplitude data and phase data of the first frequency point as the second input data.

[0019] According to an embodiment of the present disclosure, obtaining the second input data based on the amplitude data and phase data of the first frequency point includes: obtaining an unfocused image of each of M positions based on the amplitude data and phase data of the first frequency point, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the unfocused image of each position as the second input data.

[0020] According to an embodiment of the present disclosure, the second input data is obtained through a training target based on the signal characteristics of each single-frequency millimeter wave acquisition, including: obtaining a non-focused image of each of M positions based on the third two-dimensional image, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the non-focused image of each position as the second input data.

[0021] According to an embodiment of the present disclosure, obtaining N first input data according to the N first echo signals includes: using amplitude data and phase data of the N first echo signals as the N first input data.

[0022] According to an embodiment of the present disclosure, obtaining N first input data based on N first echo signals includes performing the following operations for each of the first echo signals: obtaining an unfocused image of each of M positions based on the amplitude data and phase data of the first echo signal, wherein the M positions are distributed along the distance direction and M is greater than or equal to 2; and using the unfocused image of each position as the first input data corresponding to the first echo signal.

[0023] Another aspect of an embodiment of the present disclosure provides a method for training an image reconstruction model, comprising: obtaining second input data through a training target based on signal characteristics of single-frequency millimeter wave acquisition; inputting the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training target and second range data of at least one pixel in the second two-dimensional image; training the image reconstruction model based on the contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training target and third range data of at least one pixel in the third two-dimensional image.

[0024] According to an embodiment of the present disclosure, it also includes: reconstructing a second three-dimensional image of the training target based on the second two-dimensional image and the second range data; wherein the second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model based on the contrast loss between the second prediction result and the label data includes: training the image reconstruction model based on the contrast loss between the second three-dimensional image and the third three-dimensional image.

[0025] According to an embodiment of the present disclosure, the signal characteristics based on the single-frequency millimeter wave acquisition are used to obtain the second input data through the training target, including: using a wide-band millimeter wave to acquire a second echo signal of the training target; obtaining amplitude data and phase data of a first frequency point based on the second echo signal, wherein the wide-band millimeter wave includes the first frequency point; and obtaining the second input data based on the amplitude data and phase data of the first frequency point.

[0026] According to an embodiment of the present disclosure, obtaining the second input data according to the amplitude data and phase data of the first frequency point includes: using the amplitude data and phase data of the first frequency point as the second input data.

[0027] According to an embodiment of the present disclosure, obtaining the second input data based on the amplitude data and phase data of the first frequency point includes: obtaining an unfocused image of each of M positions based on the amplitude data and phase data of the first frequency point, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the unfocused image of each position as the second input data.

[0028] According to an embodiment of the present disclosure, the signal characteristics based on the single-frequency millimeter wave acquisition are used to obtain the second input data through a training target, including: obtaining an unfocused image of each of M positions based on the third two-dimensional image, wherein the M positions are distributed along the distance direction, and M is greater than or equal to 2; and using the unfocused image of each position as the second input data.

[0029] Another aspect of an embodiment of the present disclosure provides a millimeter wave image reconstruction device, including: a first acquisition module, used to use a single-frequency millimeter wave to acquire a first echo signal of a detected target; a first processing module, used to obtain first input data based on the first echo signal; a second processing module, used to use a pre-trained image reconstruction model to process the first input data to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the detected target and first range data of at least one pixel in the first two-dimensional image; a first reconstruction module, used to reconstruct a first three-dimensional image of the detected target based on the first prediction result.

[0030] Another aspect of an embodiment of the present disclosure provides a millimeter wave image reconstruction device, including: a second acquisition module, used to use N single-frequency millimeter waves to acquire N first echo signals of the inspected target, wherein the center frequencies of any two single-frequency millimeter waves among the N single-frequency millimeter waves are different, and the N single-frequency millimeter waves correspond one-to-one to the N first echo signals, and N is an integer greater than or equal to 2; a third processing module, used to obtain N first input data based on the N first echo signals; a fourth processing module, used to process the N first input data using a pre-trained image reconstruction model to obtain N first prediction results, wherein each of the N first prediction results includes a first two-dimensional image of the inspected target; and an image fusion module, used to perform image fusion based on the N first prediction results to reconstruct a fourth two-dimensional image of the inspected target.

[0031] Another aspect of an embodiment of the present disclosure provides a training device for an image reconstruction model, including: a fifth processing module, used to obtain second input data through a training target based on the signal characteristics of single-frequency millimeter wave acquisition; a model prediction module, used to input the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training target and second distance data of at least one pixel in the second two-dimensional image; a model training module, used to train the image reconstruction model based on the contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training target and third distance data of at least one pixel in the third two-dimensional image.

[0032] Another aspect of an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0033] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method described above.

[0034] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0035] One or more of the above-described embodiments have the following beneficial effects: by processing a first echo signal of a detected target acquired by a single-frequency millimeter wave, and using first input data obtained from the first echo signal as input to a pre-trained image reconstruction model, a first two-dimensional image of the detected target and first range-direction data of at least one pixel thereof can be obtained based on a first prediction result output by the image reconstruction model, thereby reconstructing a first three-dimensional image of the detected target. This can at least partially address the problem of current single-frequency systems lacking range resolution, and thus can utilize a single-frequency system to achieve, to a certain extent, the imaging performance of a broadband system, resulting in a clearer three-dimensional image. This can improve real-time detection, reduce hardware costs, and minimize bandwidth resource usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0037] Figure 1 Schematically illustrates the geometry of single-frequency millimeter wave imaging according to an embodiment of the present disclosure;

[0038] Figure 2 The flowchart of the training method of the image reconstruction model according to the embodiment of the present disclosure is schematically shown;

[0039] Figure 3 Schematically shows a flow chart of a method for training an image reconstruction model according to another embodiment of the present disclosure;

[0040] Figure 4 Schematically shows a flow chart of obtaining second input data according to an embodiment of the present disclosure;

[0041] Figure 5 Schematically shows a flow chart of obtaining second input data according to another embodiment of the present disclosure;

[0042] Figure 6 Schematically shows a flow chart of obtaining second input data according to another embodiment of the present disclosure;

[0043] Figure 7The flowchart of the millimeter wave image reconstruction method according to the embodiment of the present disclosure is schematically shown;

[0044] Figure 8 Schematically shows a flow chart of obtaining first input data according to an embodiment of the present disclosure;

[0045] Figure 9 Schematically shows an architecture diagram suitable for implementing a millimeter wave image reconstruction method according to an embodiment of the present disclosure;

[0046] Figure 10 Schematically shows a flow chart of a millimeter wave image reconstruction method according to another embodiment of the present disclosure;

[0047] Figure 11 Schematically shows an architecture diagram suitable for implementing a millimeter wave image reconstruction method according to another embodiment of the present disclosure;

[0048] Figure 12 The following schematically shows a structural block diagram of a training device for an image reconstruction model according to an embodiment of the present disclosure;

[0049] Figure 13 The structure of the millimeter wave image reconstruction device according to the embodiment of the present disclosure is schematically shown;

[0050] Figure 14 The structure of the millimeter wave image reconstruction device according to the embodiment of the present disclosure is schematically shown;

[0051] Figure 15 The block diagram schematically shows an electronic device suitable for implementing a millimeter wave image reconstruction method or an image reconstruction model training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] In order to facilitate understanding of the technical solutions of the embodiments of the present disclosure, some technical terms involved in the present disclosure are first introduced.

[0053] Single-frequency system: A general term for software and hardware that uses single-frequency millimeter waves to achieve imaging.

[0054] Wideband system: A general term for software and hardware that uses broadband millimeter waves to achieve imaging.

[0055] Detected target: The target detected in the single-frequency millimeter wave image reconstruction scenario.

[0056] Training target: In the image reconstruction model training scenario, the data source of the training samples.

[0057] First two-dimensional image: In the usage scenario, the two-dimensional image output by the image reconstruction model.

[0058] Second 2D image: In the training scenario, the 2D image output by the image reconstruction model.

[0059] The third two-dimensional image: a two-dimensional image obtained directly according to the training target, which serves as the label data in the training sample.

[0060] Fourth two-dimensional image: obtained by fusing N two-dimensional images output by the image reconstruction model during image reconstruction using N single-frequency millimeter waves.

[0061] First 3D image: in the usage scenario, a 3D image reconstructed according to the prediction result output by the image reconstruction model.

[0062] Second 3D image: In the training scenario, a 3D image reconstructed based on the prediction results output by the image reconstruction model.

[0063] Third 3D image: A 3D image obtained directly based on the training target, used as label data in the training sample.

[0064] Fourth three-dimensional image: obtained by fusing the fourth two-dimensional image and the range data.

[0065] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0066] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0067] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0068] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0069] In the prior art, single-frequency systems lack range resolution. Signals scattered at different distances interfere with each other across the scanning plane, ultimately causing severe defocusing and blurring in the reconstructed image, making it difficult to meet practical requirements. In the prior art, single-frequency systems have been unable to compensate for the shortcomings of broadband systems and achieve millimeter-wave holographic images that meet the requirements.

[0070] Embodiments of the present disclosure provide a millimeter wave image reconstruction method capable of processing a first echo signal of a target under inspection acquired by a single-frequency millimeter wave, and using first input data obtained from the first echo signal as input to a pre-trained image reconstruction model. Based on a first prediction result output by the image reconstruction model, a first two-dimensional image of the target under inspection and first range data of at least one pixel therein can be obtained, thereby reconstructing a first three-dimensional image of the target under inspection. This method can at least partially address the problem of current single-frequency systems lacking range resolution, and thus can utilize single-frequency systems to achieve, to a certain extent, the imaging performance indicators of broadband systems, reconstructing a clear three-dimensional image, thereby improving detection real-time performance, reducing hardware costs, and reducing bandwidth resource usage.

[0071] Furthermore, the penetration and ultimate resolution of single-frequency millimeter waves are determined by frequency, and the two change in opposite directions with frequency. Low-frequency millimeter waves have strong penetration but low resolution, while high-frequency millimeter waves have the opposite effect: as the frequency increases, the resolution improves but the penetration decreases.

[0072] Another embodiment of the present disclosure provides a millimeter wave image reconstruction method, which uses N single-frequency millimeter waves with different frequencies to collect N first echo signals of the target to be inspected, and obtains corresponding N first input data. The N first input data are further processed using a pre-trained image reconstruction model to obtain N first two-dimensional images of the target to be inspected. The N first two-dimensional images are fused to obtain a fourth two-dimensional image of the target to be inspected. This can at least partially resolve the contradiction between the penetration and resolution of single-frequency millimeter waves, and can achieve the effect of high penetration and high resolution by fusing two-dimensional images obtained from different single-frequency millimeter waves.

[0073] The embodiments of the present disclosure also provide a training method for an image reconstruction model, which can obtain second input data through a training target based on the signal characteristics of single-frequency millimeter wave acquisition, and use the second input data as input to train the image reconstruction model. During the training process, the rules of specific scenes (such as the continuity and regularity of the outer surface of the training target) are learned as prior knowledge, as well as the ability to learn a high-approximation simulation of the inverse process of the point diffusion process. It has the function of restoring non-focused images at different focus positions and predicting the distance data of pixel points.

[0074] Figure 1 The geometric diagram of single-frequency millimeter wave imaging according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 1 As shown, the single-frequency millimeter wave source 111 is located on the scanning surface 110. Since the single-frequency system in the related art does not have the resolution capability in the range direction, the coordinates of any scattering point on the target 120 are (x, y, z=0). Figure 1 In the xyz coordinate system shown, the x-axis represents the azimuth, the y-axis represents the altitude, and the z-axis represents the range. It should be noted that the scanning surface is not limited to a planar system, and the transmitting and receiving system is not limited to a single-transmitting and single-receiving system.

[0076] Figure 2 The flowchart of the method for training an image reconstruction model according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 2 As shown, the image reconstruction model training method of this embodiment includes operations S210 to S230.

[0078] In operation S210, second input data is obtained through a training target based on signal characteristics collected by a single-frequency millimeter wave.

[0079] For example, the signal characteristics may include characteristics of an echo signal acquired by a single-frequency millimeter wave, such as the absence of range data. The signal characteristics may also include characteristics of an unfocused image obtained based on the echo signal acquired by the single-frequency millimeter wave. Operation S210 simulates an echo signal acquired by a single-frequency millimeter wave, and an unfocused image obtained based on the echo signal, and uses these as input to an image reconstruction model.

[0080] Exemplarily, the number of training targets can be one or more. The second input data can be obtained based on the surface features of the training target, wherein the surface features include spatial surface features of the outer surface of the training target. Specifically, training targets of the same type have similar flow structures of the outer surface. Taking humans as an example of training targets, although different people have differences in body shape, they have the same body composition, such as head, torso and limbs. Therefore, the outer surfaces of different people have the same or similar continuity and regularity (i.e., spatial surface features).

[0081] Exemplarily, a set of training sample pairs is obtained based on a training target, including second input data and label data. The label data includes a third two-dimensional image of the training target and third range data for at least one pixel in the third two-dimensional image. The third two-dimensional image and third range data can be obtained via a broadband system or other methods, such as a depth-of-field camera, multi-angle illumination, or manually moving a focus plane.

[0082] In operation S220, the second input data is input into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training target and second range data of at least one pixel in the second two-dimensional image.

[0083] For example, an image reconstruction model can be constructed based on a machine learning algorithm (such as a neural network, Bayesian, random forest, or support vector machine). For example, a neural network-based image reconstruction model can be constructed, such as a U-Net structured convolutional neural network or a recurrent neural network (RNN) type deep learning model.

[0084] For example, referring to Figure 1 , the distance data may include the coordinate data of each pixel on the z-axis.

[0085] In operation S230 , an image reconstruction model is trained based on the contrast loss between the second prediction result and the label data, wherein the trained image reconstruction model can be used to reconstruct a single-frequency millimeter wave image.

[0086] Exemplarily, a loss function can be set to calculate the contrast loss between the second prediction result and the label data. The loss function may include a region-based loss function, such as obtaining edge indicators of the second two-dimensional image and the third two-dimensional image respectively through an edge detection algorithm, and calculating the contrast loss based on this. The loss function may also include functions such as the L2 norm or the mean square error. Among them, the contrast loss between the second prediction result and the two-dimensional image in the label data can be calculated separately, or the contrast loss between the second prediction result and the distance data in the label data can be calculated separately, or the losses of both the two-dimensional image and the distance data can be jointly calculated to obtain the loss.

[0087] For example, the image reconstruction model training completion can be determined based on whether the loss function value converges. For example, if the loss function value is less than or equal to a preset threshold (such as 5%, which is only an example), the image reconstruction model can be considered to meet the requirements.

[0088] For example, the training sample pairs can be divided into a training set, a validation set, and a test set, and the image reconstruction model can be trained based on PyTorch or other frameworks. The validation set is used for verification during the training process, and the test set is used for testing the model after training is completed.

[0089] According to the embodiments of the present disclosure, to address the range resolution issue in single-frequency system image reconstruction, a data-driven deep learning approach is employed to reconstruct images. Based on the signal characteristics of single-frequency millimeter wave acquisition, a second input data is obtained from a training target. This second input data is then used to train the image reconstruction model. During the training process, the model learns the patterns of specific scenes (such as the continuity and regularity of the training target's outer surface) as prior knowledge, as well as the ability to closely simulate the inverse of the point diffusion process. This allows the system to recover unfocused images at different focus positions and predict the range data of pixel points.

[0090] Figure 3 The flowchart of the method for training an image reconstruction model according to another embodiment of the present disclosure is schematically shown.

[0091] The training method of the image reconstruction model of this embodiment may include operations S210 to S230. Figure 3 As shown, the image reconstruction model training method of this embodiment may further include operations S310 to S320. Operation S230 may include operation S320.

[0092] In operation S310 , a second three-dimensional image of the training object is reconstructed based on the second two-dimensional image and the second range data.

[0093] In operation S320, an image reconstruction model is trained based on a contrast loss between the second 3D image and the third 3D image, wherein the second prediction result includes the second 3D image and the label data includes the third 3D image.

[0094] According to embodiments of the present disclosure, an image reconstruction model can be trained based on the contrast loss between 3D images. This means that the convergence condition of the loss function is related to the 3D image. This allows the trained model to achieve better 3D image reconstruction performance in scenarios involving reconstructing 3D images of inspected objects.

[0095] Figure 4 The flowchart of obtaining the second input data according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 4 As shown, in operation S210 , obtaining the second input data through the training target based on the signal characteristics of the single-frequency millimeter wave acquisition includes operations S410 to S430 .

[0097] In operation S410 , a second echo signal of a training target is collected using a broadband millimeter wave, wherein the second echo signal includes surface features.

[0098] Reference Figure 1 Broadband millimeter waves can not only obtain data in altitude and azimuth, but also in range. The second echo signal including surface features means that, for example, if the target object is a person, imaging is performed by detecting the human body surface (including passive imaging or active imaging). Accordingly, the range data of the human body surface in space is different, and the second echo signal collected at each scattering point is also different, thus reflecting the surface features.

[0099] In operation S420, amplitude data and phase data of a first frequency point are obtained according to the second echo signal, wherein the broadband millimeter wave includes the first frequency point.

[0100] For example, the first frequency point is a single frequency point of the broadband millimeter wave, and the first frequency point can be determined according to the parameters of the single-frequency millimeter wave and the usage scenario. Specifically, the first frequency point can be the same as the frequency of the single-frequency system in the usage scenario.

[0101] In operation S430, second input data is obtained according to the amplitude data and the phase data of the first frequency point.

[0102] According to embodiments of the present disclosure, the second echo signal can be used to obtain a third two-dimensional image of the training target and third range data for at least one pixel in the third two-dimensional image. A third three-dimensional image can also be reconstructed from the third two-dimensional image and the third range data. The broadband millimeter wave holographic imaging process can be implemented using existing methods and will not be further described here.

[0103] According to the embodiments of the present disclosure, one of the goals of training an image reconstruction model is to enable it to predict range data, thereby enabling a single-frequency system to achieve the same performance as a broadband system. Therefore, using a broadband system to collect a second echo signal and obtain second input data for training can yield a more robust image reconstruction model.

[0104] According to an embodiment of the present disclosure, the amplitude data and phase data of the first frequency point can be used as the second input data in operation S430. This is because the amplitude data and phase data of the first frequency point can be directly obtained from the second echo signal. Using these data as input to the image reconstruction model allows the image reconstruction model to learn more prior knowledge during training, thereby achieving better performance.

[0105] The following combination Figure 5 Another embodiment of operation S430 is further introduced.

[0106] Figure 5 The flowchart of obtaining second input data according to another embodiment of the present disclosure is schematically shown.

[0107] like Figure 5 As shown, obtaining the second input data according to the amplitude data and the phase data of the first frequency point in operation S430 includes operations S510 to S520.

[0108] In operation S510 , an unfocused image of each of M positions is obtained according to amplitude data and phase data of a first frequency point, where the M positions are distributed along a range direction and M is greater than or equal to 2.

[0109] For example, the unfocused image may be obtained based on Fourier transform or back-projection technology.

[0110] In operation S520, the unfocused image of each position is taken as second input data.

[0111] For example, multi-focus synthesis involves combining two or more images with different focal points to create a desired image. Similar to multi-focus synthesis, M positions are used as M focus positions, and an unfocused image is obtained at each focus position. Multiple unfocused images are then fed into the image reconstruction model, allowing it to synthesize multiple unfocused images during training to obtain prediction results.

[0112] Figure 6 The flowchart of obtaining second input data according to another embodiment of the present disclosure is schematically shown.

[0113] like Figure 6As shown, in operation S210 , obtaining the second input data through the training target based on the signal characteristics of the single-frequency millimeter wave acquisition includes operations S610 to S620 .

[0114] In operation S610 , a non-focused image of each of M positions is obtained according to a third two-dimensional image, where the M positions are distributed along a range direction, and M is greater than or equal to 2.

[0115] For example, the training target may be photographed by a camera to obtain the third two-dimensional image. The camera may include a surveillance camera or a terminal device with a photographing function, etc., which is not limited in this disclosure.

[0116] In operation S620, the unfocused image of each position is taken as second input data.

[0117] In some embodiments, diffusion can be performed directly based on the third two-dimensional image to obtain an unfocused image. Diffusion can be achieved by adding noise to the third two-dimensional image using related techniques (such as Gaussian blur) to obtain images with different blur standards, i.e., unfocused images at multiple locations.

[0118] In other embodiments, assuming the training target is a person, since the third two-dimensional image is a planar image of the person, the various regions of the person in the planar image can be mapped onto a spatial curved surface according to the body spatial template, thereby generating a three-dimensional image. This allows each pixel to have different distance data. Label data can be obtained from the mapped three-dimensional image. Furthermore, diffusion processing can be performed on the three-dimensional image to generate multiple unfocused images.

[0119] According to the embodiments of the present disclosure, the second input data is obtained through the third two-dimensional image, which can get rid of the limitation of the broadband system to a certain extent and expand the source of obtaining the second input data.

[0120] According to the embodiments of the present disclosure, such as in the use scenario of millimeter wave body security inspection, the flow structure of the human body surface can be learned through large-scale data learning. The network trained based on such data carries relevant information and can restore the unfocused image within a certain depth of field. At the same time, the longitudinal distance of each pixel point is obtained based on the learned flow structure of the human body surface and the focusing inversion method.

[0121] In addition, related technologies generally implement millimeter wave imaging based on analytical or iterative algorithms, which have many factors that affect imaging, such as taking approximate values or basing on some assumptions, and ignoring some factors that are considered non-essential. This will cause some useful information to be lost during the reconstruction process, and the computational complexity is high. The image reconstruction model training method of the embodiment of the present disclosure does not simulate a specific physical process, but learns the mapping relationship between input information and the final effect. By using neural networks and big data training samples, the factors that affect the final effect are comprehensively simulated as much as possible, thus avoiding the loss of useful information to a certain extent.

[0122] Figure 7 The flowchart of the millimeter wave image reconstruction method according to an embodiment of the present disclosure is schematically shown.

[0123] like Figure 7 As shown, the millimeter wave image reconstruction method of this embodiment includes operations S710 to S740.

[0124] In operation S710 , a first echo signal of a detected object is collected using a single-frequency millimeter wave.

[0125] Exemplarily, after a single-frequency millimeter wave is scattered by any scattering point of the detected target, a corresponding first echo signal can be collected.

[0126] In operation S720, first input data is obtained according to the first echo signal.

[0127] According to an embodiment of the present disclosure, amplitude data and phase data of the first echo signal may be used as the first input data.

[0128] Figure 8 The flowchart of obtaining first input data according to an embodiment of the present disclosure is schematically shown.

[0129] like Figure 8 As shown, obtaining the first input data according to the first echo signal in operation S720 includes operations S810 to S820.

[0130] In operation S810 , a non-focused image of each of M positions is obtained according to amplitude data and phase data of a first echo signal, where the M positions are distributed along a range direction, and M is greater than or equal to 2.

[0131] For example, the unfocused image may be obtained based on Fourier transform or back-projection technology.

[0132] In operation S820, a non-focused image of each position is taken as first input data.

[0133] According to an embodiment of the present disclosure, whether the amplitude data and phase data of the first echo signal are used as the first input data in operation S720 or the unfocused image of each position is used as the first input data may be consistent with the second input data in the training process.

[0134] In operation S730, the first input data is processed using a pre-trained image reconstruction model to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the inspected object and first range data of at least one pixel in the first two-dimensional image.

[0135] In operation S740, a first three-dimensional image of the detected object is reconstructed according to the first prediction result.

[0136] According to the embodiments of the present disclosure, the defocus problem is solved in the case of single-frequency millimeter wave imaging. The first echo signal of the detected target collected by the single-frequency millimeter wave can be processed, and the first input data obtained based on the first echo signal is used as the input of a pre-trained image reconstruction model. Based on the first prediction result output by the image reconstruction model, a first two-dimensional image of the detected target and first range data of at least one pixel therein can be obtained, thereby reconstructing a first three-dimensional image of the detected target. This can at least partially solve the problem that current single-frequency systems do not have range resolution capabilities, and thus can use single-frequency systems to achieve the imaging performance indicators of broadband systems to a certain extent, reconstructing a clear three-dimensional image, and can achieve the effects of improving detection real-time performance, reducing hardware costs, and reducing frequency band resource usage.

[0137] Figure 9 The following schematically shows an architecture diagram suitable for implementing a millimeter wave image reconstruction method according to an embodiment of the present disclosure.

[0138] like Figure 9 As shown in the figure, during the training process, a broadband system is used for data acquisition. A single frequency point in the broadband system is used to reconstruct a two-dimensional (2D) image, generating a 2D unfocused image as input. Furthermore, the data collected by the broadband system is used to reconstruct a three-dimensional (3D) image and serve as label data. This allows the image reconstruction model to be trained using a U-Net neural network, resulting in a trained model.

[0139] During the use process (i.e., inference process), a single-frequency system is used to collect data and perform 2D image reconstruction under single frequency. The 2D unfocused image is obtained and input into the trained model. Finally, a 3D image is reconstructed based on the model's prediction results.

[0140] Figure 10 The flowchart of the millimeter wave image reconstruction method according to another embodiment of the present disclosure is schematically shown. Figure 11The following schematically shows an architecture diagram suitable for implementing a millimeter wave image reconstruction method according to another embodiment of the present disclosure.

[0141] like Figure 10 As shown, the millimeter wave image reconstruction method of this embodiment includes operations S1010 to S1050.

[0142] In operation S1010, N first echo signals of the detected target are collected using N single-frequency millimeter waves, wherein the center frequencies of any two of the N single-frequency millimeter waves are different, the N single-frequency millimeter waves correspond one-to-one to the N first echo signals, and N is an integer greater than or equal to 2.

[0143] Reference Figure 11 , N single-frequency millimeter waves may include low frequency points, medium frequency points and high frequency points, for example, the center frequency of the low frequency point is 20GHZ, the center frequency of the medium frequency point is 50GHZ, and the center frequency of the high frequency point is 70GHZ (just an example).

[0144] Exemplarily, operation S710 is performed based on N single-frequency millimeter waves respectively, and N first echo signals are obtained.

[0145] In operation S1020, N first input data are obtained according to the N first echo signals.

[0146] Exemplarily, operation S720 is performed to process N first echo signals respectively to obtain N first input data. Specifically, various embodiments, technical problems solved, functions implemented, and technical effects achieved in operation S720 are also applicable to operation S1020 and are not described in detail here.

[0147] In operation S1030 , N first input data are processed using a pre-trained image reconstruction model to obtain N first prediction results, wherein each of the N first prediction results includes a first two-dimensional image of the detected object.

[0148] For example, in operation S1030, N pre-trained image reconstruction models can be used to process N first input data respectively, and output N first prediction results respectively. In other words, in this embodiment, each of the N single-frequency millimeter waves has a corresponding image reconstruction model, which can be referred to Figures 2 to 6 The embodiment pre-trains the image reconstruction model corresponding to each single-frequency millimeter wave, which will not be described in detail here.

[0149] Exemplarily, operation S730 may be performed to process N first input data respectively to obtain N first prediction results, which will not be described in detail here.

[0150] In operation S1040 , image fusion is performed according to the N first prediction results to reconstruct a fourth two-dimensional image of the inspected object.

[0151] Reference Figure 11 The two-dimensional images corresponding to the low-frequency point system, the medium-frequency point system, and the high-frequency point system are fused to obtain a fourth two-dimensional image. For example, wavelet technology can be used to perform image fusion.

[0152] In some embodiments, in order to resolve the contradiction between the penetration and resolution of a single frequency point, since each frequency point has its own advantages in detection ability and resolution for different materials, a combined imaging method of N frequencies is used to obtain a clear fourth two-dimensional image, which can meet the requirements for image clarity to a certain extent.

[0153] In other embodiments, if there is further demand for a three-dimensional image, operation S1050 may be performed to obtain it.

[0154] In operation S1050, a fourth three-dimensional image of the detected object is reconstructed based on the fourth two-dimensional image and the first range data in at least one first prediction result, wherein each first prediction result also includes the first range data of at least one pixel in the first two-dimensional image.

[0155] Reference Figure 11 The fourth two-dimensional image can also be fused with the depth map corresponding to the high-frequency point system to obtain a fourth three-dimensional image. Each depth map includes the first distance data of at least one pixel in the fourth two-dimensional image of the corresponding frequency point.

[0156] In some embodiments, the depth map corresponding to any frequency point can be selected and fused with the fourth two-dimensional image. In other embodiments, the depth maps corresponding to the various frequency points can be combined (e.g., averaged) and the result of the operation can be fused with the fourth two-dimensional image, which is not limited in this disclosure.

[0157] According to an embodiment of the present disclosure, N first echo signals of the inspected target are collected using N single-frequency millimeter waves of different frequencies, and corresponding N first input data are obtained. The N first input data are further processed using a pre-trained image reconstruction model to obtain N first two-dimensional images of the inspected target. The N first two-dimensional images are fused to obtain a fourth two-dimensional image of the inspected target. This can at least partially resolve the contradiction between the penetration and resolution of single-frequency millimeter waves, and can achieve high penetration and high resolution by fusing two-dimensional images obtained from different single-frequency millimeter waves, thereby broadening the application scenarios of single-frequency millimeter waves.

[0158] Based on the above-mentioned training method of the image reconstruction model, the present disclosure also provides a training device for the image reconstruction model. Figure 12 The device is described in detail.

[0159] Figure 12 The structural block diagram of the image reconstruction model training device 1200 according to an embodiment of the present disclosure is schematically shown.

[0160] like Figure 12 As shown, the image reconstruction model training device 1200 of this embodiment includes a fifth processing module 1210 , a model prediction module 1220 and a model training module 1230 .

[0161] The fifth processing module 1210 may perform operation S210 to obtain second input data through a training target based on the signal characteristics of the single-frequency millimeter wave acquisition.

[0162] The model prediction module 1220 can perform operation S220 to input the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training target and second distance data of at least one pixel in the second two-dimensional image.

[0163] The model training module 1230 can perform operation S230 to train an image reconstruction model based on the contrast loss between the second prediction result and the label data, wherein the trained image reconstruction model is used to reconstruct a single-frequency millimeter wave image, and the label data includes a third two-dimensional image of the training target and third distance data of at least one pixel in the third two-dimensional image.

[0164] According to an embodiment of the present disclosure, the training device 1200 may further include a 3D reconstruction module that may perform operation S310 to reconstruct a second 3D image of the training target based on the second 2D image and the second range data. The model training module 1230 may further perform operation S320 to train an image reconstruction model based on a contrast loss between the second 3D image and the third 3D image.

[0165] According to an embodiment of the present disclosure, the fifth processing module 1210 may further perform operations S410 to S430 to acquire a second echo signal of the training target using a broadband millimeter wave, wherein the second echo signal includes surface features. Amplitude data and phase data of a first frequency point are obtained based on the second echo signal, wherein the broadband millimeter wave includes the first frequency point. Second input data is obtained based on the amplitude data and phase data of the first frequency point. In some embodiments, the amplitude data and phase data of the first frequency point may be used as the second input data.

[0166] According to an embodiment of the present disclosure, the fifth processing module 1210 may further perform operations S510 to S520 to obtain an unfocused image at each of M positions based on the amplitude data and phase data of the first frequency point, where the M positions are distributed along the range direction and M is greater than or equal to 2. The unfocused image at each position is used as second input data.

[0167] According to an embodiment of the present disclosure, the fifth processing module 1210 may further perform operations S610 to S620 to obtain an unfocused image at each of M positions based on the third two-dimensional image, where the M positions are distributed along the range direction and M is greater than or equal to 2. The unfocused image at each position is used as second input data.

[0168] Based on the above-mentioned millimeter wave image reconstruction method, the present disclosure also provides a millimeter wave image reconstruction device 1300. Figure 13 The device is described in detail.

[0169] Figure 13 The figure schematically shows a structural block diagram of a millimeter wave image reconstruction device 1300 according to an embodiment of the present disclosure.

[0170] like Figure 13 As shown, the millimeter wave image reconstruction device 1300 of this embodiment includes a first acquisition module 1310 , a first processing module 1320 , a second processing module 1330 and a first reconstruction module 1340 .

[0171] The first acquisition module 1310 may perform operation S710 for acquiring a first echo signal of a detected target by using a single-frequency millimeter wave.

[0172] The first processing module 1320 may perform operation S720 to obtain first input data according to the first echo signal.

[0173] According to an embodiment of the present disclosure, amplitude data and phase data of the first echo signal may be used as the first input data.

[0174] According to an embodiment of the present disclosure, the first processing module 1320 may perform operations S810 to S820 to obtain an unfocused image at each of M positions based on the amplitude data and phase data of the first echo signal, where the M positions are distributed along the range direction and M is greater than or equal to 2. The unfocused image at each position is used as first input data.

[0175] The second processing module 1330 can execute operation S730 to process the first input data using a pre-trained image reconstruction model to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the inspected target and first distance data of at least one pixel in the first two-dimensional image.

[0176] The first reconstruction module 1340 may perform operation S740 to reconstruct a first three-dimensional image of the detected object according to the first prediction result.

[0177] Based on the millimeter wave image reconstruction method of another embodiment above, the present disclosure also provides a millimeter wave image reconstruction device 1400. Figure 14 The device is described in detail.

[0178] Figure 14 The figure schematically shows a structural block diagram of a millimeter wave image reconstruction device 1400 according to an embodiment of the present disclosure.

[0179] like Figure 14 As shown, the millimeter wave image reconstruction device 1400 of this embodiment includes a second acquisition module 1410 , a third processing module 1420 , a fourth processing module 1430 and an image fusion module 1440 .

[0180] The second acquisition module 1410 can perform operation S1010 to acquire N first echo signals of the detected target using N single-frequency millimeter waves, wherein the center frequencies of any two of the N single-frequency millimeter waves are different, the N single-frequency millimeter waves correspond one-to-one to the N first echo signals, and N is an integer greater than or equal to 2.

[0181] The third processing module 1420 may perform operation S1020 to obtain N first input data according to the N first echo signals.

[0182] The fourth processing module 1430 may perform operation S1030 to process N first input data using a pre-trained image reconstruction model to obtain N first prediction results, wherein each of the N first prediction results includes a first two-dimensional image of the detected object.

[0183] The image fusion module 1440 may perform operation S1040 to perform image fusion according to the N first prediction results to reconstruct a fourth two-dimensional image of the inspected object.

[0184] According to an embodiment of the present disclosure, the millimeter wave image reconstruction device 1400 may further include a second reconstruction module for reconstructing a fourth three-dimensional image of the inspected target by performing image fusion based on the fourth two-dimensional image and the first range data in at least one first prediction result.

[0185] It should be noted that the implementation methods, technical problems solved, functions implemented, and technical effects achieved of each module / unit / sub-unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions implemented, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0186] According to embodiments of the present disclosure, any multiple modules in the training device 1200, the millimeter-wave image reconstruction device 1300, or the millimeter-wave image reconstruction device 1400 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module.

[0187] According to an embodiment of the present disclosure, at least one module in the training device 1200, the millimeter wave image reconstruction device 1300, or the millimeter wave image reconstruction device 1400 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware through any other reasonable means of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one module in the training device 1200, the millimeter wave image reconstruction device 1300, or the millimeter wave image reconstruction device 1400 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0188] Figure 15 The block diagram schematically shows an electronic device suitable for implementing a millimeter wave image reconstruction method or an image reconstruction model training method according to an embodiment of the present disclosure.

[0189] like Figure 15 As shown, the electronic device 1500 according to an embodiment of the present disclosure includes a processor 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage portion 1508 into a random access memory (RAM) 1503. The processor 1501 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1501 may also include an onboard memory for caching purposes. The processor 1501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0190] Various programs and data required for the operation of the electronic device 1500 are stored in the RAM 1503. The processor 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. The processor 1501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1502 and / or the RAM 1503. It should be noted that the programs may also be stored in one or more memories other than the ROM 1502 and the RAM 1503. The processor 1501 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0191] According to an embodiment of the present disclosure, the electronic device 1500 may further include an input / output (I / O) interface 1505, which is also connected to the bus 1504. The electronic device 1500 may further include one or more of the following components connected to the I / O interface 1505: an input portion 1506 including a keyboard, a mouse, etc.; an output portion 1507 including a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, etc.; a storage portion 1508 including a hard disk, etc.; and a communication portion 1509 including a network interface card such as a LAN card or a modem. The communication portion 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the I / O interface 1505 as needed. A removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 1510 as needed, so that a computer program read therefrom can be installed into the storage portion 1508 as needed.

[0192] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments. Alternatively, the computer-readable storage medium may exist independently, without being incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the methods according to the embodiments of the present disclosure.

[0193] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1502 and / or RAM 1503 described above and / or one or more memories other than ROM 1502 and RAM 1503.

[0194] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiments of the present disclosure.

[0195] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 1501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0196] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal over a network medium, downloaded and installed via communication portion 1509, and / or installed from removable media 1511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0197] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1509 and / or installed from the removable medium 1511. When the computer program is executed by the processor 1501, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0198] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0201] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A millimeter wave image reconstruction method, comprising: Use single-frequency millimeter waves to collect the first echo signal of the target under inspection; obtaining first input data according to the first echo signal; Processing the first input data using a pre-trained image reconstruction model to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the inspected object and first range data of at least one pixel in the first two-dimensional image; reconstructing a first three-dimensional image of the detected object according to the first prediction result; The image reconstruction model is trained according to the following operations: Based on the signal characteristics of the single-frequency millimeter wave acquisition, second input data is obtained through a training target; Inputting the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training object and second range data of at least one pixel in the second two-dimensional image; training the image reconstruction model based on a contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training object and third range data of at least one pixel in the third two-dimensional image; Among them, also include: reconstructing a second three-dimensional image of the training object based on the second two-dimensional image and the second range data; The second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model according to the contrast loss between the second prediction result and the label data includes: The image reconstruction model is trained according to the contrast loss between the second three-dimensional image and the third three-dimensional image.

2. The method according to claim 1, wherein The obtaining of second input data by training a target based on the signal characteristics of the single-frequency millimeter wave acquisition includes: collecting a second echo signal of the training target using broadband millimeter waves; Obtaining amplitude data and phase data of a first frequency point according to the second echo signal, wherein the broadband millimeter wave includes the first frequency point; The second input data is obtained according to the amplitude data and phase data of the first frequency point.

3. The method according to claim 2, wherein: The obtaining of the second input data according to the amplitude data and phase data of the first frequency point includes: The amplitude data and phase data of the first frequency point are used as the second input data.

4. The method according to claim 2, wherein: The obtaining of the second input data according to the amplitude data and phase data of the first frequency point includes: Obtaining an unfocused image of each of M positions according to the amplitude data and the phase data of the first frequency point, wherein the M positions are distributed along the range direction, and M is greater than or equal to 2; The unfocused image of each position is used as the second input data.

5. The method according to claim 1, wherein The obtaining of second input data by training a target based on the signal characteristics of the single-frequency millimeter wave acquisition includes: Obtaining a non-focused image of each of M positions according to the third two-dimensional image, wherein the M positions are distributed along the range direction, and M is greater than or equal to 2; The unfocused image of each position is used as the second input data.

6. The method according to claim 1, wherein The obtaining first input data according to the first echo signal includes: The amplitude data and phase data of the first echo signal are used as the first input data.

7. The method according to claim 1, wherein The obtaining first input data according to the first echo signal includes: Obtaining an unfocused image of each of M positions according to the amplitude data and the phase data of the first echo signal, wherein the M positions are distributed along the range direction, and M is greater than or equal to 2; The unfocused image of each position is used as the first input data.

8. A millimeter wave image reconstruction method, comprising: Using N single-frequency millimeter waves to collect N first echo signals of the detected target, wherein the center frequencies of any two of the N single-frequency millimeter waves are different, the N single-frequency millimeter waves correspond one-to-one to the N first echo signals, and N is an integer greater than or equal to 2; Obtaining N first input data according to the N first echo signals; Processing N first input data using a pre-trained image reconstruction model to obtain N first prediction results, wherein each first prediction result of the N first prediction results includes a first two-dimensional image of the detected object and each first prediction result also includes first range data of at least one pixel in the first two-dimensional image; Performing image fusion according to the N first prediction results to reconstruct a fourth two-dimensional image of the inspected object; The method further comprises: Reconstructing a fourth three-dimensional image of the inspected object by performing image fusion based on the fourth two-dimensional image and first range data in at least one first prediction result; The image reconstruction model is trained according to the following operations: Based on the signal characteristics of the single-frequency millimeter wave acquisition, second input data is obtained through a training target; Inputting the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training object and second range data of at least one pixel in the second two-dimensional image; training the image reconstruction model based on a contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training object and third range data of at least one pixel in the third two-dimensional image; Among them, also include: reconstructing a second three-dimensional image of the training object based on the second two-dimensional image and the second range data; The second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model according to the contrast loss between the second prediction result and the label data includes: The image reconstruction model is trained according to the contrast loss between the second three-dimensional image and the third three-dimensional image.

9. A method for training an image reconstruction model, comprising: Based on the signal characteristics of single-frequency millimeter wave acquisition, the second input data is obtained through the training target; Inputting the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training object and second range data of at least one pixel in the second two-dimensional image; training the image reconstruction model based on a contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training object and third range data of at least one pixel in the third two-dimensional image; Among them, also include: reconstructing a second three-dimensional image of the training object based on the second two-dimensional image and the second range data; The second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model according to the contrast loss between the second prediction result and the label data includes: The image reconstruction model is trained according to the contrast loss between the second three-dimensional image and the third three-dimensional image.

10. A millimeter wave image reconstruction device, comprising: A first acquisition module is used to acquire a first echo signal of the detected target using a single-frequency millimeter wave; a first processing module, configured to obtain first input data according to the first echo signal; a second processing module, configured to process the first input data using a pre-trained image reconstruction model to obtain a first prediction result output by the image reconstruction model, wherein the first prediction result includes a first two-dimensional image of the inspected object and first range data of at least one pixel in the first two-dimensional image; a first reconstruction module, configured to reconstruct a first three-dimensional image of the detected object according to the first prediction result; The image reconstruction model is trained according to the following operations: Based on the signal characteristics of the single-frequency millimeter wave acquisition, second input data is obtained through a training target; Inputting the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training object and second range data of at least one pixel in the second two-dimensional image; training the image reconstruction model based on a contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training object and third range data of at least one pixel in the third two-dimensional image; Among them, also include: reconstructing a second three-dimensional image of the training object based on the second two-dimensional image and the second range data; The second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model according to the contrast loss between the second prediction result and the label data includes: The image reconstruction model is trained according to the contrast loss between the second three-dimensional image and the third three-dimensional image.

11. A millimeter wave image reconstruction device, comprising: a second acquisition module, configured to acquire N first echo signals of the detected target using N single-frequency millimeter waves, wherein the center frequencies of any two of the N single-frequency millimeter waves are different, the N single-frequency millimeter waves correspond one-to-one to the N first echo signals, and N is an integer greater than or equal to 2; a third processing module, configured to obtain N first input data according to the N first echo signals; a fourth processing module, configured to process N of the first input data using a pre-trained image reconstruction model to obtain N first prediction results, wherein each of the N first prediction results includes a first two-dimensional image of the inspected object and further includes first range data of at least one pixel in the first two-dimensional image; an image fusion module, configured to perform image fusion according to the N first prediction results to reconstruct a fourth two-dimensional image of the inspected object; a second reconstruction module, configured to reconstruct a fourth three-dimensional image of the inspected object by performing image fusion based on the fourth two-dimensional image and first distance data in at least one first prediction result; The image reconstruction model is trained according to the following operations: Based on the signal characteristics of the single-frequency millimeter wave acquisition, second input data is obtained through a training target; Inputting the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training object and second range data of at least one pixel in the second two-dimensional image; training the image reconstruction model based on a contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training object and third range data of at least one pixel in the third two-dimensional image; Among them, also include: reconstructing a second three-dimensional image of the training object based on the second two-dimensional image and the second range data; The second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model according to the contrast loss between the second prediction result and the label data includes: The image reconstruction model is trained according to the contrast loss between the second three-dimensional image and the third three-dimensional image.

12. A training device for an image reconstruction model, comprising: A fifth processing module, configured to obtain second input data through a training target based on signal characteristics of the single-frequency millimeter wave acquisition; a model prediction module, configured to input the second input data into the image reconstruction model to obtain a second prediction result output by the image reconstruction model, wherein the second prediction result includes a second two-dimensional image of the training object and second range data of at least one pixel in the second two-dimensional image; a model training module, configured to train the image reconstruction model based on a contrast loss between the second prediction result and label data, wherein the label data includes a third two-dimensional image of the training target and third range data of at least one pixel in the third two-dimensional image; Wherein, it also includes a three-dimensional reconstruction module, which is used to reconstruct a second three-dimensional image of the training target according to the second two-dimensional image and the second range data; The second prediction result includes the second three-dimensional image, the label data includes a third three-dimensional image, and the training of the image reconstruction model according to the contrast loss between the second prediction result and the label data includes: The image reconstruction model is trained according to the contrast loss between the second three-dimensional image and the third three-dimensional image.

13. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 9.

14. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 9.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images