A method, device, equipment and medium for reconstructing non-viewing area images

Abstract

The present application discloses a non-line-of-sight image reconstruction method, device, equipment and medium, and relates to the field of non-line-of-sight imaging technology. The method includes: obtaining a spot image of an image to be reconstructed in a non-line-of-sight scene; obtaining an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; using an encoder in the backbone network of the image reconstruction model to extract features and estimate distribution of the spot image; using a high-dimensional manifold establishment module to perform information interactive fusion on underlying features and distribution estimation information; using a projection selection module to project a high-dimensional manifold in a high-dimensional manifold information space onto a low-dimensional distribution and determine the target manifold distribution closest to the spot image; using a decoder in the backbone network to sample and decode the target manifold distribution to obtain a reconstructed image result corresponding to the spot image. The solution of the present invention can solve the problem that the current passive non-line-of-sight imaging based on deep learning requires a large amount of prior data and annotation costs.

Description

Technical Field

[0001] The present invention relates to the field of non-line-of-sight imaging technology, and in particular to a non-line-of-sight image reconstruction method, device, equipment and medium. Background Art

[0002] In the past few years, non-line-of-sight imaging (NLOS) technology has made significant progress in many fields, providing important technical support for solving perception problems in various complex environments. At present, non-line-of-sight imaging methods mainly include active and passive imaging methods, which have achieved remarkable results to varying degrees. Among them, active non-line-of-sight imaging is mainly the time-of-flight method, that is, using single photons or lidar to image non-line-of-sight scenes. LiDAR imaging technology uses a laser to emit a laser beam and receive the light signal reflected back by the target, and obtains the position and shape information of the target by analyzing the time delay and intensity of the light signal. It has the characteristics of high precision and high resolution, but the high cost and complex technical conditions make it difficult to apply in complex environments and engineering.

[0003] Passive imaging is more suitable in some application scenarios because of its cheap equipment, simple optical path and easy operation. Passive imaging is mainly based on shadow imaging and deep learning-based imaging technology. However, at present, the cost of data acquisition and annotation is high, and factors such as changes in lighting conditions and background noise interference will affect the quality and accuracy of imaging.

[0004] Therefore, how to provide a solution to the problem in the prior art that passive imaging methods have high data acquisition and annotation costs, and factors such as changes in lighting conditions and background noise interference can affect the quality and accuracy of imaging is a problem that technical personnel in this field currently need to solve. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a non-line-of-sight image reconstruction method, device, equipment and medium, which can solve the problem that the current deep learning-based passive non-line-of-sight imaging method requires a large amount of prior data and annotation costs, and improve the accuracy of restoration tasks for simple targets. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a non-viewing area image reconstruction method, comprising:

[0007] Obtaining a spot image of the image to be reconstructed in a non-viewing scene;

[0008] Acquire an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module, and a projection selection module;

[0009] Inputting the spot image into the image reconstruction model, and using the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information;

[0010] Using the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space;

[0011] The projection selection module is used to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution, and a target projection closest to the spot image is determined from the generated projections according to a preset judgment condition, so as to obtain a target manifold distribution corresponding to the target projection;

[0012] The decoder in the backbone network is used to perform corresponding sampling decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

[0013] Optionally, the step of obtaining a spot image of the image to be reconstructed in a non-viewing scene includes:

[0014] The image to be reconstructed is periodically projected at a preset image projection frequency through an imaging display screen placed behind the shielding wall, so as to display the information to be reconstructed on the imaging wall;

[0015] The information to be reconstructed on the imaging wall is imaged and captured by a capture camera at a preset capture frequency to obtain a spot image in a non-viewing scene that is consistent with the image to be reconstructed; wherein the preset image projection frequency is consistent with the preset capture frequency.

[0016] Optionally, the using the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information includes:

[0017] Using the encoder in the backbone network to perform feature extraction on the spot image to obtain underlying features of the spot image;

[0018] The encoder is used to perform distribution estimation on the underlying features to obtain distribution estimation information of the spot image.

[0019] Optionally, the using the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space includes:

[0020] Using the high-dimensional manifold establishment module, based on a first loss function and a second loss function, information interaction and fusion are performed on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space;

[0021] The first loss function is used to redefine the information features of the spot image on a high-dimensional manifold, and the second loss function is used to fuse the information features with the spot image.

[0022] Optionally, the using the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space includes:

[0023] Performing sampling estimation on the spot image based on the underlying features and the distribution estimation information, and estimating and outputting the manifold obtained by the sampling estimation to obtain a high-dimensional manifold estimation;

[0024] A target operator is estimated and trained based on the high-dimensional manifold, and an operator constraint condition of the target operator is determined, so as to establish a high-dimensional manifold information space based on the target operator and the operator constraint condition.

[0025] Optionally, the projecting and generating a high-dimensional manifold in the high-dimensional manifold information space to a low-dimensional distribution by using the projection selection module, and determining a target projection closest to the spot image from the generated projections according to a preset judgment condition, includes:

[0026] Using the projection selection module, based on the third loss function and the fourth loss function, the high-dimensional manifold in the high-dimensional manifold information space is projected to generate a low-dimensional distribution, and according to a preset judgment condition, a target projection closest to the spot image is determined from the generated projections;

[0027] The third loss function and the fourth loss function are used to select the generated projection according to the preset judgment condition to obtain the target projection, and estimate and establish the target manifold distribution corresponding to the target projection.

[0028] Optionally, the non-viewing area image reconstruction method further includes:

[0029] A target network structure is determined from any network structure having an encoding-decoding structure, and the target network structure is determined as the backbone network.

[0030] In a second aspect, the present application discloses a non-viewing area image reconstruction device, comprising:

[0031] An image acquisition module is used to acquire a spot image of an image to be reconstructed in a non-viewing scene;

[0032] A model acquisition module, used to acquire an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module and a projection selection module;

[0033] A first processing module, used for inputting the spot image into the image reconstruction model, and performing feature extraction and distribution estimation on the spot image using the encoder in the backbone network to obtain corresponding underlying features and distribution estimation information;

[0034] A second processing module is used to use the high-dimensional manifold establishment module to perform information interaction and fusion on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space;

[0035] A third processing module is used to use the projection selection module to project the high-dimensional manifold in the high-dimensional manifold information space to a low-dimensional distribution, and determine the target projection closest to the spot image from the generated projections according to a preset judgment condition, so as to obtain a target manifold distribution corresponding to the target projection;

[0036] The image reconstruction module is used to use the decoder in the backbone network to perform corresponding sampling decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

[0037] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the non-field of view image reconstruction method as described above.

[0038] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the non-field of view image reconstruction method as described above.

[0039] The present application provides a non-field of view image reconstruction method, comprising: obtaining a spot image of an image to be reconstructed in a non-field of view scene; obtaining an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model comprises a backbone network, a high-dimensional manifold establishment module and a projection selection module; inputting the spot image into the image reconstruction model, and using an encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information; using the high-dimensional manifold establishment module to perform information interactive fusion on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space; using the projection selection module to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution to generate, and determining the target projection closest to the spot image from the generated projection according to a preset judgment condition to obtain a target manifold distribution corresponding to the target projection; using a decoder in the backbone network to perform corresponding sampling and decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

[0040] The beneficial technical effects of the present application are: taking high-dimensional manifolds as the starting point, an innovative image reconstruction model based on high-dimensional manifolds and projection selection methods is constructed, providing a new framework for existing deep learning-based imaging technology. The same reconstruction results can be obtained through the image reconstruction model with less data support. Correspondingly, the image reconstruction model can also achieve universal neural network training under small data sets. Through the technical solution of the present application, the problem that the current passive non-field of view imaging method based on deep learning requires a large amount of prior data and annotation costs is solved while ensuring the correct restoration rate, and the accuracy of the restoration task of simple targets is improved. In addition, the image reconstruction process does not rely on the structure of the main network, so there is no need to spend additional research and development costs in miniaturization transplantation.

[0041] In addition, the present application provides a non-viewing area image reconstruction device, equipment and storage medium, which correspond to the above-mentioned non-viewing area image reconstruction method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A flowchart of a non-viewing area image reconstruction method disclosed in this application;

[0044] Figure 2 A schematic diagram of collecting a spot image for a non-viewing-area imaging scenario disclosed in the present application;

[0045] Figure 3 A schematic diagram of a high-dimensional manifold building module disclosed in this application;

[0046] Figure 4 A schematic diagram of information interaction and fusion disclosed in this application;

[0047] Figure 5 A schematic diagram of a projection selection module disclosed in the present application;

[0048] Figure 6 A schematic diagram of an original image, a collected light spot and a reconstructed image disclosed in this application;

[0049] Figure 7 This is a schematic diagram of the structure of a non-viewing area image reconstruction device disclosed in this application;

[0050] Figure 8 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] At present, although artificial intelligence has made great breakthroughs in various fields using algorithm models, non-line-of-sight scenes have problems such as too many unknown optical parameters, serious information loss, and extremely large optical transmission matrix parameters. Currently available non-line-of-sight imaging methods based on deep learning require a large amount of prior data as a training support for the model, which poses a great challenge to the cost of data acquisition and annotation. At the same time, existing deep learning relies on the main network framework, and its application in engineering often requires miniaturization and redesign, which will have a great impact on the reconstruction accuracy.

[0053] To this end, the present application provides a non-line-of-sight image reconstruction solution that can solve the problem that the current deep learning-based passive non-line-of-sight imaging method requires a large amount of prior data and annotation costs, and improve the accuracy of simple target restoration tasks.

[0054] The embodiment of the present invention discloses a non-viewing area image reconstruction method, see Figure 1 As shown, the method includes:

[0055] Step S11: obtaining a spot image of the image to be reconstructed in a non-viewing scene.

[0056] like Figure 2 The figure shows a non-field of view image acquisition scenario provided as an example in an embodiment of the present application. Arrange a shielding wall and an imaging wall made of any material in a darkroom environment, place an imaging display screen behind the shielding wall, and regularly project the image to be reconstructed on the screen; place a capture camera on the other side of the shielding wall, and use the capture camera to capture and image the imaging wall to collect the spot image in the non-field of view scene consistent with the image to be reconstructed. It should be pointed out that the acquisition frequency of the acquisition camera should be consistent with the image projection frequency of the imaging display screen.

[0057] Specifically, the step of obtaining a spot image of an image to be reconstructed in a non-viewing scene includes the following steps:

[0058] The image to be reconstructed is periodically projected at a preset image projection frequency through an imaging display screen placed behind the shielding wall, so as to display the information to be reconstructed on the imaging wall;

[0059] The information to be reconstructed on the imaging wall is imaged and captured by a capture camera at a preset capture frequency to obtain a spot image in a non-viewing scene that is consistent with the image to be reconstructed; wherein the preset image projection frequency is consistent with the preset capture frequency.

[0060] Step S12: obtaining an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module and a projection selection module.

[0061] In the embodiments of the present application, a universal image reconstruction model based on high-dimensional manifolds and projection selection method is innovatively constructed with high-dimensional manifolds as the starting point, providing a new framework for existing deep learning-based imaging technology.

[0062] Among them, the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module and a projection selection module. It is worth noting that the backbone network of the image reconstruction model in the embodiment of the present application can use a network structure of any encoding-decoding structure. Specifically, the target network structure is determined from any network structure with an encoding-decoding structure, and the target network structure is determined as the backbone network. The target network structure includes but is not limited to: a basic U-Net structure, a U-Net structure with cross-layer connections, a network structure using an attention mechanism, etc.

[0063] Step S13: inputting the spot image into the image reconstruction model, and using the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information.

[0064] In an embodiment of the present application, an image reconstruction model is used to reconstruct the collected spot image. First, the encoder in the backbone network is used to extract features and estimate the distribution of the spot image. Specifically, the encoder in the backbone network is used to extract features from the spot image to obtain the underlying features of the spot image; the encoder is used to estimate the distribution of the underlying features to obtain the distribution estimation information of the spot image. That is, the encoder embedded in the backbone network is used to first extract features from the prior spot image to obtain the underlying features; and then the encoder is used to further process the extracted underlying features to obtain the distribution estimation information of the spot image.

[0065] Step S14: using the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space.

[0066] In an embodiment of the present application, a high-dimensional manifold establishment module is used to perform information interaction and information fusion on the underlying features and distribution estimation information to obtain the high-dimensional manifold information distribution and information correspondence after information interaction and fusion, and further establish a high-dimensional manifold information space based on the high-dimensional manifold information distribution and information correspondence.

[0067] It should be noted that when performing information interaction and fusion, data processing is performed based on the first loss function and the second loss function. Specifically, the high-dimensional manifold establishment module is used to perform information interaction and fusion on the underlying features and the distribution estimation information based on the first loss function and the second loss function to establish a high-dimensional manifold information space; wherein the first loss function is used to redefine the information features of the spot image on the high-dimensional manifold, and the second loss function is used to perform information fusion on the information features and the spot image.

[0068] In a specific implementation, the first loss function may be a loss function rec (Loss of Particular Solution Reconstruction Optimization), which has the following specific form: ; The second loss function can be the loss function cl, which is the manifold information integration and interaction optimization loss, where cl is the manifold representation, and its specific form is: .

[0069] Step S15: Use the projection selection module to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution, and determine the target projection closest to the spot image from the generated projections according to preset judgment conditions to obtain a target manifold distribution corresponding to the target projection.

[0070] In an embodiment of the present application, a high-dimensional manifold is projected onto a low-dimensional distribution through a projection selection module, and a popular distribution closest to the spot image of the image to be reconstructed is selected. Specifically, the projection selection module is used to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution based on a third loss function and a fourth loss function, and the target projection closest to the spot image is determined from the generated projections according to a preset judgment condition; wherein the third loss function and the fourth loss function are used to select the generated projections according to the preset judgment condition to obtain the target projection, and estimate and establish the target manifold distribution corresponding to the target projection.

[0071] In a specific implementation, the third loss function may be a loss function propX (Loss of projection to X, projection optimization loss), and its specific form is: ; The fourth loss function can be the loss function tight (Loss of tighten manifold, manifold tightening constraint optimization loss), and its specific form is: .

[0072] Step S16: using the decoder in the backbone network to perform corresponding sampling decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

[0073] After the established high-dimensional information distribution and information correspondence are input into the projection selection module and the projection of the high-dimensional information distribution under the low-dimensional information is obtained, the decoder of the backbone network performs corresponding sampling decoding on the obtained target manifold distribution to obtain the reconstructed image result of the corresponding spot image. In this way, the backbone network that implements the image reconstruction model can reconstruct the image under the estimated distribution by feature fusion and feature mapping of the input features.

[0074] The present application provides a non-field of view image reconstruction method, comprising: obtaining a spot image of an image to be reconstructed in a non-field of view scene; obtaining an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model comprises a backbone network, a high-dimensional manifold establishment module and a projection selection module; inputting the spot image into the image reconstruction model, and using an encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information; using the high-dimensional manifold establishment module to perform information interactive fusion on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space; using the projection selection module to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution to generate, and determining the target projection closest to the spot image from the generated projection according to a preset judgment condition to obtain a target manifold distribution corresponding to the target projection; using a decoder in the backbone network to perform corresponding sampling and decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

[0075] The beneficial technical effects of the present application are: taking high-dimensional manifolds as the starting point, an innovative image reconstruction model based on high-dimensional manifolds and projection selection methods is constructed, providing a new framework for existing deep learning-based imaging technology. The same reconstruction results can be obtained through the image reconstruction model with less data support. Correspondingly, the image reconstruction model can also achieve universal neural network training under small data sets. Through the technical solution of the present application, the problem that the current passive non-field of view imaging method based on deep learning requires a large amount of prior data and annotation costs is solved while ensuring the correct restoration rate, and the accuracy of the restoration task of simple targets is improved. In addition, the image reconstruction process does not rely on the structure of the main network, so there is no need to spend additional research and development costs in miniaturization transplantation.

[0076] Furthermore, this embodiment describes the main steps of the high-dimensional manifold establishment module. The method of using the underlying features and the distribution estimation information to perform information interaction and fusion to establish a high-dimensional manifold information space specifically includes the following steps:

[0077] Performing sampling estimation on the spot image based on the underlying features and the distribution estimation information, and estimating and outputting the manifold obtained by the sampling estimation to obtain a high-dimensional manifold estimation;

[0078] A target operator is estimated and trained based on the high-dimensional manifold, and an operator constraint condition of the target operator is determined, so as to establish a high-dimensional manifold information space based on the target operator and the operator constraint condition.

[0079] like Figure 3 FIG. 1 is a schematic diagram of a high-dimensional manifold establishment module provided by this embodiment. It may include two main steps. The first is a sampling process: sampling and estimating the acquired spot image, analyzing its characteristics and distribution, and estimating the distribution of the manifold domain where the spot image is located. and The second main step is the operator The training constraint process can be roughly divided into four parts, including:

[0080] The first part mainly describes that in order to achieve the goal of approximating the target projection surface, the optimization goal of HDPS (Highdimensional manifolds Projection selection) is to train an operator , which can produce an output as close to the target as possible given the input data. Assume a specific case, when the input data does not suffer any distortion, that is, When , it is easy to find the most suitable operator .when When you can export , the goal is perfectly achieved, and the constructed high-dimensional manifold space refers to and Therefore, given the distance metric , the drift measure between the output obtained from the input and the ideal output can be defined as: .in, For operator The parameters to be trained in ; In order to obtain the most perfect solution for the particular solution, the goal requires the drift metric to take the minimum value .

[0081] Part 2 mainly describes that in complex non-line-of-sight scenes, the data images captured by the camera are often subject to significant distortion, attenuation, and interference caused by scattered noise. Therefore, in the non-line-of-sight propagation environment, and The distribution of is often significantly different. In an environment where data collection is difficult and computing resources are limited, a single constraint is used to establish and Therefore, the goal is to train the operator in this way. , so that Developed into , and with In this step, if Figure 4 As shown, the information integration is successfully completed, which means constructing a high-dimensional manifold. Therefore, the ideal situation involves minimizing and The distance between them, while complying with the specified constraints, can obtain the second optimization constraint: .

[0082] The third and fourth parts mainly describe: In order to ensure that the output target can be closer to , so we hope that the operator Able to give Some with Similar characteristics. When the input looks like the target itself (i.e., the image is not degraded), the operator The change obtained is minimal. Therefore, assuming that the operator Satisfying the first two constraints, we can assume that Close enough to Therefore, it is hoped that in the operator Under the influence of also exhibits similar characteristics; therefore, after establishing the high-dimensional manifold, As the output from the high-dimensional manifold to the low-dimensional plane, the goal requires the selection of the projection output to obtain a better output result, such as Figure 5 As shown, the third optimization constraint is obtained: .

[0083] However, the third optimization constraint poses a problem in that two , both play a role, which means that two different gradient paths can be found. Therefore, it must be divided into two parts.

[0084] In order to better output the manifold Aligned to the target manifold, initially only optimizing Figure 3 The upper path in the training set is considered as frozen. Considering the concept of embodying “close enough” to the target manifold, the target is obtained by training the upper path. Reflects the target manifold Similar features, thus minimizing and Therefore, the optimization objective of Part 3 is as follows: .

[0085] Although the optimization goal of Section 3 encourages output Closer to the target, but this is not enough. We need to reduce the neighborhood of the target manifold to as small an extent as possible, allowing only one "really close enough" To achieve this goal, maximize and The distance between Figure 3 When the upper path in is frozen, Considered as this round operator In order to have a smaller target manifold neighborhood, try to as a defective output, which will encourage Give a closer one in the next round This only optimizes the lower path, and the optimization goal of Part 4 is as follows: The overall framework of HDPS is as follows Figure 6 As shown, the specific implementation in the above steps is combined and will not be described again here.

[0086] Correspondingly, the embodiment of the present application also discloses a non-viewing area image reconstruction device, see Figure 7 As shown, the device comprises:

[0087] An image acquisition module 11 is used to acquire a spot image of an image to be reconstructed in a non-viewing scene;

[0088] A model acquisition module 12 is used to acquire an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module and a projection selection module;

[0089] The first processing module 13 is used to input the spot image into the image reconstruction model, and use the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information;

[0090] A second processing module 14 is used to perform information interaction and fusion on the underlying features and the distribution estimation information by using the high-dimensional manifold establishment module to establish a high-dimensional manifold information space;

[0091] The third processing module 15 is used to use the projection selection module to project the high-dimensional manifold in the high-dimensional manifold information space to a low-dimensional distribution, and determine the target projection closest to the spot image from the generated projections according to a preset judgment condition, so as to obtain a target manifold distribution corresponding to the target projection;

[0092] The image reconstruction module 16 is used to perform corresponding sampling decoding on the target manifold distribution using the decoder in the backbone network to obtain a reconstructed image result corresponding to the spot image.

[0093] Among them, for more specific working processes of the above-mentioned modules, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0094] It can be seen that the above scheme of this embodiment includes: obtaining a spot image of the image to be reconstructed in a non-viewing scene; obtaining an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module and a projection selection module; inputting the spot image into the image reconstruction model, and using the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information; using the high-dimensional manifold establishment module to perform information interactive fusion on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space; using the projection selection module to project the high-dimensional manifold in the high-dimensional manifold information space to a low-dimensional distribution to generate, and determining the target projection closest to the spot image from the generated projection according to a preset judgment condition to obtain a target manifold distribution corresponding to the target projection; using the decoder in the backbone network to perform corresponding sampling and decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

[0095] The beneficial technical effects of the present application are: taking high-dimensional manifolds as the starting point, an innovative image reconstruction model based on high-dimensional manifolds and projection selection methods is constructed, providing a new framework for existing deep learning-based imaging technology. The same reconstruction results can be obtained through the image reconstruction model with less data support. Correspondingly, the image reconstruction model can also achieve universal neural network training under small data sets. Through the technical solution of the present application, the problem that the current passive non-field of view imaging method based on deep learning requires a large amount of prior data and annotation costs is solved while ensuring the correct restoration rate, and the accuracy of the restoration task of simple targets is improved. In addition, the image reconstruction process does not rely on the structure of the main network, so there is no need to spend additional research and development costs in miniaturization transplantation.

[0096] Furthermore, the present application also discloses an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0097] Figure 8 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the non-viewing area image reconstruction method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be a computer.

[0098] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0099] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The data 223 may include various data. The storage method may be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the non-viewing area image reconstruction method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0101] Furthermore, the embodiment of the present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium mentioned here includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. Wherein, when the computer program is executed by the processor, the aforementioned non-field of view image reconstruction method is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0102] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0103] The steps of the non-viewing area image reconstruction method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0104] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0105] The above is a detailed introduction to the non-field of view image reconstruction method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A non-viewing area image reconstruction method, characterized in that: include: Obtaining a spot image of the image to be reconstructed in a non-viewing scene; Acquire an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module, and a projection selection module; Inputting the spot image into the image reconstruction model, and using the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information; Using the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space; The projection selection module is used to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution, and a target projection closest to the spot image is determined from the generated projections according to a preset judgment condition, so as to obtain a target manifold distribution corresponding to the target projection; The decoder in the backbone network is used to perform corresponding sampling decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

2. The non-viewing area image reconstruction method according to claim 1, characterized in that: The step of obtaining a spot image of an image to be reconstructed in a non-viewing scene includes: The image to be reconstructed is periodically projected at a preset image projection frequency through an imaging display screen placed behind the shielding wall, so as to display the information to be reconstructed on the imaging wall; The information to be reconstructed on the imaging wall is imaged and captured by a capture camera at a preset capture frequency to obtain a spot image in a non-viewing scene that is consistent with the image to be reconstructed; wherein the preset image projection frequency is consistent with the preset capture frequency.

3. The non-viewing area image reconstruction method according to claim 1, characterized in that: The using the encoder in the backbone network to perform feature extraction and distribution estimation on the spot image to obtain corresponding underlying features and distribution estimation information includes: Using the encoder in the backbone network to perform feature extraction on the spot image to obtain underlying features of the spot image; The encoder is used to perform distribution estimation on the underlying features to obtain distribution estimation information of the spot image.

4. The non-viewing area image reconstruction method according to claim 1, characterized in that: The utilizing the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space includes: Using the high-dimensional manifold establishment module, based on a first loss function and a second loss function, information interaction and fusion are performed on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space; The first loss function is used to redefine the information features of the spot image on a high-dimensional manifold, and the second loss function is used to fuse the information features with the spot image.

5. The non-viewing area image reconstruction method according to claim 1, characterized in that: The utilizing the high-dimensional manifold establishment module to interactively fuse the underlying features and the distribution estimation information to establish a high-dimensional manifold information space includes: Performing sampling estimation on the spot image based on the underlying features and the distribution estimation information, and estimating and outputting the manifold obtained by the sampling estimation to obtain a high-dimensional manifold estimation; A target operator is estimated and trained based on the high-dimensional manifold, and an operator constraint condition of the target operator is determined, so as to establish a high-dimensional manifold information space based on the target operator and the operator constraint condition.

6. The non-viewing area image reconstruction method according to claim 1, characterized in that: The projecting and generating of the high-dimensional manifold in the high-dimensional manifold information space to the low-dimensional distribution by using the projection selection module, and determining the target projection closest to the spot image from the generated projections according to a preset judgment condition, comprises: Using the projection selection module, based on the third loss function and the fourth loss function, the high-dimensional manifold in the high-dimensional manifold information space is projected to generate a low-dimensional distribution, and according to a preset judgment condition, a target projection closest to the spot image is determined from the generated projections; The third loss function and the fourth loss function are used to select the generated projection according to the preset judgment condition to obtain the target projection, and estimate and establish the target manifold distribution corresponding to the target projection.

7. The non-viewing area image reconstruction method according to any one of claims 1 to 6, characterized in that: Also includes: A target network structure is determined from any network structure having an encoding-decoding structure, and the target network structure is determined as the backbone network.

8. A non-viewing area image reconstruction device, characterized in that: include: An image acquisition module is used to acquire a spot image of an image to be reconstructed in a non-viewing scene; A model acquisition module, used to acquire an image reconstruction model pre-constructed based on a high-dimensional manifold and a projection selection method; the image reconstruction model includes a backbone network, a high-dimensional manifold establishment module and a projection selection module; A first processing module, used for inputting the spot image into the image reconstruction model, and performing feature extraction and distribution estimation on the spot image using the encoder in the backbone network to obtain corresponding underlying features and distribution estimation information; A second processing module is used to use the high-dimensional manifold establishment module to perform information interaction and fusion on the underlying features and the distribution estimation information to establish a high-dimensional manifold information space; a third processing module, configured to project the high-dimensional manifold in the high-dimensional manifold information space onto a low-dimensional distribution using the projection selection module, and determine a target projection closest to the spot image from the generated projections according to a preset judgment condition, so as to obtain a target manifold distribution corresponding to the target projection; The image reconstruction module is used to use the decoder in the backbone network to perform corresponding sampling decoding on the target manifold distribution to obtain a reconstructed image result corresponding to the spot image.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the non-viewing area image reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein when the computer program is executed by a processor, the non-viewing area image reconstruction method according to any one of claims 1 to 7 is implemented.

Images