Image generation methods and fusion imaging systems
By extracting, integrating, and fusing features from images from single-photon cameras and infrared cameras through a target image generation model, the problem of low imaging resolution of single-photon imaging radar in adverse weather and at night is solved, and high-resolution target images are generated.
Patent Information
- Application Number
- CN202310326803.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-29
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-03-29
AI Technical Summary
Single-photon imaging radar has low imaging resolution in adverse weather conditions and at night, resulting in loss of detail in the target scene and the inability to obtain clear three-dimensional images.
The target image generation model is used to extract features and perform similarity analysis on the 3D images acquired by the single-photon camera. The grayscale images acquired by the infrared camera are then combined and fused to generate a depth image to improve the resolution.
By integrating features and fusing images, the resolution of the target image was improved, the problem of poor imaging capabilities at night and in foggy weather was solved, and a high-resolution target image was generated.
Smart Images

Figure CN116342452B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image generation method, a fusion imaging system, a storage medium, and a program product. Background Technology
[0002] In recent years, single-photon detection technology has developed rapidly, enabling high-precision time measurement of weak optical signals. Single-photon imaging radar combines single-photon detection technology with traditional laser imaging radar, and can use extremely weak echo signals to perform high-precision three-dimensional imaging of target scenes.
[0003] In realizing the present invention, the inventors discovered that the related technology has at least the following problems: In some special environments, such as at night or in foggy weather, the single-photon imaging radar has a low imaging resolution, and when imaging a large field of view, it will lose the details of the target scene and can only obtain a relatively rough three-dimensional image of the target scene. Summary of the Invention
[0004] In view of the above, embodiments of this disclosure provide an image generation method, a fusion imaging system, a storage medium, and a program product.
[0005] One aspect of this disclosure provides an image generation method, comprising: acquiring a first image and a second image, wherein the first image is a three-dimensional image of a target scene acquired by a single-photon camera, and the second image is a grayscale image of the target scene acquired by an infrared camera; extracting features from the first image using a target image generation model to generate first feature data; performing similarity analysis on the first image using the target image generation model to generate second feature data; integrating the first feature data and the second feature data to generate a depth image; and fusing the depth image and the second image to generate a target image.
[0006] According to embodiments of this disclosure, the above-described method of using a target image generation model to extract features from the first image and generate first feature data includes: reducing the resolution of the first image to generate an intermediate image; segmenting the intermediate image to generate multiple segmented images; and using the target image generation model to perform similarity analysis on adjacent segmented images to generate first feature data.
[0007] According to an embodiment of this disclosure, the intermediate image includes a plurality of intermediate pixels. The above-mentioned similarity analysis of the first image using the target image generation model to generate second feature data includes: determining the pixel features of each of the intermediate pixels; performing similarity analysis using the target image generation model for any two of the pixel features to generate intermediate feature data; and obtaining second feature data based on the plurality of intermediate feature data.
[0008] According to an embodiment of this disclosure, the above-mentioned feature integration of the first feature data and the second feature data to generate a depth image includes: integrating the first feature data and the second feature data to generate depth data; and reconstructing an image based on the depth data to generate a depth image.
[0009] According to embodiments of this disclosure, the target image generation model is trained by the following method: filtering multiple initial pixels in a training image to obtain a denoised image, wherein the training image is a real image captured by a single-photon camera; inputting the denoised image into a forward imaging model to output a test dataset; inputting the test dataset into an initial image generation model to output a predicted image; determining the difference value between the predicted image and the training image; inputting the difference value into the loss function of the initial image generation model to output a loss value; and training the initial image generation model based on the loss value to obtain the target image generation model.
[0010] According to embodiments of this disclosure, the above-described denoising of the training image to obtain a denoised image includes: generating a noise floor curve based on the photon response time of a plurality of initial pixels; performing initial screening on the initial pixels based on the noise floor curve to obtain an initial screening image, the initial screening image including a plurality of initial screening pixels; determining the response region of each of the initial screening pixels, the response region including the initial screening pixel and a plurality of contrast pixels; and performing fine screening on the initial screening pixels based on the contrast response time to obtain the denoised image, the contrast response time being the photon response time corresponding to each of the contrast pixels.
[0011] According to an embodiment of this disclosure, the above-mentioned preliminary screening of the initial pixels based on the noise floor curve to obtain a preliminary screened image includes: determining the noise floor level of each of the initial pixels based on the noise floor curve; determining the response intensity corresponding to each of the initial pixels based on the noise floor level; and screening the initial pixels whose photon response time is greater than the response intensity to obtain the preliminary screened image.
[0012] According to embodiments of this disclosure, the aforementioned difference value includes a predicted difference value, a predicted sharpness value, and a predicted continuity value. Determining the difference value between the predicted image and the training image includes: determining the predicted difference value based on real pixels and predicted pixels in the predicted image, wherein the real pixels are pixels in the training image corresponding to the predicted pixels; determining the predicted sharpness value based on horizontal and vertical differences, wherein the horizontal difference is determined based on the horizontal pixel values of the predicted pixels and the real pixels, and the vertical difference is determined based on the vertical pixel values of the predicted pixels and the real pixels; and determining the predicted continuity value based on the predicted pixels and adjacent consecutive pixels.
[0013] Another aspect of this disclosure provides a fusion imaging system, comprising: a single-photon camera for acquiring a first image, wherein the first image is a three-dimensional image of a target scene; an infrared camera for acquiring a second image, wherein the second image is a grayscale image of the target scene; and an image processing device including a target image generation model, wherein the image processing device is configured to: extract features from the first image to generate first data; perform similarity analysis on the first image to generate second data; integrate the first data and the second data to generate a depth image; and fuse the depth image and the second image to generate a target image.
[0014] According to an embodiment of this disclosure, the system further includes: a signal generating device that transmits a signal to a laser; a laser that generates a laser pulse in response to the transmitted signal, and uses a laser emitting device of the laser to irradiate the laser pulse onto a target scene; and a telescope that collects reflected photons from the target scene and transmits the reflected photons to the single-photon camera.
[0015] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.
[0016] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.
[0017] According to embodiments of this disclosure, by employing a technique of extracting features from a first image using a target image generation model to generate first feature data; performing similarity analysis on the first image using the same model to generate second feature data; and integrating the first and second feature data to generate a depth image, the technical problem of poor imaging effect caused by unclear and low-resolution first images is at least partially overcome. By fusing the depth image and the second image to generate the target image, and given the strong fog and haze penetration capability of infrared cameras, the fusion of the depth image and the second image acquired by the infrared camera at least partially solves the technical problem of poor imaging capability at night and in foggy weather, thereby achieving the technical effect of improving the resolution of the target image. Attached Figure Description
[0018] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0019] Figure 1 An exemplary system architecture of an image generation method according to an embodiment of this disclosure is illustrated schematically;
[0020] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically;
[0021] Figure 3 A flowchart illustrating an image generation method according to another embodiment of the present disclosure is shown schematically;
[0022] Figure 4 A schematic diagram illustrating the structure of a target image generation model according to an embodiment of the present disclosure is shown.
[0023] Figure 5 An image reconstruction quality test diagram according to an embodiment of the present disclosure is illustrated schematically;
[0024] Figure 6 A block diagram of a fusion imaging system according to an embodiment of the present disclosure is shown schematically;
[0025] Figure 7 A block diagram of a fusion imaging system according to another embodiment of the present disclosure is shown schematically;
[0026] Figure 8 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown schematically. Detailed Implementation
[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.
[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0030] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).
[0031] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.
[0032] In recent years, single-photon detection technology has flourished. This technology boasts detection response sensitivity at the single-photon level and time resolution accuracy at the picosecond level, enabling high-precision time measurement of extremely weak light signals. It has found extensive applications in research fields such as lidar, biological imaging, and molecular spectroscopy. Single-photon imaging radar combines single-photon detection technology with traditional lidar imaging radar, offering advantages such as high sensitivity and high time measurement accuracy compared to traditional lidar. It can utilize extremely weak echo signals to perform high-precision 3D imaging of target scenes, showing immense application potential in autonomous driving, geographic mapping, and situational awareness.
[0033] Currently, the receivers of single-photon imaging radar can be mainly divided into single-point single-photon detectors and single-photon detector arrays. Single-point single-photon detectors have good detection performance, but they can only image one pixel of the target at a time, requiring scanning to obtain a complete three-dimensional image of the target, which usually results in a long imaging time. The emergence of single-photon detector arrays reduces the scanning time required for imaging the target object, enabling single-photon imaging radar to achieve a higher imaging frame rate.
[0034] However, due to current limitations in single-photon detector integration and data transmission bandwidth, the number of pixels in common single-photon detector arrays is typically below 100,000, while the number of pixels in single-photon detector arrays integrating high-precision time-to-digital converters (TDCs) is almost always below 10,000. Compared to the tens of millions of pixels in traditional CCD (charge-coupled device) and CMOS (complementary metal-oxide-semiconductor) detector arrays, current single-photon imaging radar based on single-photon detector arrays has lower imaging resolution. In adverse weather conditions and at night, it loses detail of the target scene during large field-of-view imaging, resulting in only a relatively coarse 3D image of the target scene.
[0035] In view of the above, embodiments of this disclosure provide an image generation method, a fusion imaging system, and a storage medium. The method includes: acquiring a first image and a second image, wherein the first image is a three-dimensional image of a target scene acquired by a single-photon camera, and the second image is a grayscale image of the target scene acquired by an infrared camera; extracting features from the first image using a target image generation model to generate first feature data; performing similarity analysis on the first image using the target image generation model to generate second feature data; integrating the first feature data and the second feature data to generate a depth image; and fusing the depth image and the second image to generate a target image.
[0036] Figure 1 An exemplary system architecture of an image generation method according to embodiments of this disclosure is illustrated schematically. It should be noted that... Figure 1The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.
[0037] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links.
[0038] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).
[0039] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0040] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.
[0041] It should be noted that the image generation method provided in this disclosure embodiment can generally be executed by server 105. The image generation method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. The image generation method provided in this disclosure embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103.
[0042] For example, the first image and the second image may originally be stored in any one of terminal devices 101, 102, or 103 (e.g., terminal device 101, but not limited thereto), or they may be stored on an external storage device and imported into terminal device 101. Then, terminal device 101 may execute the image generation method provided in the embodiments of this disclosure locally, or send the first image and the second image to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the image to be processed execute the image generation method provided in the embodiments of this disclosure.
[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0044] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically.
[0045] like Figure 2 As shown, the method includes operations S201 to S205.
[0046] In operation S201, a first image and a second image are acquired. The first image is a three-dimensional image of the target scene acquired by a single-photon camera, and the second image is a grayscale image of the target scene acquired by an infrared camera.
[0047] In operation S202, the target image generation model is used to extract features from the first image to generate the first feature data.
[0048] In operation S203, the target image generation model is used to perform similarity analysis on the first image to generate second feature data.
[0049] In operation S204, the first feature data and the second feature data are integrated to generate a depth image.
[0050] In operation S205, the depth image and the second image are fused to generate the target image.
[0051] According to embodiments of this disclosure, a single-photon camera uses a single-photon array detector as a receiving device to receive the acquired first image. Compared to a scanning-based single-point detector single-photon imaging system, the single-photon array detector can achieve parallel detection of multiple pixels, greatly improving the imaging speed. It can achieve an imaging frame rate of 20Hz even at long distances (>20km), enabling imaging of moving targets and possessing higher practical value. The first image acquired by the single-photon camera is a three-dimensional image of the target scene. An infrared camera acquires a second image, which is a high-resolution grayscale image. According to embodiments of this disclosure, the resolution of the first image is lower than that of the second image.
[0052] According to embodiments of this disclosure, a target image generation model is used to extract features from a first image. Feature extraction involves extracting relatively universal single-photon image information from the input first image, including the distance and brightness of pixels in the first image. The target image generation model is then used to perform similarity analysis on the first image. This similarity analysis can be a non-local analysis of the features of pixels in the first image, analyzing potential non-local spatiotemporal information in the three-dimensional image, including the distance, brightness, and histogram shape of each pixel. Through similarity analysis, pixels with similar features in the first image can borrow signals from each other.
[0053] According to embodiments of this disclosure, first feature data and second feature data are integrated. The relatively universal single-photon image information in the first feature data and the non-local information in the second feature data are integrated, thereby strengthening the signal of pixels in the first image. After feature integration, image reconstruction is performed by restoring the features to real data, thereby generating a depth image. The depth image is clearer than the first image and is a two-dimensional image.
[0054] According to embodiments of this disclosure, a depth image and a second image are fused. A high-resolution grayscale image acquired by an infrared camera is fused with a low-resolution depth image to obtain a high-resolution single-photon imaging result, i.e., the target image. During the fusion process, pixel registration can be performed on the two images based on their image features. Pixel values in the depth image are then filtered and corrected based on the pixel intensity values provided by the second image. Additionally, the depth image is augmented using the overall image features provided by the second image, thereby generating the target image.
[0055] According to embodiments of this disclosure, by employing a technique of extracting features from a first image using a target image generation model to generate first feature data; performing similarity analysis on the first image using the same model to generate second feature data; and integrating the first and second feature data to generate a depth image, the technical problem of poor imaging effect caused by unclear and low-resolution first images is at least partially overcome. By fusing the depth image and the second image to generate the target image, and given the strong fog and haze penetration capability of infrared cameras, the fusion of the depth image and the second image acquired by the infrared camera at least partially solves the technical problem of poor imaging capability at night and in foggy weather, thereby achieving the technical effect of improving the resolution of the target image.
[0056] According to embodiments of this disclosure, extracting features from a first image using a target image generation model to generate first feature data may include the following steps.
[0057] The resolution of the first image is reduced to generate an intermediate image; the intermediate image is segmented to generate multiple segmented images; the similarity analysis of adjacent segmented images is performed using the target image generation model to generate the first feature data.
[0058] According to embodiments of this disclosure, reducing the resolution of the first image facilitates feature extraction. The intermediate image is then segmented based on signal intensity, resulting in multiple segmented images. The intermediate image, composed of these segmented images, is input into a target generation model for similarity analysis, analyzing the similarity between adjacent segmented images. Segmented images containing real signals have low similarity to those without. By analyzing adjacent segmented images, those containing real signals are extracted to obtain universal single-photon image information, generating first feature data.
[0059] According to embodiments of this disclosure, the intermediate image includes multiple intermediate pixels. The second feature data is generated by performing similarity analysis on the first image using a target image generation model, which may include the following operations.
[0060] Determine the pixel features of each intermediate pixel; for any two pixel features, perform similarity analysis using the target image generation model to generate intermediate feature data; obtain the second feature data based on multiple intermediate feature data.
[0061] According to embodiments of this disclosure, the pixel features of intermediate pixels include the distance between pixels, their brightness, and histogram shape. Similarity analysis of the first image can be performed directly on the intermediate image; alternatively, the resolution of the first image can be reduced to generate an analysis image, and similarity analysis can be performed on the analysis image. The resolution of the analysis image is greater than that of the intermediate image. For any two pixel features, a similarity analysis is performed using a target image generation model. Non-locality analysis is conducted on the pixels to determine the similarity between pixels in different regions, generating intermediate feature data. If the similarity is high, signals can be borrowed from each other.
[0062] According to embodiments of this disclosure, feature integration of first feature data and second feature data to generate a depth image may include the following operations.
[0063] The first feature data and the second feature data are integrated to generate depth data; the image is reconstructed based on the depth data to generate a depth image.
[0064] According to embodiments of this disclosure, universal information in the first feature data and non-local information in the second feature data are integrated to enhance the signal of pixels in the first image. After feature integration, a target image generation model is used to perform depth prediction based on depth data, restoring the features to the real data, thereby completing image reconstruction and outputting a depth image. During image reconstruction, non-signal data is removed to achieve denoising, resulting in a high resolution depth image.
[0065] Figure 3 A flowchart illustrating an image generation method according to another embodiment of the present disclosure is shown schematically.
[0066] According to embodiments of this disclosure, such as Figure 3 As shown, the input single-photon camera data is globally filtered to obtain denoised single-photon camera data, which is still a 3D image. The denoised single-photon camera data is then input into a neural network for feature extraction. Relatively universal single-photon image information is extracted from the input 3D data. Potential non-local spatiotemporal information in the single-photon camera data is analyzed. The extracted universal and non-local information are integrated. Based on the feature-integrated output data, depth prediction is performed, completing image reconstruction and outputting a depth map. The infrared camera data and the depth map result are fused using an image fusion algorithm to finally output a reconstructed depth map. The reconstructed depth map has high resolution, and except for the image captured by the infrared camera (which is grayscale), all other images are color images. Figure 3 This is just an illustration.
[0067] According to embodiments of this disclosure, the target image generation model is trained using the following method:
[0068] Multiple initial pixels in the training image are filtered to obtain a denoised image. The training image is a real image captured by a single-photon camera. The denoised image is input into the forward imaging model, and the output is a test dataset. The test dataset is input into the initial image generation model, and the output is a predicted image. The difference between the predicted image and the training image is determined. The difference value is input into the loss function of the initial image generation model, and the output is the loss value. The initial image generation model is trained based on the loss value to obtain the target image generation model.
[0069] According to embodiments of this disclosure, a test dataset is generated before training the target image generation model, and the training images are denoised to enhance the signal. Since the training images are three-dimensional images, the corresponding single-photon camera data is also three-dimensional data, represented as a three-dimensional matrix. Each pixel is represented by three-dimensional coordinates, forming a three-dimensional histogram with a coordinate system of m×n×h. Here, m×n represents the spatial dimension, i.e., the number of pixels, where m is the horizontal pixel value and n is the vertical pixel value; h represents the temporal dimension, characterizing the scale on the time axis, i.e., the photon flight time.
[0070] According to embodiments of this disclosure, a denoised image is input into a forward imaging model to generate a test dataset. The forward imaging model is as follows:
[0071] y p,q [i]~Poisson(ηα p,q s p,q [i]+b) (1)
[0072] Among them, y p,q [i] represents the histogram response at the i-th time axis scale at pixel (p, q), η represents the system detection efficiency, and α p,q This represents the reflectivity of the pixel, s p,q [i] represents the time response of the pixel, b is noise, and Poisson represents a non-uniform Poisson process.
[0073] According to embodiments of this disclosure, a difference value between a predicted image and a training image is determined. This difference value includes a predicted difference value between the predicted image and the training image, a predicted sharpness value between the predicted image and the training image, and a predicted continuity value for the predicted image. The predicted difference value characterizes the difference between the predicted image and the training image and is a core objective of training. The predicted sharpness value indicates the edge sharpness of the predicted image; optimizing this objective helps improve the local edge abruptness information of the predicted image. The predicted continuity value characterizes the continuity of the predicted image; optimizing this objective helps improve the continuity information within each local area of the predicted image.
[0074] According to embodiments of this disclosure, the difference value is input into the loss function of the initial image generation model, and the loss value is output. The initial image generation model is trained based on the loss value; by reducing the loss value, the model's predictions become more accurate. The loss function is as follows:
[0075] ObjFunc=D KL +μ·Grad+γ·TV (2)
[0076] Where ObjFunc represents the loss value, and D KL represents the predicted difference value, Grad represents the predicted sharp value, TV represents the predicted continuous value, and μ and γ represent the weight values.
[0077] Figure 4 The schematic diagram illustrates the structure of a target image generation model according to an embodiment of the present disclosure.
[0078] According to embodiments of this disclosure, such as Figure 4 As shown, the target image generation model is divided into a feature extraction module 410, a nonlocality analysis module 420, a feature integration module 430, and an image reconstruction module 440. The feature extraction module 410 segments the image acquired by the single-photon camera, performs similarity analysis on adjacent images, and then extracts relatively universal single-photon image information. The nonlocality analysis module 420 is used to analyze the potential nonlocal spatiotemporal information in the 3D data. The feature integration module 430 integrates the extracted universal and nonlocal information. The image reconstruction module 440 performs depth prediction based on the output of the feature integration module, completes image reconstruction, and outputs the image estimation result.
[0079] According to embodiments of this disclosure, denoising a training image to obtain a denoised image may include the following operations.
[0080] A noise floor curve is generated based on the photon response time of multiple initial pixels. The initial pixels are then initially screened based on the noise floor curve to obtain an initial screened image, which includes multiple initial screened pixels. The response region of each initial screened pixel is determined, which includes the initial screened pixel and multiple contrast pixels. The initial screened pixels are then finely screened based on the contrast response time to obtain a denoised image, where the contrast response time is the photon response time corresponding to each contrast pixel.
[0081] According to embodiments of this disclosure, a response histogram is generated based on the photon response times of multiple initial pixels. The horizontal axis of the response histogram represents the photon response time of each initial pixel, and the vertical axis represents the number of times that photon response time occurs. A noise floor curve is fitted based on the response histogram, in which each initial pixel has a corresponding noise level. The initial pixels are then initially screened based on the noise floor curve to denoise the training image, removing data that is not a signal.
[0082] According to embodiments of this disclosure, the initial screening image is further filtered to determine the response region of each initial screening pixel. That is, for the response histogram of each initial screening pixel, the initial screening pixel is further screened by combining the response of the comparison pixel in the surrounding p×p response region (typical values of p are generally 3, 5, 7, 9), and the pixel with the highest response and longest response time in the response region is selected, and the response of other pixels in the response region is set to 0.
[0083] According to embodiments of this disclosure, initial screening of initial pixels based on the noise floor curve to obtain an initial screened image may include the following operations.
[0084] The noise level of each initial pixel is determined based on the noise floor curve; the response intensity of each initial pixel is determined based on the noise floor level; pixels with photon response time greater than response intensity are selected from the initial pixels to obtain the initial screened image.
[0085] According to embodiments of this disclosure, the noise floor level can be represented by σ, and the response intensity changes with the noise floor level. The response intensity can be set to... Pixels with photon response times greater than their response intensities are selected from the initial pixel count, and the remaining pixels with lower response times are removed to obtain the initial screened image.
[0086] According to embodiments of this disclosure, determining the difference between the predicted image and the training image may include the following operations.
[0087] The prediction difference value is determined based on the real pixels and the predicted pixels in the prediction image, where the real pixels are the pixels in the training image that correspond to the predicted pixels. The prediction sharpness value is determined based on the horizontal difference and the vertical difference, where the horizontal difference is determined based on the horizontal pixel values of the predicted pixels and the real pixels, and the vertical difference is determined based on the vertical pixel values of the predicted pixels and the real pixels. The prediction continuity value is determined based on the predicted pixels and the consecutive pixels adjacent to the predicted pixels.
[0088] According to embodiments of this disclosure, the true pixels are the pixels in the training image corresponding to the predicted pixels. In calculating the difference value, the predicted pixels include a first predicted pixel (p, q), a second predicted pixel (p+1, q), and a third predicted pixel (p, q+1). The true pixels include a first true pixel (p, q) corresponding to the first predicted pixel, a second true pixel (p+1, q) corresponding to the second predicted pixel, and a third true pixel (p, q+1) corresponding to the third predicted pixel. The prediction difference value is determined based on the ratio of the true pixels to the predicted pixels in the prediction image, wherein the prediction difference value is calculated using the following formula:
[0089]
[0090] Where y represents the three-dimensional data of the training image, This represents the three-dimensional data of the predicted image. The subscripts p and q represent the x-coordinate and y-coordinate of the first predicted pixel, and the x-coordinate and y-coordinate of the first real pixel corresponding to the first predicted pixel, respectively. i represents the coordinate of the corresponding pixel on the time axis, and n represents the number of photon flight times on the time axis in the three-dimensional histogram.
[0091] According to embodiments of this disclosure, a predicted sharpness value is determined based on the difference between the actual pixel and the predicted pixel, wherein the formula for calculating the predicted sharpness value is as follows:
[0092]
[0093] Wherein, the subscripts p+1 and q represent the x and y coordinates of the second predicted pixel and the first real pixel corresponding to the second predicted pixel, respectively; and the subscripts p and q+1 represent the x and y coordinates of the third predicted pixel and the third real pixel corresponding to the third predicted pixel, respectively.
[0094] According to embodiments of this disclosure, a predicted continuous value is determined based on the predicted pixel and its adjacent consecutive pixels, wherein the formula for calculating the predicted continuous value is as follows:
[0095]
[0096] Wherein, the subscripts p and q+1 represent the x-coordinate and y-coordinate of the third predicted pixel, respectively.
[0097] Figure 5 An image reconstruction quality test diagram according to an embodiment of the present disclosure is illustrated schematically.
[0098] According to embodiments of this disclosure, such as Figure 5 As shown, the processing of single-photon camera data is applicable to low signal-to-noise ratio (SNR) data (≤1 signal photon / pixel) with few photons. On one hand, the global filtering part utilizes the different temporal distribution characteristics of signal photons and noise photons to filter noise, thereby suppressing some of the noise's influence. On the other hand, the nonlocality analysis and other modules included in the target image generation model further filter out noise and enhance the signal by considering the spatiotemporal correlations such as nonlocality and continuity inherent in the natural scene itself. Ultimately, the combined effect of these two parts can transform the input single-photon camera data with extremely low SNR into a clear depth estimation map. Figure 5The reconstruction performance of this image generation method is compared with other single-photon imaging methods under a series of signal-to-noise ratio (SNR) conditions. It can be seen that this method exhibits superior reconstruction performance under various SNR conditions, especially at SNRs as low as 0.01.
[0099] Figure 6 A block diagram of a fusion imaging system according to an embodiment of the present disclosure is shown schematically.
[0100] like Figure 6 As shown, the fusion imaging system 600 includes: a single-photon camera 610 for acquiring a first image, which is a three-dimensional image of the target scene; an infrared camera 620 for acquiring a second image, which is a grayscale image of the target scene; and an image processing device 630, which includes a target image generation model. The image processing device 630 is used to: extract features from the first image to generate first data; perform similarity analysis on the first image to generate second data; integrate the first data and the second data to generate a depth image; and fuse the depth image and the second image to generate a target image.
[0101] According to embodiments of this disclosure, the single-photon camera 610 can be an APD avalanche focal plane array with a pixel size of 64×64. Each pixel has an integrated time-to-digital converter (TDC) with a time resolution of 1 ns, capable of recording the interval between photon arrival and emission times. The single-photon camera 610 transmits control commands and data to a computer via a Cameralink (data transmission) cable. The infrared camera 620 can be a mid-wave infrared detector with 640×512 pixels, operating in the 3.7-4.8 μm band, with a maximum focal length of 1100 mm. The field of view of the infrared camera 620 and the imaging field of view of the single-photon camera 610 are aligned through calibration to ensure matching of the imaging field of view. The first image transmitted from the single-photon camera 610 to the image processing device 630 and the second image transmitted from the infrared camera 620 to the image processing device 630 are processed.
[0102] According to embodiments of this disclosure, by employing a technique of extracting features from a first image using a target image generation model to generate first feature data; performing similarity analysis on the first image using the same model to generate second feature data; and integrating the first and second feature data to generate a depth image, the technical problem of poor imaging effect caused by unclear and low-resolution first images is at least partially overcome. By fusing the depth image and the second image to generate the target image, and given the strong fog and haze penetration capability of infrared cameras, the fusion of the depth image and the second image acquired by the infrared camera at least partially solves the technical problem of poor imaging capability at night and in foggy weather, thereby achieving the technical effect of improving the resolution of the target image.
[0103] Figure 7 A block diagram of a fusion imaging system according to another embodiment of the present disclosure is shown schematically.
[0104] According to an embodiment of this disclosure, the fusion imaging system 600 further includes: a signal generating device 640 for transmitting a transmission signal to a laser 650; a laser 650 for generating a laser pulse in response to the transmission signal, and using the laser emitting device of the laser 650 to irradiate the target scene with the laser pulse; and a telescope 660 for collecting reflected photons from the target scene and transmitting the reflected photons to a single-photon camera 610.
[0105] According to embodiments of this disclosure, such as Figure 7 As shown, the signal generating device 640 can be an arbitrary function generator (AFG). The signal generating device 640 generates two synchronous pulse signals with fixed frequencies, which are respectively used as the start signal for the time-to-digital converter (TDC) of the single-photon camera 610 and as the emission signal for the laser pulse of the laser 650. The laser pulse generated by the laser 650 is emitted onto the target scene through the laser emitting device. In the laser emission section, a near-infrared pulsed fiber laser is triggered by a signal source to generate periodic pulses, which are then emitted onto the target scene through the laser emitting device. In this embodiment, a pulsed fiber laser with a wavelength of 1550nm, a repetition frequency of 25kHz, and a pulse width of 1ns can be selected as the light source. Narrow-pulse-width nanosecond, sub-nanosecond, or picosecond pulsed lasers in the near-infrared band can also be selected. The laser emitting device can employ a fiber collimator design or a convex lens group.
[0106] According to embodiments of this disclosure, reflected photons from the target scene reach the single-photon camera 610 via telescope 660. Telescope 660 can be a Cassegrain reflecting telescope with a 280mm aperture and 2800mm focal length, or other telescopes with different apertures and focal lengths, and an optimally shaped convex lens as the field-of-view adjustment lens. After each pixel of the single-photon camera 610 receives the single-photon signal, the time-of-arrival (TDC) record after each pixel records the photon arrival time. The difference between this TDC and the initial signal time yields the photon's time of flight, thus determining the distance to the target portion corresponding to that pixel. All data measured by the pixel TDCs are transmitted to the image processing device 630, where they are processed by an algorithm to obtain a depth image.
[0107] According to an embodiment of this disclosure, the infrared camera 620 and the field of view center of the single-photon imaging system are aligned to the same position after adjustment. The infrared camera 620 simultaneously obtains a high-resolution, large-pixel grayscale image of the target, and the data is transmitted to the image processing device 630.
[0108] According to embodiments of this disclosure, such as Figure 7As shown, the reflected photons from the target scene pass through a lens and a spectral filter before reaching the single-photon camera 610. The spectral filter can be a 1500nm long-pass filter or a 1550nm narrow-pass filter to suppress background noise from sunlight during the day, enabling all-day operation. Both the single-photon camera 610 and the infrared camera 620 are equipped with motorized turntables. The image processing device 630 can control the motorized turntables to rotate towards the target to be imaged, and, with guidance, can also achieve tracking and imaging of moving targets.
[0109] It should be noted that the fusion imaging system part in the embodiments of this disclosure corresponds to the image generation method part in the embodiments of this disclosure. The description of the fusion imaging system part is specifically referred to in the image generation method part, and will not be repeated here.
[0110] Figure 8 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown schematically.
[0111] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0112] like Figure 8 As shown, an image processing apparatus 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from storage portion 808 into random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0113] RAM 803 stores various programs and data required for the operation of the image processing device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.
[0114] According to embodiments of this disclosure, the image processing device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The image processing device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.
[0115] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0116] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.
[0117] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0118] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.
[0119] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the image generation method provided in the embodiments of this disclosure.
[0120] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiment.
[0121] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0122] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.
[0124] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.
Claims
1. An image generation method, comprising: Acquire a first image and a second image. The first image is a three-dimensional image of the target scene captured by a single-photon camera, and the second image is a grayscale image of the target scene captured by an infrared camera. The infrared camera is a mid-wave infrared detector with the ability to penetrate fog and haze. The first image is used to extract features from the target image generation model, and single-photon image information is extracted from the first image to generate first feature data. The target image generation model is used to perform nonlocal analysis on the first image to analyze the nonlocal spatiotemporal information in the first image, so that pixels with the same features in the first image can borrow signals from each other to generate second feature data. The single-photon image information in the first feature data and the non-local spatiotemporal information in the second feature data are integrated to generate a depth image; The depth image and the second image are fused to generate the target image; The step of fusing the depth image and the second image to generate a target image includes: performing pixel registration on the depth image and the second image according to the image features of the depth image and the second image; filtering and correcting the pixel values of the depth image according to the pixel intensity values provided by the second image; and expanding the depth image using the overall image features provided by the second image to generate the target image. The target image generation model is trained using the following method: Multiple initial pixels in the training image are filtered to obtain a denoised image, wherein the training image is a real image captured by a single-photon camera; The denoised image is input into the forward imaging model, and the test dataset is output. The test dataset is input into the initial image generation model, which outputs a predicted image. The difference value between the predicted image and the training image is determined. The difference value includes a predicted difference value, a predicted sharpness value, and a predicted continuity value. The predicted sharpness value is determined based on the horizontal and vertical differences between the predicted pixels in the predicted image and the corresponding real pixels in the training image. The predicted sharpness value is used to constrain the edge sharpness of the predicted image. The difference value is input into the loss function of the initial image generation model, and the loss value is output. The initial image generation model is trained based on the loss value to obtain the target image generation model.
2. The method according to claim 1, wherein, The step of extracting features from the first image using a target image generation model to generate first feature data includes: Reduce the resolution of the first image to generate an intermediate image; The intermediate image is segmented to generate multiple segmented images; The similarity analysis of adjacent segmented images is performed using a target image generation model to generate first feature data.
3. The method according to claim 2, wherein, The intermediate image includes multiple intermediate pixels. The step of using the target image generation model to perform similarity analysis on the first image to generate second feature data includes: Determine the pixel features of each of the intermediate pixels; For any two pixel features, similarity analysis is performed using the target image generation model to generate intermediate feature data; The second feature data is obtained based on multiple intermediate feature data.
4. The method according to claim 1, wherein, The step of integrating the first feature data and the second feature data to generate a depth image includes: The first feature data and the second feature data are integrated to generate depth data; Based on the depth data, the image is reconstructed to generate a depth image.
5. The method according to claim 1, wherein, The step of denoising the training image to obtain a denoised image includes: A noise floor curve is generated based on the photon response time of multiple initial pixels. The initial pixels are initially screened based on the noise floor curve to obtain a preliminary screened image, which includes multiple preliminary screened pixels. Determine the response region for each of the initial screening pixels, wherein the response region includes the initial screening pixel and a plurality of comparison pixels; The initial screening pixels are further screened based on the contrast response time to obtain the denoised image, wherein the contrast response time is the photon response time corresponding to each contrast pixel.
6. The method according to claim 5, wherein, The initial screening of the initial pixels based on the noise floor curve to obtain the initial screened image includes: The noise level of each initial pixel is determined based on the noise floor curve. The response intensity corresponding to each initial pixel is determined based on the noise floor level; The initial image is obtained by selecting pixels whose photon response time is greater than the response intensity from the initial pixels.
7. The method according to claim 1, wherein, The difference value includes a predicted difference value, a predicted sharpness value, and a predicted continuity value. Determining the difference value between the predicted image and the training image includes: The prediction difference value is determined based on the ratio of the real pixel to the predicted pixel in the prediction image, wherein the real pixel is the pixel in the training image that corresponds to the predicted pixel. The predicted sharpness value is determined based on the difference between the actual pixel and the predicted pixel. A predicted continuous value is determined based on the predicted pixel and the consecutive pixels adjacent to the predicted pixel.
8. A fusion imaging system, comprising: A single-photon camera is used to acquire a first image, which is a three-dimensional image of the target scene; An infrared camera is used to acquire a second image, which is a grayscale image of the target scene. The infrared camera is a mid-wave infrared detector with the ability to penetrate fog and haze. Image processing device, the image processing device including a target image generation model, the image processing device being used for: Feature extraction is performed on the first image to extract single-photon image information and generate first data; nonlocal analysis is performed on the first image to analyze the nonlocal spatiotemporal information in the first image, so that pixels with the same features in the first image can borrow signals from each other to generate second data; the single-photon image information in the first data and the nonlocal spatiotemporal information in the second data are integrated to generate a depth image; The depth image and the second image are fused to generate the target image; The step of fusing the depth image and the second image to generate a target image includes: performing pixel registration on the depth image and the second image according to the image features of the depth image and the second image; filtering and correcting the pixel values of the depth image according to the pixel intensity values provided by the second image; and expanding the depth image using the overall image features provided by the second image to generate the target image. The target image generation model is trained using the following method: Multiple initial pixels in the training image are filtered to obtain a denoised image, wherein the training image is a real image captured by a single-photon camera; The denoised image is input into the forward imaging model, and the test dataset is output. The test dataset is input into the initial image generation model, which outputs a predicted image. The difference value between the predicted image and the training image is determined. The difference value includes a predicted difference value, a predicted sharpness value, and a predicted continuity value. The predicted sharpness value is determined based on the horizontal and vertical differences between the predicted pixels in the predicted image and the corresponding real pixels in the training image. The predicted sharpness value is used to constrain the edge sharpness of the predicted image. The difference value is input into the loss function of the initial image generation model, and the loss value is output. The initial image generation model is trained based on the loss value to obtain the target image generation model.
9. The system according to claim 8, further comprising: A signal generating device that transmits a signal to a laser; A laser is used to generate laser pulses in response to the emission signal, and the laser pulses are irradiated onto the target scene using the laser emitting device of the laser. A telescope is used to collect reflected photons from the target scene and transmit the reflected photons to the single-photon camera.
Citation Information
Patent Citations
Three-dimensional imaging system and method for fusing single-photon laser radar and short-wave infrared image
CN112731443A
