A deep learning-based correlation imaging method and system

Abstract

The present invention provides a deep learning-based correlation imaging method and system. The system includes: a light source for emitting a speckle matrix toward a target to be imaged; a bucket detector for measuring light intensity values ​​of the speckle matrix after being projected onto the target; and a computing unit, the computing unit being configured to: acquire an initial image of the target to be imaged; divide the initial image into a plurality of image blocks; perform similarity clustering on the image blocks; determine a matching speckle-like matrix for each image block based on the clustering; combine the speckle matrices of each image block into a complete speckle matrix and provide the complete speckle matrix to the light source; and obtain a reconstructed image of the target through correlation calculation based on the speckle matrix and the measurement values ​​of the bucket detector.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning technology and associative imaging, and more specifically, to a deep learning-based associative imaging method and system. Background Art

[0002] Correlation imaging has been a hot topic and cutting-edge technology in quantum optics for decades. To overcome the challenges of detecting objects obscured by clouds or complex backgrounds, as well as the effects of atmospheric disturbances, ocean turbulence, system noise, and relative motion, both inherent and objective factors can degrade image quality, often resulting in image distortion and blurring, leading to low image recognition. Correlation imaging can effectively improve the image's robustness against interference.

[0003] Compared with traditional imaging technology, correlation imaging is non-local. It can be achieved by placing a detector with spatial resolution capability on a reference light path that does not contain the object to be measured, and then obtaining the reconstructed object image through coincidence calculation. Figure 1 is a schematic scene diagram of associated imaging in the prior art, such as Figure 1 As shown in , the light source generates speckles, which are projected over the object and then received and measured by a detector without spatial resolution. Finally, a reconstructed image of the object is obtained through correlation calculation.

[0004] With the research and development of correlation imaging technology, thermal-optical correlation imaging and computational correlation imaging have been proposed. These technologies are simpler to implement, allowing correlation imaging to be applied in a wider range of fields. Currently, correlation imaging can quickly and effectively provide information about the target object, even when it is obscured or in a complex background, using only a bucket detector without spatial resolution. This information includes various high-altitude, ground-based, or underwater observation data, and can be applied in fields such as topographic mapping, military reconnaissance, ocean exploration, high-altitude detection, medical imaging, satellite remote sensing, and lidar.

[0005] However, in practical applications, larger images require more speckle patterns, leading to longer reconstruction times. This requires the use of other algorithms to improve imaging time and quality. To improve the quality and resolution of correlative imaging, one approach that can be considered is to increase the resolution of the imaging hardware. However, this approach has the disadvantages of high hardware cost, high technical difficulty, and limited potential for improvement. Therefore, it is desirable to provide a more effective solution for improving the quality and resolution of correlative imaging. Summary of the Invention

[0006] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In order to solve the technical problems mentioned in the background technology section, the present invention provides a deep learning-based associative imaging method and system, which optimizes speckle through image segmentation and dictionary learning technology, so that the image of an unknown object can be reconstructed with a minimum number of speckles.

[0008] According to one aspect of the present invention, a method for training an optimized speckle matrix for correlation imaging is provided, the method comprising:

[0009] Divide the training image into multiple image blocks;

[0010] Perform similarity clustering on image blocks;

[0011] Perform dictionary learning on each block cluster to obtain multiple class dictionaries; and

[0012] An optimized speckle-like matrix corresponding to each block cluster is calculated based on the obtained class dictionary.

[0013] According to a further embodiment of the present invention, performing similarity clustering on the image blocks further comprises:

[0014] The K-means clustering algorithm is performed on the divided image blocks to group multiple image blocks with similar image structural features into block clusters.

[0015] According to a further embodiment of the present invention, performing dictionary learning on each block cluster further includes:

[0016] For different block clusters, different dictionary learning methods are selected according to the image structure characteristics of the block clusters to perform dictionary learning.

[0017] According to a further embodiment of the present invention, calculating the optimized speckle-like matrix corresponding to each block cluster based on the obtained class dictionary further includes:

[0018] For different block clusters, different speckle optimization methods are selected according to the image structure characteristics of the block clusters to perform speckle optimization.

[0019] According to another aspect of the present invention, a correlation imaging method is provided, the method comprising:

[0020] Acquiring an initial image of the target to be imaged;

[0021] Dividing the initial image into a plurality of image blocks;

[0022] Perform similarity clustering on image blocks;

[0023] Determine a matching speckle-like matrix for each image block based on clustering;

[0024] Combining the speckle matrices of each image block into a complete speckle matrix; and

[0025] The combined speckle matrix is ​​used for correlated imaging with the bucket detector.

[0026] According to a further embodiment of the present invention, performing correlated imaging using the combined speckle matrix and the corresponding combined bucket detector further comprises:

[0027] Using each set of speckle matrices and the corresponding bucket detectors to perform correlation imaging to obtain low-resolution image blocks; and

[0028] A high-resolution image is obtained by weighting the obtained low-resolution image blocks.

[0029] According to a further embodiment of the present invention, performing similarity clustering on the image blocks further comprises:

[0030] The K-means clustering algorithm is performed on the divided image blocks to group multiple image blocks with similar image structural features into block clusters.

[0031] According to a further embodiment of the present invention, determining a matching speckle-like matrix for each image block based on clustering further comprises:

[0032] By comparing the means, a closest pre-trained block cluster is determined for each block cluster to which the current image block belongs in multiple pre-trained block clusters;

[0033] The optimized speckle-like matrix corresponding to the closest pre-training block cluster is used as the speckle matrix of the current image block.

[0034] According to a further embodiment of the present invention, combining the speckle matrix of each image block into a complete speckle matrix further comprises:

[0035] The speckle matrix is ​​combined according to the position of each image block in the initial image, thereby obtaining the complete speckle matrix.

[0036] According to yet another aspect of the present invention, there is provided a correlation imaging system, the system comprising:

[0037] a light source for emitting a speckle matrix toward a target to be imaged;

[0038] a bucket detector, configured to measure the light intensity value of the speckle matrix after projecting through the target; and

[0039] A computing unit, the computing unit being configured to:

[0040] Acquiring an initial image of the target to be imaged;

[0041] Dividing the initial image into a plurality of image blocks;

[0042] Perform similarity clustering on image blocks;

[0043] Determine a matching speckle-like matrix for each image block based on clustering;

[0044] Combining the speckle matrix of each image block into a complete speckle matrix and providing the complete speckle matrix to the light source; and

[0045] Based on the speckle matrix and the measurement value of the bucket detector, a reconstructed image of the target is obtained through correlation calculation.

[0046] Compared with the solutions in the prior art, the correlation imaging system and method provided by the present invention have at least the following advantages:

[0047] 1. Through image segmentation and clustering, the number of speckles and the computational complexity required for speckle matrix at high resolution are reduced;

[0048] 2. Perform multi-dictionary learning on clusters, and perform dictionary learning and speckle optimization based on the image structure characteristics of clusters, further reducing the number of speckles and computational complexity, and improving reconstruction clarity.

[0049] These and other features and advantages will become apparent from reading the following detailed description and referring to the associated drawings.It is to be understood that both the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order that the manner in which the above-mentioned features of the present invention are understood in detail, a more particular description of the contents briefly summarized above may be given with reference to various embodiments, some aspects of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only certain typical aspects of the invention and are not to be considered limiting of its scope, as the description may admit to other equally effective aspects.

[0051] Figure 1 It is a schematic scene diagram of associated imaging in the prior art.

[0052] Figure 2 4 is a schematic flowchart of a deep learning-based optimized speckle matrix training method according to an embodiment of the present invention.

[0053] Figure 3 is a schematic flow chart of a correlation imaging method according to an embodiment of the present invention.

[0054] Figure 4 is a schematic structural diagram of a correlation imaging system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below with reference to the accompanying drawings, and the features of the present invention will be further apparent in the following specific description.

[0056] As mentioned earlier, a problem with existing techniques is that as the resolution requirements for correlation imaging increase, the number of speckle patterns required for image reconstruction increases significantly. This leads to a corresponding increase in the speckle projection time required for correlation imaging and the correlation computational complexity during reconstruction. To address this issue, the present invention proposes training a speckle adaptive selection model based on image segmentation and dictionary learning. Using this trained model, during image reconstruction, optimized speckle patterns are adaptively selected based on image segmentation, and then assembled into a complete speckle matrix for correlation imaging. This significantly reduces the number of speckle patterns required and the computational complexity of the resulting speckle matrix.

[0057] Figure 2 The following is a schematic flow chart of a deep learning-based optimized speckle matrix training method 200 according to one embodiment of the present invention. Method 200 begins at step 202, where a training image is divided into multiple image blocks. The training images can be selected from a training image set. Preferably, different training image sets are used for different imaging scenarios. For example, for remote sensing imaging applications, the publicly available NWPU VHR-10 aerospace remote sensing dataset can be used. Similarly, other fields often have corresponding training image sets. It is understood that in scenarios where dedicated training image sets are unavailable, real images of the scene that meet the required resolution can be collected to construct a training image set.

[0058] Image segmentation refers to dividing the original image into multiple small image blocks of a certain size. It is understandable that the smaller the block size, the smaller the amount of speckle and the amount of computation required for each block. At the same time, the likelihood of non-local similarities between blocks increases. For example, in remote sensing scenes, many similar local objects can be found in aerial images, such as airplanes, vehicles, ships, oil tanks, stadiums, farmlands, ports, bridges, etc. These objects or targets all exhibit similar shapes and image features in aerial images. At the same time, it is also understandable that if the block size is too large or too small, these objects or targets may not be properly accommodated in a block or cannot be effectively identified. Therefore, the appropriate block size can be selected based on the actual scenario.

[0059] In step 204, the image blocks are clustered based on similarity. In one example, a K-means clustering algorithm can be used to cluster the image blocks. The K-means clustering algorithm uses Euclidean distance to determine whether images are similar and is a commonly used clustering algorithm.

[0060] In each block cluster, any selected central image block x i Non-local self-similar image patches The weighted formula is as follows:

[0061]

[0062] in Represents similar blocks The weighting coefficient of .

[0063] During clustering, the optimal initial cluster center is first determined. The sum of squared errors between the central image block and similar images is then calculated to determine whether it reaches a minimum. If so, the blocks are clustered together. The more similar the blocks are, the better they are clustered together. Blocks clustered together have a strong similarity structure, meaning that blocks in the same cluster are non-locally similar to each other. This similarity clustering process yields multiple block clusters.

[0064] If it is the minimum value, it is clustered into one category, and the sum of squared errors is defined as follows:

[0065]

[0066] Where x is the image block, M i It is sample C i The above formula to some extent expresses the difference between each image block in the sample and the average value, J C It represents the sum of squared errors and can reflect the differences between all test samples. If the difference between each image block in the sample is smaller, the smaller its value is, the more similar each image block is, and the better it can be clustered together. The image blocks clustered into one category have a strong similar structure.

[0067] In step 206, dictionary learning is performed on each block cluster to obtain multiple class dictionaries. Dictionary learning is a machine learning method commonly used in image reconstruction. The basic idea is to train a dictionary based on the image, and then use the learned dictionary to reconstruct the low-resolution image into a high-resolution image. Preferably, the present invention can use a sparse representation-based method, also known as a sparse dictionary learning method. In the sparse dictionary learning method, the reconstructed image block can be obtained by the following formula:

[0068] x=D·α

[0069] Where x represents the reconstructed image block (assuming dimension d), D represents the class dictionary of the block cluster corresponding to the image block (dimension d×K), and α is the class sparse matrix corresponding to dictionary D (dimension K). According to the sparse dictionary learning method, when multiple image blocks belonging to a cluster are provided as training input, the method can learn the class dictionary D and class sparse matrix α corresponding to the cluster, where the class dictionary D is saved. According to the dictionary learning algorithm, as described below, the class sparse coefficient of each image block can be calculated during reconstruction, and then the reconstructed image block can be obtained by x=D·α.

[0070] Preferably, each cluster can use a different dictionary learning algorithm, such as the PCA algorithm, the MOD algorithm, the K-SVD algorithm, and the like. In other words, the dictionary learning of the present invention can be based on a multi-dictionary model. It can be understood that the characteristics of images of different scenes may vary greatly, and even within a single image of the same scene, the characteristic structures of different parts may also be significantly different. For example, in aerial remote sensing images, man-made objects such as buildings and vehicles usually have regular shapes, while natural landforms usually have irregular shapes, and features such as texture, edges, and structures all show large differences. In this case, by dividing the image into blocks and clustering them, image blocks with similar structural features can be effectively classified into one category, and a dictionary learning algorithm suitable for the characteristics of this type of image can be selected.

[0071] In step 208, an optimized speckle-like matrix is ​​calculated based on the obtained class dictionary. According to the correlation imaging theory, the basic mathematical expression is:

[0072] y=Фx

[0073] Here, y is the observation value of the bucket detector, Φ is the speckle matrix projected by the light source, and x is the image to be reconstructed. It can be understood that the observation value y is a known value that can be obtained through measurement, and the speckle pattern Φ is also known, so the image x can be solved using the above formula. However, while the speckle pattern Φ can theoretically be arbitrary, and random speckle is indeed used in many real-world scenarios, for a specific image, different speckle matrices may affect the quality of the reconstructed image, such as the error from the original image. Conversely, to achieve a certain reconstruction effect, the number of speckles required by different speckle matrices can vary significantly. Therefore, the goal of optimizing the speckle matrix is ​​to achieve the required restoration accuracy with the minimum number of speckles possible.

[0074] The varying effects of different speckle matrices are primarily influenced by image characteristics. In this regard, the clustering of image blocks in the previous step of the present invention has already grouped blocks with substantially identical or sufficiently similar image characteristics, thus providing a good foundation for finding optimized speckle matrices tailored to specific image characteristics. Currently, various speckle optimization methods exist in the prior art, any of which can be employed in the present invention. Similar to the use of different dictionary learning methods for different clusters, different speckle optimization approaches can also be employed to obtain optimized speckle-like matrices. This results in multiple speckle-like matrices optimized for different clusters.

[0075] As an example, by adaptively learning a sub-dictionary D for each class i , thus obtaining the class sparse coefficient α i , then the sparse regularized image super-resolution reconstruction model is introduced into remote sensing correlation imaging:

[0076]

[0077] st X=T k α k +T n α n

[0078]

[0079] Among them, D i =Φ i Ψ i is a class dictionary, α i is the class sparse coefficient, x i =Ψ i α i is the image to be reconstructed, λ is the sparsity regularization parameter, and η is the regularization parameter of the nonlocal constraint term. λ is used to balance the fidelity term and the sparsity regularization term, while η is used to balance the fidelity term and the nonlocal self-similar structure sparsity term.

[0080] Where λ i , η i It can be obtained by the following formula:

[0081]

[0082] Among them, σ i is the α of similar image patches in the i-th cluster i The obtained standard deviation, τ1, τ2 are set constants, and ε is a small integer to avoid λ i , η i Too big.

[0083] Use l1 norm to constrain α iThe sparsity of l2 norm constrains the non-local self-similarity error, so that the remote sensing image is fast x i It can be obtained by coefficient encoding, that is:

[0084] x i =D i α i

[0085] At the same time, the sampling matrix Φ as the speckle of the associated imaging will be optimized by dictionary learning, so as to obtain the speckle with a certain compression ratio required for imaging, which will reduce the number of sampling times.

[0086] Finally, the reconstructed image is obtained through the second-order correlation formula

[0087]

[0088] In the formula, G represents the second-order correlation function, M represents the number of speckle measurements, and y (m) Indicates the value of the detector, <> indicates the arithmetic average, Φ m represents the spatial intensity distribution of the mth speckle.

[0089] Figure 3 3 is a schematic flow chart of a correlation imaging method 300 according to an embodiment of the present invention. The method 300 begins at step 302, where an initial image of the target to be imaged is obtained. In one example, the initial image can be an image obtained by correlation imaging using random speckle, or can be an image of the target obtained in any other manner. For example, in a remote sensing scenario, the initial image can be a historical remote sensing image or a traditional optical image of the target area. It is understood that this initial image may have a lower resolution, clarity, or accuracy than the current requirements, or may be partially unavailable due to the presence of certain obstructions, etc. Optionally, in the case of using a low-resolution initial image, the initial image can be further preprocessed to convert it into an image with the desired high resolution.

[0090] In step 304, the initial image is divided into a plurality of image blocks. This step is combined with the previous Figure 2 The image segmentation step 202 described above is similar, and for example, the image segmentation can be performed according to the image segmentation parameters (such as size, step length, overlap, etc.) used during training.

[0091] In step 306, similarity clustering is performed on the image blocks. This step is combined with the previous Figure 1 The image segmentation step 204 described is similar, for example, the same clustering algorithm may be used for clustering.

[0092] In step 308, a matching speckle-like matrix is ​​determined for each image block based on clustering. Figure 2The training process described above yields speckle-like matrices for different clusters of image blocks, with each speckle-like matrix corresponding to a cluster. During reconstruction, it is necessary to first determine which of the clusters obtained in step 306 matches or is most similar to the clusters formed during training. This determination can be performed using a variety of methods. As a non-limiting example, the mean of the image blocks in each cluster can be calculated and compared with the mean of the image blocks in each cluster during training to determine which cluster in the training model best matches the current cluster. Subsequently, the speckle-like matrix corresponding to the cluster to which each image block belongs is used as the speckle matrix for that image block.

[0093] In step 310, the speckle matrices of each image block are combined into a complete speckle matrix. This step can be completed by, for example, combining the speckle matrices according to the positional relationship of each image block, thereby obtaining a speckle matrix corresponding to the formal correlation imaging to be performed.

[0094] In step 312, correlative imaging is performed using the combined speckle matrix and bucket detectors. This process is consistent with existing correlative imaging procedures: a light source projects speckle onto a target area or object, then the bucket detectors receive light intensity measurements, and a correlative calculation is performed to obtain the target image. Optionally, a corresponding array of bucket detectors can be configured during correlative imaging. Accordingly, during correlative imaging, each combination of the speckle matrix and the bucket detectors can be used to perform correlative imaging to obtain low-resolution image blocks. A high-resolution image is then obtained by weighting the resulting low-resolution image blocks, where the weighting coefficients can utilize those obtained in the clustering process. This approach further reduces computational complexity and computational effort compared to directly correlating the combined full speckle matrix to obtain a high-resolution image.

[0095] Compared with the existing technology, since the speckle matrix obtained through the above process is used, each part of the speckle matrix is ​​optimized. That is, the speckle matrix formed according to the characteristics of the target to be imaged contains as few speckles as possible while ensuring accuracy. Therefore, as a whole, it is possible to obtain a target reconstructed image that meets the resolution and clarity requirements with as few speckles as possible and as low computational complexity as possible.

[0096] Optionally, the above process can be performed repeatedly. For example, the reconstructed image obtained in step 312 can be used as the initial image in the next iteration, thereby continuously improving image clarity and continuously reducing the required speckle amount and computational complexity.

[0097] Figure 4 FIG. 4 is a schematic structural diagram of a correlation imaging system 400 according to an embodiment of the present invention. Figure 4As shown in , the correlation imaging system 400 may include a light source 401 for emitting a speckle matrix to a target to be imaged, a bucket detector 402 for measuring the light intensity value of the speckle matrix after being projected through the target, and a calculation unit 403.

[0098] The computing unit 403 can be communicatively coupled with the light source 401 and the bucket detector 402 to transmit a speckle matrix for projection to the light source 401 and receive measured light intensity values ​​from the bucket detector 402. According to one embodiment of the present invention, the computing unit 403 can be configured to acquire an initial image of the target to be imaged, divide the initial image into multiple image blocks, perform similarity clustering on the image blocks, determine a matching speckle-like matrix for each image block based on the clustering, combine the speckle matrices of each image block into a complete speckle matrix and provide it to the light source 401, and obtain a reconstructed image of the target through a correlation calculation based on the speckle matrix and the measured values ​​of the bucket detector 402. Optionally, the bucket detector 402 can be a bucket detector array consisting of multiple bucket detectors.

[0099] What has been described above includes examples of various aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one skilled in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Thus, the disclosed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method for training an optimized speckle matrix for correlation imaging, characterized in that: The method comprises: Divide the training image into multiple image blocks; Perform similarity clustering on image blocks; Perform dictionary learning on each block cluster to obtain multiple class dictionaries; and Based on the obtained class dictionary, the optimized speckle-like matrix corresponding to each block cluster is calculated. The similarity clustering of image blocks further includes: The K-means clustering algorithm is performed on the divided image blocks to group multiple image blocks with similar image structural features into block clusters.

2. The method according to claim 1, wherein Dictionary learning for each block cluster further includes: For different block clusters, different dictionary learning methods are selected according to the image structure characteristics of the block clusters to perform dictionary learning.

3. The method according to claim 1, wherein Calculating an optimized speckle-like matrix corresponding to each block cluster based on the obtained class dictionary further includes: For different block clusters, different speckle optimization methods are selected according to the image structure characteristics of the block clusters to perform speckle optimization.

4. A correlation imaging method, characterized in that: The method comprises: Acquiring an initial image of the target to be imaged; Dividing the initial image into a plurality of image blocks; Perform similarity clustering on image blocks; Determine a matching speckle-like matrix for each image block based on clustering; Combining the speckle matrices of each image block into a complete speckle matrix; and Use the combined speckle matrix and bucket detector for correlation imaging, The step of determining a matching speckle-like matrix for each image block based on clustering further comprises: By comparing the means, a closest pre-trained block cluster is determined for each block cluster to which the current image block belongs in multiple pre-trained block clusters; The optimized speckle-like matrix corresponding to the closest pre-training block cluster is used as the speckle matrix of the current image block.

5. The method according to claim 4, wherein Performing correlated imaging using the combined speckle matrix and the bucket detector further includes: Using each set of speckle matrices and the corresponding bucket detectors to perform correlation imaging to obtain low-resolution image blocks; and A high-resolution image is obtained by weighting the obtained low-resolution image blocks.

6. The method according to claim 4, wherein Similarity clustering of image blocks further includes: The K-means clustering algorithm is performed on the divided image blocks to group multiple image blocks with similar image structural features into block clusters.

7. The method according to claim 4, wherein Combining the speckle matrix of each image block into a complete speckle matrix further includes: The speckle matrix is ​​combined according to the position of each image block in the initial image, thereby obtaining the complete speckle matrix.

8. A correlation imaging system, characterized in that: The system comprises: a light source for emitting a speckle matrix toward a target to be imaged; a bucket detector, configured to measure the light intensity value of the speckle matrix after projecting through the target; and A computing unit, the computing unit being configured to: Acquiring an initial image of the target to be imaged; Dividing the initial image into a plurality of image blocks; Perform similarity clustering on image blocks; Determine a matching speckle-like matrix for each image block based on clustering; Combining the speckle matrix of each image block into a complete speckle matrix and providing the complete speckle matrix to the light source; and Based on the speckle matrix and the measurement values ​​of the bucket detector array, a reconstructed image of the target is obtained through correlation calculation.

Images