A high-speed imaging method, device and electronic equipment
By using the feature extraction and fusion module of an end-to-end neural network and leveraging the relationship between the mask matrix and the compressed signal, the problem of long computation time in the DeSCI method is solved, and fast and high-precision video reconstruction is achieved.
Patent Information
- Application Number
- CN202210834487.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-14
- Publication Date
- 2025-11-11
- Estimated Expiration
- 2042-07-14
AI Technical Summary
Existing DeSCI methods are computationally time-consuming in compressed high-speed imaging techniques and cannot meet the requirements of real-time imaging.
An end-to-end neural network is used. Through the feature extraction module and the feature fusion module, the relationship between the mask matrix and the acquired compressed signal is used to reconstruct the signal and obtain the inverse mapping to shorten the computation time.
It enables fast and high-precision video reconstruction on mainstream computing devices, meeting the needs of compressed high-speed imaging.
Smart Images
Figure CN115100430B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of high-speed imaging technology, and more specifically, to a high-speed imaging method, apparatus, and electronic device. Background Technology
[0002] Compressed high-speed imaging technology is a novel type of high-speed imaging technology developed in recent years based on compressed sensing theory. In this system, the camera operates at a lower capture rate, and within each camera exposure cycle, the target image is compressed using a higher-frequency modulation device, thus capturing a measurement frame. This frame can then be reconstructed into multiple high-speed frames using reconstruction methods based on compressed sensing theory. For example, in… Figure 1 In the proposed compressed imaging system, the CCD camera operates at a rate of 30 frames per second, while the DMD (Digital Micromirror Device) modulation device operates at a rate of 128 fps per cycle. Thus, we can eventually reconstruct a video rate of up to 3840 fps.
[0003] Meanwhile, theoretical research on compressed sensing shows that when the coding scheme meets certain conditions, the aforementioned high-speed imaging system can reconstruct the original signal with a very high probability, which verifies the feasibility of this novel imaging technology from a theoretical perspective. In practical engineering, the reconstruction effect of a high-speed imaging system naturally depends on the designed imaging method. A common high-speed imaging method is DeSCI (Decompress Snapshot Compressive Imaging). DeSCI reconstructs information by combining nonlocal self-similarity and rank minimization prior information within the ADMM (Alternating Direction Method of Multipliers) optimization framework. Specifically, the ADMM optimization method provides a series of convergent iterative formulas for the solution process; secondly, the image is segmented into multiple patches to utilize nonlocal self-similarity; finally, weighted kernel norm minimization patches provide prior information for the underdetermined problem.
[0004] However, the DeSCI method is computationally time-consuming and cannot meet the requirements of real-time imaging. Summary of the Invention
[0005] To address the issue that the DeSCI method is computationally expensive and cannot meet the requirements of real-time imaging, this application provides a high-speed imaging method:
[0006] According to one aspect of the embodiments of this application, a high-speed imaging method is provided, the method comprising:
[0007] Based on known information, initial information is determined. The known information includes a pre-set mask matrix and the acquired compressed signal. The initial information is obtained based on the relationship between the mask matrix and the acquired compressed signal, which is beneficial for signal reconstruction in subsequent steps.
[0008] The initial information is input into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction. The inverse mapping is learned by the end-to-end neural network under supervised training with a large amount of known information. The end-to-end neural network includes multiple feature extraction modules and feature fusion modules stacked sequentially. The feature extraction module is used to extract features, and the feature fusion module is used to fuse features. The inverse mapping is obtained through multiple iterations based on the feature extraction module and the feature fusion module.
[0009] Based on the inverse mapping, reconstructed information is output; the inverse mapping is a mapping relationship in compressed sensing, and the reconstructed information is obtained by the inverse mapping according to the initial information, which is the final output reconstructed image.
[0010] In some embodiments, in the step of inputting the initial information into an end-to-end neural network including the feature extraction module and the feature fusion module to obtain an inverse mapping for signal reconstruction, the method further includes:
[0011] The initial information is input into the feature extraction module to obtain latent features, which are obtained by the feature extraction module from the initial information.
[0012] Based on the feature fusion module, a two-dimensional feature map is obtained, which is constructed by the feature fusion module according to the latent features;
[0013] Based on the two-dimensional feature map, reconstructed information is obtained to acquire the inverse mapping. The inverse mapping refers to the process of transforming the initial information into the reconstructed information. Based on a large amount of known information, supervised training is performed on a majority of end-to-end neural networks to acquire the inverse mapping.
[0014] In some embodiments, in the step of obtaining a two-dimensional feature map based on the feature fusion module, the method further includes:
[0015] The latent features are input into the feature fusion module to obtain the first image patch. Based on the latent features, n first image patches of the same size and overlapping each other are divided.
[0016] Based on the first graph tile, the graph information of the first graph tile is obtained. The graph information refers to the information formed by constructing the graph structure. The graph structure is not an image, but a data structure, which consists of a series of node information and edge information, and is used to represent the data relationship between multiple objects.
[0017] Based on the graph information of the first graph block, feature fusion is performed to obtain a two-dimensional feature map of the first graph block. The feature fusion refers to matrix multiplication.
[0018] In some embodiments, in the step of obtaining the graph information of each of the first graph blocks based on the first graph blocks, the method further includes:
[0019] Each of the first tiles is searched sequentially to obtain the k most similar other tiles. Since there are n first tiles, each of the n first tiles needs to be searched to obtain the k most similar other tiles.
[0020] The first map tile and the k other map tiles most similar to it are used as nodes to obtain node information. The node information refers to a matrix composed of target nodes and similar nodes. The target node refers to the first map tile, and the similar nodes refer to the k other map tiles most similar to it.
[0021] Based on the node information, edge information is obtained, which is the relationship between the nodes.
[0022] Based on the node and edge information in the graph information, a graph structure is constructed to obtain graph information. When constructing the graph structure, since there are n first graph tiles, n different first graph tiles need to be regarded as the target nodes to obtain n different graph information, which is in matrix form.
[0023] In some embodiments, in the step of sequentially searching each of the first tiles to obtain the k most similar other tiles, the method further includes:
[0024] A search area is obtained with the target image patch as the center. The target image patch is the image patch to be searched. Since there are n first image patches, each first image patch needs to be searched. The size of the search area is related to the size of the first image patch.
[0025] Within the search area, based on the minimum Euclidean distance, k other tiles most similar to the first tile being searched are obtained. Euclidean distance refers to the distance between two tiles, representing the difference between the two tiles. The smaller the distance, the smaller the difference, and vice versa. Based on the minimum Euclidean distance, the tile with the smallest difference from the first tile, i.e., the most similar tile, can be obtained.
[0026] In some embodiments, in the step of obtaining edge information based on the node information, the method further includes:
[0027] Based on the node information, two-dimensional node information is obtained. The two-dimensional node information refers to a two-dimensional matrix of the node information, which is obtained by flattening the node information composed of the target node and similar nodes.
[0028] Based on the two-dimensional node information, two frequency domain information are obtained. The two frequency domain information refer to two matrices obtained by transforming the two-dimensional node information to two different frequency domains through two fully connected layers.
[0029] Based on the frequency domain information, edge information is obtained. The edge information is obtained by multiplying the transposes of two frequency domain information. Multiplying the transposes of two frequency domain information can yield a feature matrix, which includes the edge information.
[0030] In some embodiments, in the step of obtaining reconstructed information based on the two-dimensional feature map, the method further includes:
[0031] Based on the two-dimensional feature map, a corresponding similar map patch is obtained. The two-dimensional feature map is obtained by feature fusion based on the latent features, and then converted back into a corresponding similar map patch based on the two-dimensional feature map. This can obtain more complete information, which is beneficial for information completion and reconstruction. The similar map patch is a map patch with the similar features.
[0032] Based on the similar image patches, reconstruction information is obtained. By placing n similar image patches of the same size and overlapping each other at the original image position, the reconstructed original image, i.e., the reconstruction information, is obtained.
[0033] In some embodiments, in the step of determining initial information based on known information, the method further includes:
[0034] The collected compressed signals from the known information are normalized;
[0035] A coarse estimate is made based on the normalized result of the collected compressed signal from the known information.
[0036] According to another aspect of the embodiments of this application, a high-speed imaging apparatus is provided, the apparatus comprising:
[0037] Input module: used to determine initial information based on known information, including a pre-set mask matrix and the acquired compressed signal. The initial information is obtained based on the relationship between the mask matrix and the acquired compressed signal, which is beneficial for signal reconstruction in subsequent steps.
[0038] Processing module: The initial information is input into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction. The inverse mapping is learned by the end-to-end neural network under supervised training with a large amount of known information. The end-to-end neural network includes multiple feature extraction modules and feature fusion modules stacked sequentially. The feature extraction module is used to extract features, and the feature fusion module is used to fuse features. The inverse mapping is obtained through multiple iterations based on the feature extraction module and the feature fusion module.
[0039] Output module: used to output reconstruction information based on the inverse mapping, wherein the inverse mapping is a mapping relationship in compressed sensing, and the reconstruction information is obtained by the inverse mapping according to the initial information, and the reconstruction information is the final output reconstructed image.
[0040] According to another aspect of the embodiments of this application, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the high-speed imaging methods described above.
[0041] The beneficial effects of this application are as follows: By determining the initial information, the relationship between the mask matrix and the acquired compressed signal can be effectively utilized, which is beneficial for signal reconstruction in subsequent steps; furthermore, by inputting the initial information into an end-to-end neural network including a feature extraction module and a feature fusion module, an inverse mapping for signal reconstruction can be obtained; furthermore, by obtaining the inverse mapping, the reconstruction information to be output can be determined, thereby shortening the computation time and meeting the requirements of compressed high-speed imaging. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A flowchart illustrating a high-speed imaging method according to an embodiment of this application is shown;
[0044] Figure 2 A flowchart illustrating the initialization of known information is shown in another embodiment of this application;
[0045] Figure 3 A schematic diagram of the process for training the inverse mapping is shown in another embodiment of this application;
[0046] Figure 4A flowchart illustrating feature fusion in another embodiment of this application is shown;
[0047] Figure 5 A flowchart illustrating the construction of a graph structure is shown in another embodiment of this application;
[0048] Figure 6 A flowchart illustrating the process of searching for similar tiles is shown in another embodiment of this application;
[0049] Figure 7 A flowchart illustrating the process of obtaining edge information is shown in another embodiment of this application;
[0050] Figure 8 A schematic diagram of the process for obtaining reconstruction information is shown in another embodiment of this application;
[0051] Figure 9 A schematic diagram of the structure of a high-speed imaging device provided in an embodiment of this application is shown;
[0052] Figure 10 A schematic diagram of the physical structure of an electronic device according to an embodiment of this application is shown;
[0053] Figure 11 A schematic diagram of a prior art compressed high-speed imaging system based on aperture coding is shown;
[0054] Figure 12 A schematic diagram of the structure of an end-to-end neural network in another embodiment of this application is shown;
[0055] Figure 13 A schematic diagram of the feature extraction module in another embodiment of this application is shown;
[0056] Figure 14 A schematic diagram of the residual block structure is shown in another embodiment of this application;
[0057] Figure 15 A schematic diagram of the feature fusion module in another embodiment of this application is shown;
[0058] Figure 16 A schematic diagram of the system flow of the feature fusion module in another embodiment of this application is shown. Detailed Implementation
[0059] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.
[0060] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.
[0061] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.
[0062] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.
[0063] Humanity's desire to capture and record rapid phenomena can be traced back to the invention of film photography in the 19th century, which achieved rapid imaging by strictly controlling film exposure time. With the continuous advancement of solid-state electronic imaging sensors based on CCD (charge-coupled device) and CMOS (Complementary Metal-Oxide-Semiconductor) technologies, high-speed cameras have become a common tool for rapid image recording in fields such as industrial intelligent manufacturing, automotive crash testing, basic scientific research, and biomedical observation.
[0064] However, due to the limitations of traditional camera imaging mechanisms—the time required to read data from the sensor array is significant—the temporal resolution of high-speed cameras capturing fast-moving processes is limited to the order of 10⁻⁷. Improving the temporal resolution of imaging systems to achieve fast or even ultrafast imaging to observe events occurring in fleeting moments that are imperceptible to the human eye has always been an important research direction in computational optical imaging.
[0065] Compressed high-speed imaging technology is a new type of high-speed imaging technology developed based on the compressed sensing theory proposed in recent years. In these compressed imaging systems, the camera operates at a lower capture rate, and within each camera exposure cycle, the target image is compressed by a higher-frequency modulation device, thereby capturing a measurement frame. We can then use reconstruction algorithms based on compressed sensing theory to reconstruct multiple high-speed frames from each measurement frame.
[0066] For example, Figure 11 A schematic diagram of a prior art compressed high-speed imaging system based on aperture coding is shown, such as... Figure 11As shown, in the proposed compressed imaging system, the CCD camera operates at a rate of 30 frames per second, while the DMD (Digital Micromirror Device) modulation device operates at a rate of 128 fps per cycle. Thus, we can eventually reconstruct a video rate of up to 3840 fps.
[0067] Theoretical studies on compressed sensing show that when the coding scheme meets certain conditions, the above-mentioned high-speed imaging system can reconstruct the original signal with a very high probability, which verifies the feasibility of this new imaging technology from a theoretical perspective.
[0068] In practical engineering, the reconstruction effect of a high-speed imaging system naturally depends on the designed imaging algorithm. A common compression-type high-speed imaging algorithm is DeSCI (Decompress Snapshot Compressive Imaging). DeSCI reconstructs signals by combining nonlocal self-similarity and rank minimization prior information within the ADMM (Alternating Direction Method of Multipliers) optimization framework. Specifically, the ADMM optimization method provides a series of convergent iterative formulas for the solution process; secondly, the image is segmented into multiple patches to utilize nonlocal self-similarity; finally, weighted kernel norm minimization of similar patches provides prior information for the underdetermined problem.
[0069] However, DeSCI's algorithm is computationally time-consuming and cannot meet the requirements of real-time imaging; weighted kernel norm minimization is a non-blind denoising method that is prone to oversmoothing; the algorithm has low computational efficiency, and mainstream computing devices are insufficient to perform large-scale (high-resolution) high-speed video imaging tasks; all of these greatly limit the practical application scenarios of high-speed imaging systems.
[0070] like Figure 11 The compressed sensing system shown can be summarized by a simple equation:
[0071]
[0072] Where X is the original signal, i.e. the scene being photographed; C is the mask matrix, i.e., a pre-set random matrix; ⊙ represents element-wise multiplication; E is the system noise; B is the compression ratio; and Y is the compressed signal, i.e., the final acquired measurement.
[0073] Therefore, the camera on the hardware side compresses the original signal X into signal Y based on the mask C, while the reconstruction algorithm on the software side aims to recover the original signal X using the known mask C and the acquired signal Y, thereby achieving high-speed video imaging.
[0074] In fact, the acquired signal Y is low-dimensional, while the original signal X is high-dimensional. Therefore, the reconstructed signal constitutes an underdetermined problem, and how to solve this underdetermined problem is the key challenge of compressed high-speed imaging technology.
[0075] Based on the above introduction, a novel reconstruction algorithm is needed that can better utilize priors (sparseness, redundancy, periodicity, low rank) to solve underdetermined problems.
[0076] Therefore, to address the above problems, this application proposes a high-speed imaging method and apparatus that can effectively utilize the inherent low-rank prior information of similar patches to provide the necessary regular prior for the reconstructed signal and improve the overall computational efficiency of the algorithm.
[0077] The following is combined Figures 1-16 This application describes the high-speed imaging method provided.
[0078] Figure 1 A flowchart of a high-speed imaging method according to an embodiment of this application is shown. This method can be executed by a high-speed imaging method partitioning device / electronic device.
[0079] like Figure 1 As shown, the method includes the following steps:
[0080] Step 110: Determine the initial information based on the known information.
[0081] The known information includes a pre-set mask matrix and the acquired compressed signal. The initial information is obtained based on the relationship between the mask matrix and the acquired compressed signal, which is beneficial for signal reconstruction in subsequent steps.
[0082] Step 120: Input the initial information into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction.
[0083] The inverse mapping is learned by an end-to-end neural network under supervised training with a large amount of known information. The end-to-end neural network includes multiple feature extraction modules and feature fusion modules stacked sequentially. The feature extraction module is used to extract features, and the feature fusion module is used to fuse features. The inverse mapping is obtained by repeatedly iterating through the feature extraction module and the feature fusion module.
[0084] Step 130: Output the reconstructed information based on the inverse mapping.
[0085] In this context, the inverse mapping is the mapping relationship in compressed sensing. The reconstructed information is obtained by the inverse mapping based on the initial information, and the reconstructed information is the final output reconstructed image.
[0086] As can be seen from some embodiments of this application, Figure 12A schematic diagram of the structure of an end-to-end neural network in another embodiment of this application is shown, such as... Figure 12 As shown, an end-to-end neural network framework is designed. Under the hardware conditions of mainstream computing power, this framework can still process higher resolution video compression input and output reconstruction results quickly and with high accuracy, meeting the practical needs of compressed high-speed imaging systems.
[0087] According to the definition of compressed sensing above, the measured signal Y can be regarded as a weighted (C) sum of the original signal X, where the values in the encoding matrix C only take the values 0 or 1. Therefore, it is usually a non-energy-normalized image, meaning that some pixels in Y may take only one pixel value from X, while others may take B pixel values from X. Based on this, it is necessary to initialize the Y values.
[0088] In some embodiments, in order to provide an end-to-end neural network with an initial value rich in prior information based on known information, Figure 2 A flowchart illustrating the initialization of known information in another embodiment of this application is shown, such as... Figure 2 As shown, the step of determining the initial information based on known information also includes the following steps:
[0089] Step 210: Normalize the collected compressed signal from the known information.
[0090] The normalization formula is:
[0091]
[0092] Step 220: Perform a rough estimate of the normalized result of the collected compressed signal from the known information.
[0093] The rough estimation formula is:
[0094]
[0095] Where Y is the acquired compressed signal, C is the mask matrix, and ⊙ denotes element-wise multiplication. This indicates element-wise division.
[0096] It can be seen that by introducing a novel graph-based attention mechanism into visual tasks, this method can effectively utilize the inherent low-rank prior information of similar graph patches to provide the necessary regular prior for the reconstructed signal.
[0097] In some embodiments, to improve the overall network's flexible mapping capability and obtain a more accurate inverse mapping, Figure 3 A schematic diagram of the training inverse mapping process in another embodiment of this application is shown, such as... Figure 3As shown, the process of inputting initial information into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction also includes the following steps:
[0098] Step 310: Input the initial information into the feature extraction module to obtain the latent features.
[0099] Among them, latent features are obtained by the feature extraction module from the initial information. Figure 13 A schematic diagram of the feature extraction module in another embodiment of this application is shown. Figure 14 A schematic diagram of the residual block structure in another embodiment of this application is shown, in conjunction with... Figure 13 ,and Figure 14 As shown, the feature extraction module can map local spatial features to higher-dimensional channel features through convolution operations to represent more complex latent image features. This allows the downstream GAM module to further develop and utilize such latent features. In addition, with the stacked operation of the "feature extraction module + feature fusion module" mode, the neural network will gradually become deeper. The feature extraction module can provide deeper feature extraction in subsequent stages. This strategy can greatly improve the overall network's flexible mapping capability.
[0100] Furthermore, the feature extraction module consists of multiple residual blocks (Res, Residual Connection Blocks). These residual blocks are stacked sequentially to form a nonlinear mapping function, which is a commonly used network structure in deep learning. The residual block includes two convolutional layers (Conv, Convolution), a nonlinear activation function, and a residual connection mechanism. The residual block structure can effectively reduce the training difficulty of deep neural networks and improve the overall network's fitting ability.
[0101] The formula for residual block operation is:
[0102] X=Conv(ReLU(Conv(X)))+X(4).
[0103] Step 320: Based on the feature fusion module, obtain a two-dimensional feature map.
[0104] The two-dimensional feature map is constructed by the feature fusion module based on latent features. Figure 15 A schematic diagram of the feature fusion module in another embodiment of this application is shown. Figure 16 A system flow diagram of the feature fusion module in another embodiment of this application is shown, in conjunction with... Figure 15 ,and Figure 16As shown, the feature fusion module is based on the graph attention mechanism, which is the same as the design idea of the existing DeSCI technology. The feature fusion module uses the similarity information between graph patches to regularize the image. However, the feature fusion module adopts a completely different design structure and processing method. The feature fusion module builds a graph structure based on the relationship between graph patches and introduces the graph learning method that has been popular in recent years into the vision task. It should be added here that a graph is not an image. A graph is a data structure that consists of a series of nodes and edges. This data structure can better represent the data relationship between multiple objects.
[0105] Step 330: Based on the two-dimensional feature map, obtain the reconstruction information to obtain the inverse mapping.
[0106] Inverse mapping refers to the process of transforming initial information into reconstructed information. Based on a large amount of known information, supervised training is performed on most end-to-end neural networks to obtain inverse mapping.
[0107] It can be seen that the data processing flow of "feature extraction plus feature fusion" does not require the introduction of known noise and can flexibly adapt and adjust to different scenarios.
[0108] In some embodiments, a more advanced data structure is introduced to implement a more efficient similar tile fusion mechanism. Figure 4 A flowchart illustrating feature fusion in another embodiment of this application is shown, as follows: Figure 4 As shown, the step of obtaining a two-dimensional feature map based on the feature fusion module also includes the following steps:
[0109] Step 410: Input the latent features into the feature fusion module to obtain the first map patch.
[0110] Based on the latent features, n groups of size P are divided. w ×P h The first overlapping tiles are the Unfold process.
[0111] Step 420: Obtain the graph information of the first graph block based on the first graph block.
[0112] Graph information refers to the information formed by constructing a graph structure. A graph structure is not an image, but a data structure that consists of a series of node information and edge information, used to represent the data relationships between multiple objects.
[0113] Step 430: Perform feature fusion based on the graph information of the first image patch to obtain a two-dimensional feature map of the first image patch.
[0114] In this process, after obtaining information on all nodes and edges in the graph, a feature fusion operation is required, which involves matrix multiplication.
[0115] It can be seen that by designing a feature fusion strategy based on graph attention mechanism, this strategy can perform parallel computation efficiently and improve the overall algorithm running efficiency.
[0116] In some embodiments, to better represent the data relationships between multiple objects, the image is regularized using similarity information between tiles. Figure 5 A flowchart illustrating the construction of a graph structure is shown in another embodiment of this application, such as... Figure 5 As shown, the step of obtaining the map information of each first map block based on the first map block also includes the following steps:
[0117] Step 510: Search each first tile sequentially to obtain the k most similar other tiles.
[0118] Since there are n first tiles, it is necessary to search each of the n first tiles to obtain the k other tiles that are most similar to it.
[0119] Step 520: Take the first tile and the k other tiles most similar to it as nodes to obtain node information.
[0120] Here, node information refers to a matrix composed of the target node and similar nodes, written as... The target node refers to the first tile, and the similar nodes refer to the k other tiles that are most similar to it.
[0121] Step 530: Obtain edge information based on node information.
[0122] Among them, edge information refers to the relationship between nodes.
[0123] Step 540: Construct the graph structure based on the node and edge information in the graph information to obtain the graph information.
[0124] In constructing the graph structure, since there are n first graph tiles, it is necessary to treat n different first graph tiles as target nodes to obtain n different graph information. The graph information is in matrix form, written as...
[0125] It can be seen that by utilizing the similarity information between tiles to regularize the image, the data relationships between multiple objects can be better represented.
[0126] In some embodiments, Figure 6 A flowchart illustrating the process of searching for similar tiles in another embodiment of this application is shown, such as... Figure 6 As shown, the step of sequentially searching for each first tile to obtain the k most similar other tiles also includes the following steps:
[0127] Step 610: Obtain the search area centered on the target tile.
[0128] The target tile is the tile to be searched. Since there are n first tiles, each first tile needs to be searched. The size of the search area is related to the size of the first tile, and the size of the search area is R. w ×R h ×R t .
[0129] Step 620: In the search area, based on the minimum Euclidean distance, obtain the k other tiles that are most similar to the first tile in the search.
[0130] Euclidean distance refers to the distance between two tiles, representing the difference between the two tiles. The smaller the distance, the smaller the difference, and vice versa. The tile with the smallest difference from the first tile can be obtained based on the minimum Euclidean distance.
[0131] It can be seen that by using the target tile as the center and the minimum Euclidean distance, the k other tiles that are most similar to the target tile can be easily obtained.
[0132] In some embodiments, to obtain the relationships between nodes, i.e., edge information, we learn them through an attention mechanism. Figure 7 A schematic diagram of the process for obtaining edge information in another embodiment of this application is shown, such as... Figure 7 As shown, the step of obtaining edge information based on node information also includes the following steps:
[0133] Step 710: Obtain two-dimensional node information based on node information.
[0134] Here, two-dimensional node information refers to a two-dimensional matrix of node information, written as... It is obtained by flattening the node information consisting of the target node and similar nodes.
[0135] Step 720: Based on the two-dimensional node information, obtain two frequency domain information.
[0136] Here, the two frequency domain information refers to the two matrices obtained by transforming the two-dimensional node information into two different frequency domains through two fully connected layers, respectively. as well as
[0137] The transformation formula is:
[0138] Q = W q V * K = W k V * (5)
[0139] Step 730: Obtain edge information based on frequency domain information.
[0140] The side information is obtained by multiplying the transposes of two frequency domain information. Multiplying the transposes of two frequency domain information yields the feature matrix, written as...
[0141] The formula for transpose multiplication is:
[0142]
[0143] The feature matrix includes edge information between all nodes, and its calculation formula is as follows:
[0144] V′=AV * (7).
[0145] It can be seen that learning through an attention mechanism can conveniently obtain the relationships between nodes, that is, the information about edges.
[0146] In some embodiments, to make the reconstructed signal more complete and accurate, Figure 8 A schematic diagram of the process for obtaining reconstruction information in another embodiment of this application is shown, such as... Figure 8 As shown, the step of obtaining reconstructed information based on two-dimensional feature maps also includes the following steps:
[0147] Step 810: Obtain the corresponding similar patches based on the two-dimensional feature map.
[0148] The two-dimensional feature map is obtained by feature fusion based on latent features, and then converted into corresponding similar patches based on the two-dimensional feature map. This can obtain more complete information, which is beneficial for information completion and reconstruction. Similar patches are patches with similar features.
[0149] Step 820: Obtain reconstruction information based on similar map tiles.
[0150] The process involves placing n similar images of the same size that overlap with each other into the original image position, which is the fold process in the diagram. The overlapping parts are calculated using a weighted average to obtain the reconstructed original image, i.e., the reconstructed information.
[0151] As can be seen, firstly, based on known information, including a pre-set mask and the collected compressed signal, suitable initial information is constructed. Then, the initial information is input into an end-to-end neural network, which is composed of multiple feature extraction modules and feature fusion modules stacked sequentially.
[0152] The feature extraction module can extract a large number of potential features from the image for further fusion and utilization by downstream modules; the feature fusion module can flexibly combine similar features in the spatiotemporal dimension to achieve the completion and reconstruction of lost signals. As the network layers increase, the overall algorithm can search for deeper and more valuable features and fuse and reconstruct them by continuously iterating between the feature extraction module and the feature fusion module.
[0153] Thus, under supervised training with a large amount of labeled data, the end-to-end neural network can learn and fit the inverse mapping in compressed sensing, and finally output the reconstructed signal based on the inverse mapping.
[0154] The following is combined Figure 9 The high-speed imaging apparatus provided in this application is described below, and the high-speed imaging apparatus described below can be referred to in correspondence with the high-speed imaging method described above.
[0155] Figure 9 A schematic diagram of the structure of the high-speed imaging device provided in this application is shown, as follows: Figure 9 As shown, the high-speed imaging device 900 includes:
[0156] Input module 901: used to determine initial information based on known information, including a pre-set mask matrix and the acquired compressed signal; the initial information is obtained based on the relationship between the mask matrix and the acquired compressed signal, which is beneficial for signal reconstruction in subsequent steps;
[0157] Processing module 902: Inputs initial information into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction. The end-to-end neural network includes multiple feature extraction modules and feature fusion modules stacked sequentially. The feature extraction module is used to extract features, the feature fusion module is used to fuse features, and the inverse mapping is obtained based on the end-to-end neural network. The inverse mapping is a mapping relationship in compressed sensing and is obtained through multiple loops based on the feature extraction module and the feature fusion module.
[0158] Output module 903: Based on the inverse mapping, outputs reconstructed information. The inverse mapping is learned by an end-to-end neural network under supervised training with a large amount of known information. The reconstructed information is obtained by the inverse mapping based on the initial information.
[0159] Figure 10 A schematic diagram of the physical structure of an electronic device is shown, such as... Figure 10As shown, the electronic device 1000 may include a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004. The processor 1001, communication interface 1002, and memory 1003 communicate with each other via the communication bus 1004. The processor 1001 can call logical instructions from the memory 1003 to execute a high-speed imaging method.
[0160] It can be seen that by determining the initial information, the relationship between the mask matrix and the acquired compressed signal can be effectively utilized, which is beneficial for signal reconstruction in subsequent steps. Furthermore, by inputting the initial information into an end-to-end neural network including a feature extraction module and a feature fusion module, an inverse mapping for signal reconstruction can be obtained. Furthermore, by obtaining the inverse mapping, the reconstruction information to be output can be determined, thereby shortening the computation time and meeting the requirements of compressed high-speed imaging.
[0161] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific application considerations.
Claims
1. A high-speed imaging method, characterized in that, The method includes: Based on known information, initial information is determined. The known information includes a pre-set mask matrix and the acquired compressed signal. The initial information is obtained based on the relationship between the mask matrix and the acquired compressed signal, which is beneficial for signal reconstruction in subsequent steps. The initial information is input into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction. The inverse mapping is learned by the end-to-end neural network under supervised training with a large amount of known information. The end-to-end neural network includes multiple feature extraction modules and feature fusion modules stacked sequentially. The feature extraction module is used to extract features, and the feature fusion module is used to fuse features. The inverse mapping is obtained through multiple iterations based on the feature extraction module and the feature fusion module. Based on the inverse mapping, reconstructed information is output; the inverse mapping is a mapping relationship in compressed sensing, and the reconstructed information is obtained by the inverse mapping according to the initial information, which is the final output reconstructed image; In the step of inputting the initial information into an end-to-end neural network including the feature extraction module and the feature fusion module to obtain an inverse mapping for signal reconstruction, the method further includes: The initial information is input into the feature extraction module to obtain latent features, which are obtained by the feature extraction module from the initial information. Based on the feature fusion module, a two-dimensional feature map is obtained, which is constructed by the feature fusion module according to the latent features; Based on the two-dimensional feature map, reconstructed information is obtained to acquire the inverse mapping. The inverse mapping refers to the process of transforming the initial information into the reconstructed information. Based on a large amount of known information, supervised training is performed on a majority of end-to-end neural networks to acquire the inverse mapping. In the feature fusion module, the step of obtaining a two-dimensional feature map further includes: The latent features are input into the feature fusion module to obtain the first image patch. Based on the latent features, n first image patches of the same size and overlapping each other are divided. Based on the first graph tile, the graph information of the first graph tile is obtained. The graph information refers to the information formed by constructing the graph structure. The graph structure is not an image, but a data structure, which consists of a series of node information and edge information, and is used to represent the data relationship between multiple objects. Based on the graph information of the first graph patch, feature fusion is performed to obtain a two-dimensional feature map of the first graph patch. The feature fusion refers to matrix multiplication. In the step of obtaining the graph information of each first graph block based on the first graph block, the method further includes: Each of the first tiles is searched sequentially to obtain the k most similar other tiles. Since there are n first tiles, each of the n first tiles needs to be searched to obtain the k most similar other tiles. The first map tile and the k other map tiles most similar to it are used as nodes to obtain node information. The node information refers to a matrix composed of target nodes and similar nodes. The target node refers to the first map tile, and the similar nodes refer to the k other map tiles most similar to it. Based on the node information, edge information is obtained, which is the relationship between the nodes. Based on the node and edge information in the graph information, a graph structure is constructed to obtain graph information. When constructing the graph structure, since there are n first graph tiles, n different first graph tiles need to be regarded as the target nodes to obtain n different graph information, which is in matrix form.
2. The high-speed imaging method as described in claim 1, characterized in that, In the step of sequentially searching for each of the first tiles to obtain the k most similar other tiles, the method further includes: A search area is obtained with the target image patch as the center. The target image patch is the image patch to be searched. Since there are n first image patches, each first image patch needs to be searched. The size of the search area is related to the size of the first image patch. Within the search area, based on the minimum Euclidean distance, k other tiles most similar to the first tile being searched are obtained. Euclidean distance refers to the distance between two tiles, representing the difference between the two tiles. The smaller the distance, the smaller the difference, and vice versa. Based on the minimum Euclidean distance, the tile with the smallest difference from the first tile, i.e., the most similar tile, can be obtained.
3. The high-speed imaging method as described in claim 1, characterized in that, In the step of obtaining edge information based on the node information, the method further includes: Based on the node information, two-dimensional node information is obtained. The two-dimensional node information refers to a two-dimensional matrix of the node information, which is obtained by flattening the node information composed of the target node and similar nodes. Based on the two-dimensional node information, two frequency domain information are obtained. The two frequency domain information refer to two matrices obtained by transforming the two-dimensional node information to two different frequency domains through two fully connected layers. Based on the frequency domain information, edge information is obtained. The edge information is obtained by multiplying the transposes of two frequency domain information. Multiplying the transposes of two frequency domain information can yield a feature matrix, which includes the edge information.
4. The high-speed imaging method as described in claim 1, characterized in that, In the step of obtaining reconstructed information based on the two-dimensional feature map, the method further includes: Based on the two-dimensional feature map, corresponding similar patches are obtained. The two-dimensional feature map is obtained by feature fusion based on the latent features, and then converted back into corresponding similar patches based on the two-dimensional feature map. This can obtain more complete information, which is beneficial for information completion and reconstruction. The similar patches are patches with similar features. Based on the similar image patches, reconstruction information is obtained. By placing n similar image patches of the same size and overlapping each other at the original image position, the reconstructed original image, i.e., the reconstruction information, is obtained.
5. The high-speed imaging method as described in claim 1, characterized in that, In the step of determining initial information based on known information, the method further includes: The collected compressed signals from the known information are normalized; A coarse estimate is made based on the normalized result of the collected compressed signal from the known information.
6. A high-speed imaging apparatus, said apparatus being applicable to any one of the high-speed imaging methods according to claims 1 to 5, characterized in that, The device includes: Input module: used to determine initial information based on known information, including a pre-set mask matrix and the acquired compressed signal. The initial information is obtained based on the relationship between the mask matrix and the acquired compressed signal, which is beneficial for signal reconstruction in subsequent steps. Processing module: The initial information is input into an end-to-end neural network including a feature extraction module and a feature fusion module to obtain an inverse mapping for signal reconstruction. The inverse mapping is learned by the end-to-end neural network under supervised training with a large amount of known information. The end-to-end neural network includes multiple feature extraction modules and feature fusion modules stacked sequentially. The feature extraction module is used to extract features, and the feature fusion module is used to fuse features. The inverse mapping is obtained through multiple iterations based on the feature extraction module and the feature fusion module. Output module: used to output reconstruction information based on the inverse mapping, wherein the inverse mapping is a mapping relationship in compressed sensing, and the reconstruction information is obtained by the inverse mapping according to the initial information, and the reconstruction information is the final output reconstructed image.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the high-speed imaging method as described in any one of claims 1 to 5.