Adaptive optical real-time compensation method, device, storage medium and system

Abstract

The invention provides a self-adaptive optical real-time compensation method and device, a storage medium and a system, and relates to the technical field of optical communication, and the method comprises the steps: extracting historical wavefront phase diagrams of P historical time steps before a current time step from a wavefront phase diagram; constructing a node feature and an adjacent matrix based on each pixel node in the historical wavefront phase diagram to obtain diagram data of the historical wavefront phase diagram, inputting the diagram data into a pre-trained diagram multi-head attention model, and outputting Q future time steps to predict a wavefront phase; and reconstructing the predicted wavefront phase into a phase distribution form suitable for correction of the phase modulator, mapping the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator, and inputting the control voltage signal into the phase modulator, so that the phase modulator adjusts a phase structure based on the control voltage signal. And performing real-time phase compensation on the actual wavefront phases of the Q future time steps. According to the method, the problem that an existing wavefront phase prediction and turbulence compensation method has time delay can be solved.

Description

Technical Field

[0001] The present invention relates to the field of optical communication technology, and in particular to an adaptive optical real-time compensation method, device, storage medium and system. Background Art

[0002] Atmospheric laser communication is regarded as a core technology to break through the bottleneck of wireless access due to its high bandwidth, strong anti-interference and high security. It shows great application potential in satellite communications, ground long-distance transmission and other fields. However, its signal transmission is highly dependent on the stability of the atmospheric channel and is easily interfered by atmospheric turbulence. Atmospheric turbulence can cause wavefront phase distortion and light intensity flickering effects of the light beam, seriously reducing the reliability and transmission quality of the communication link. Therefore, how to compensate for the wavefront phase distortion caused by turbulence in real time and accurately has become a key challenge to improving the performance of atmospheric laser communications. Traditional wavefront adaptive optics technology relies on high-precision wavefront sensors and complex optical paths, with high hardware costs and difficult to deploy on a large scale. Therefore, non-wavefront adaptive optics technology has received attention.

[0003] Currently, traditional wavefront-free adaptive optics technologies include: using iterative optimization algorithms (such as stochastic parallel gradient descent) or neural network-based phase prediction models to directly recover wavefront distortion from light intensity information.

[0004] However, the phase information of wavefront damage caused by atmospheric turbulence has spatial and temporal correlations. The temporal and spatial characteristics of atmospheric turbulence are physically coupled. Therefore, they cannot be separated and predicted independently during the turbulence prediction process. The wavefront phase prediction established by an independent feature or the prediction generator based on a neural network structure will cause turbulence compensation to have a time delay. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide an adaptive optical real-time compensation method, device, storage medium and system to eliminate or improve one or more defects existing in the prior art, and solve the problem of time delay in the existing wavefront phase prediction and turbulence compensation methods.

[0006] One aspect of the present invention provides an adaptive optical real-time compensation method, the method comprising the following steps: Extract the historical wavefront phase diagram corresponding to P historical time steps before the current time step from the currently received wavefront phase diagram, each historical wavefront phase diagram corresponds to a historical time step vector; wherein P is an integer greater than 1; Based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase map and the connection relationship between the pixel nodes, the node features and the adjacency matrix are constructed to obtain the graph data of the historical wavefront phase map; The graph data and the historical time step vector are input into the pre-trained graph multi-head attention model, and the predicted wavefront phase of Q future time steps is output; the model includes a first fully connected layer, an encoding layer, a conversion attention layer, a decoding layer, and a second fully connected layer connected in series; the first fully connected layer is used to convert the graph data into a spatial feature embedding, and fuse the time feature embedding of the historical time step vector and the future time step vector to obtain the spatiotemporal feature embedding; the encoding layer is used to encode the spatiotemporal feature embedding and output the historical features; the conversion attention layer is used to map the historical features to the future features corresponding to the future time step; the decoding layer is used to decode the future features and output the preliminary prediction results; the second fully connected layer is used to optimize the preliminary prediction results and output the predicted wavefront phase; wherein Q is an integer greater than 1; The predicted wavefront phase is reconstructed into a phase distribution form suitable for phase modulator correction, and the reconstructed predicted wavefront phase is mapped to a control voltage signal of the phase modulator and then input into the phase modulator, so that the phase modulator adjusts the phase structure based on the control voltage signal and performs real-time phase compensation on the actual wavefront phase of Q future time steps.

[0007] In some embodiments of the present invention, the historical time step vector is a joint time vector obtained by concatenating the week code and the time code; converting the graph data into a spatial feature embedding, and fusing the time feature embedding of the historical time step vector and the future time step vector to obtain a spatiotemporal feature embedding, including: The graph data is calculated using a biased second-order random walk algorithm and converted into a spatial feature embedding matrix; Mapping the joint time vector into a time feature embedding vector; The spatial feature embedding matrix and the temporal feature embedding vector are aligned node by node and fused to obtain the spatiotemporal feature embedding.

[0008] In some embodiments of the present invention, the encoding layer and the decoding layer respectively include L serially connected self-attention modules; each self-attention module includes a spatial attention submodule, a temporal attention submodule and a gated fusion submodule; the spatial attention submodule includes K parallel spatial attention heads, and the temporal attention submodule includes K parallel temporal attention heads; wherein K is an integer greater than 1; Each spatial attention head is used to take the node weighted sum output by the spatial attention submodule in the previous self-attention module as the hidden state of the pixel node, embed the connection of each pixel node hidden state with the corresponding spatiotemporal feature, and calculate the similarity between each pixel node and other pixel nodes through the scaled dot product algorithm; normalize the similarity to obtain the pixel node attention score corresponding to each pixel node; calculate the node weighted sum corresponding to each pixel node based on the pixel node attention score and the pixel node hidden state; merge the outputs of K spatial attention heads to obtain the output result of the spatial attention submodule; Each temporal attention head is used to take the node weighted sum output by the temporal attention submodule in the previous self-attention module as the hidden state of the pixel node, embed the connection of each pixel node hidden state with the corresponding spatiotemporal feature, and calculate the time step similarity between the time step corresponding to each pixel node and other time steps through the scaled dot product algorithm; normalize the time step similarity to obtain the time step attention score; calculate the node weighted sum corresponding to each pixel node based on the time step attention score and the pixel node hidden state; merge the outputs of K temporal attention heads to obtain the output result of the temporal attention submodule; The gated fusion submodule is used to fuse the output results of the spatial attention submodule and the output results of the temporal attention submodule through the gated fusion mechanism and output them.

[0009] In some embodiments of the present invention, the conversion attention layer includes a first branch, a second branch, and an output layer; The first branch is used to perform nonlinear projection on the temporal feature embedding to obtain a nonlinear projection result; the nonlinear projection result is normalized after an inner product operation to obtain a history-future time step attention score as the output result of the first branch; the history-future time step attention score is used to indicate the correlation between the future time step and the history time step; The second branch is used to perform nonlinear projection on the historical features to obtain the output result of the second branch; The output layer is used to perform inner product operation on the output results of the first branch and the output results of the second branch, and matrix concatenate the results of the inner product operation to obtain future features.

[0010] In some embodiments of the present invention, the first branch includes a first nonlinear projection layer, a first inner product operation layer, and a normalization layer in sequence; The second branch includes a second nonlinear projection layer; The output layer includes a second inner product operation layer and a matrix concatenation layer in sequence.

[0011] In some embodiments of the present invention, the training process of the graph multi-head attention model includes: The sample training graph data and sample time series information are input into the initial graph multi-head attention model, and the prediction result is output; the initial graph multi-head attention model has the same model structure as the graph multi-head attention model; the sample training graph data and sample time series information are extracted from the wavefront phase image with time series correlation; The prediction results are compared with the real data corresponding to the sample training graph data and the sample time series information, and the prediction results are evaluated by a preset loss function to obtain a loss result; the preset loss function includes a mean absolute error loss function, a root mean square error loss function, and a mean absolute percentage error loss function; Based on the loss results, the parameters of higher layers in the initial graph multi-head attention model are iteratively updated and optimized through the back-propagation algorithm and gradient descent algorithm to obtain the graph multi-head attention model.

[0012] In some embodiments of the present invention, after reconstructing the predicted wavefront phase into a phase distribution form suitable for phase modulator correction, and mapping the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator and inputting it into the phase modulator, the method further includes: Compare the compensated wavefront phase with the predicted wavefront phase to obtain an error result; A new control voltage signal is generated based on the error result and sent to the phase modulator, so that the phase modulator adjusts the phase structure based on the new control voltage signal.

[0013] Another aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the adaptive optical real-time compensation method as described above.

[0014] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the adaptive optics real-time compensation method as described above are implemented.

[0015] Another aspect of the present invention provides an adaptive optical real-time compensation system, the system comprising: A wavefront phase receiving module is used to receive a wavefront phase image in real time and send the wavefront phase image to a historical phase extraction module; The historical phase extraction module is used to extract the historical wavefront phase diagram corresponding to P historical time steps before the current time step from the currently received wavefront phase diagram, each historical wavefront phase diagram corresponds to a historical time step vector; and send the historical wavefront phase diagram to the graph data construction module; A graph data construction module is used to construct node features and an adjacency matrix based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase map and the connection relationship between the pixel nodes, so as to obtain the graph data of the historical wavefront phase map; and send the graph data to the wavefront phase prediction module; The wavefront phase prediction module is used to input the graph data and the historical time step vector into the pre-trained graph multi-head attention model, and output the predicted wavefront phase of Q future time steps; the predicted wavefront phase is sent to the voltage signal generation module; wherein the model includes a first fully connected layer, an encoding layer, a conversion attention layer, a decoding layer, and a second fully connected layer connected in series; the first fully connected layer is used to convert the graph data into a spatial feature embedding, and fuse the time feature embedding of the historical time step vector and the future time step vector to obtain a spatiotemporal feature embedding; the encoding layer is used to encode the spatiotemporal feature embedding and output the historical features; the conversion attention layer is used to map the historical features to the future features corresponding to the future time step; the decoding layer is used to decode the future features and output the preliminary prediction results; the second fully connected layer is used to optimize the preliminary prediction results and output the predicted wavefront phase; A voltage signal generating module is used to reconstruct the predicted wavefront phase into a phase distribution form suitable for phase modulator correction, and map the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator and then input it into the phase modulator; The phase modulator is used to adjust the phase structure based on the control voltage signal to perform real-time phase compensation on the actual wavefront phase of Q future time steps.

[0016] The adaptive optical real-time compensation method and device of the present invention can solve the problem of time delay in existing wavefront phase prediction and turbulence compensation methods. It accurately predicts the time-space related turbulence model through the graph multi-head attention model, jointly estimates the wavefront phase data in the time dimension and space dimension, combines the hidden state of the historical wavefront phase with the time-space feature embedding, and significantly improves the real-time and accuracy of wavefront phase compensation.

[0017] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0018] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings: Figure 1 This is a flow chart of an adaptive optical real-time compensation method provided by one embodiment of the present invention.

[0020] Figure 2 A schematic diagram of model training provided by an embodiment of the present invention.

[0021] Figure 3 A schematic diagram of a model structure provided in one embodiment of the present invention.

[0022] Figure 4 A schematic diagram of the structure of the conversion attention layer provided for one embodiment of the present invention.

[0023] Figure 5 A schematic diagram of the principle of an adaptive optical real-time compensation method provided by an embodiment of the present invention.

[0024] Figure 6 A block diagram of an adaptive optical real-time compensation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0026] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0027] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0028] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0030] The adaptive optical real-time compensation method provided by the present application is introduced in detail below.

[0031] In some embodiments of the present invention, the executor of the adaptive optical real-time compensation method provided in the present application is an electronic device, which may be a terminal such as a computer, a mobile phone, a tablet computer, a camera, or may also be a server, a digital signal processor (DSP) or a field programmable gate array (FPGA), etc. This embodiment does not limit the implementation method of the electronic device.

[0032] This embodiment provides an adaptive optical real-time compensation method, such as Figure 1 As shown, the method at least includes steps S101 to S104: Step S101, extracting historical wavefront phase maps corresponding to P historical time steps before the current time step from the currently received wavefront phase map, each historical wavefront phase map corresponds to a historical time step vector, where P is an integer greater than 1.

[0033] In some embodiments of the present invention, the real-time wavefront phase image is received in real time from the atmospheric turbulence channel by a charge-coupled device (CCD) combined with a non-wavefront detection phase recovery algorithm and then sent to an electronic device.

[0034] Among them, the phase recovery algorithm without wavefront detection includes but is not limited to the algorithm based on Fourier transform (Fourier Transform Method, FTM), the algorithm based on the stepwise optimization of phase information (Iterative Phase Retrieval Algorith) or the algorithm based on the maximum likelihood estimation principle (Maximum Likelihood Method, MLM). This embodiment does not limit the implementation method of the phase recovery algorithm without wavefront detection.

[0035] In actual implementation, the wavefront phase can also be received in real time through devices such as fiber optic sensors, high-resolution cameras, or MEMS scanning mirrors. This embodiment does not limit the implementation method of receiving the wavefront phase.

[0036] In some embodiments of the present invention, the time intervals between adjacent wavefront phase diagrams are the same, and the historical wavefront phase diagrams corresponding to P historical time steps are a group of wavefront phase diagrams with time series correlation. Each historical wavefront phase diagram corresponds to a historical time step vector. The historical time step vector is a joint time vector obtained by splicing the week code and the time code.

[0037] For example, if there are T time steps in a day, encode the day of the week and time of each time step as the day code and time coding , then the joint time vector is expressed as .

[0038] Step S102, constructing node features and an adjacency matrix based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase map and the connection relationship between the pixel nodes, and obtaining graph data of the historical wavefront phase map.

[0039] In some embodiments of the present invention, based on the connection relationship between the pixels in the historical wavefront phase map, the historical wavefront phase map is represented as a graph G = (V, D, A). Where V is a set of pixel nodes, N = |V| represents the total number of pixel nodes; D is an edge set, representing the adjacent connections between pixel nodes; Represents the adjacency matrix, including the connectivity between each pixel node.

[0040] At the same time, the light intensity attribute of each pixel node is extracted as the node feature of the pixel node and mapped into the graph signal in the graph G. At this time, the time step The wavefront phase graph data is represented as a graph signal on the graph G Correspondingly, the graph data of the historical wavefront phase diagram can be expressed as , where P represents the number of historical time steps, C represents the dimension of pixel features, and N represents the number of pixels.

[0041] Step S103: Input the graph data and the historical time step vector into the pre-trained graph multi-head attention model, and output the predicted wavefront phases of Q future time steps, where Q is an integer greater than 1.

[0042] In some embodiments of the present invention, the wavefront phase information of Q future time steps of the atmospheric turbulence channel is predicted by using the prediction capability of the pre-trained graph multi-head attention model for spatiotemporal sequences, and then the wavefront phase is reconstructed by adjusting the phase modulator, thereby achieving the purpose of real-time compensation of the wavefront phase.

[0043] The graph multi-head attention model includes a first fully connected layer, an encoding layer, a conversion attention layer, a decoding layer, and a second fully connected layer connected in series. The training process of the graph multi-head attention model includes at least steps S1031 to S1033: Step S1031, input the sample training graph data and sample time series information into the initial graph multi-head attention model, and output the prediction result. The initial graph multi-head attention model has the same model structure as the graph multi-head attention model.

[0044] like Figure 2As shown, the sample training graph data and the sample time series information are extracted from the wavefront phase graph with timing correlation. Specifically, in some embodiments of the present invention, multiple groups of wavefront phase graphs with timing correlation in a continuous time period are obtained by combining a charge coupled device with a phase recovery technique without wavefront detection, and 70% of the data in the multiple groups of wavefront phase graphs are used for training, 10% of the data are used for verification, and 20% of the data are used for testing.

[0045] In addition, in actual implementation, if the original image has too many pixels, the image pixel data can be preprocessed, that is, the dimension can be reduced to a relatively small order of magnitude, such as 64×64. After reducing the spatial information of multiple sets of wavefront phase images, each phase image pixel is flattened into a one-dimensional vector, and all image data is aggregated into a matrix. Set the time interval information between phase images and extract the sample time series information.

[0046] In order to construct the adjacency matrix, each pixel is regarded as a node, and the adjacent relationship is used as an edge to form a graph structure. Assuming that each pixel is only related to its adjacent points, the edge value is set to 1, and the edge value between other points is set to 0, the adjacency matrix is ​​obtained. At the same time, the light intensity information of each pixel is extracted as the node feature and a biased second-order random walk algorithm (p=2,q=1) is used to generate a spatial embedding matrix, which is aggregated into sample training graph data.

[0047] The sample training graph data and sample time series information are input into the initial graph multi-head attention model to construct spatiotemporal feature embedding, and the initial graph multi-head attention model outputs the prediction results.

[0048] Step S1032, compare the prediction result with the real data corresponding to the sample training graph data and the sample time series information, evaluate the prediction result through a preset loss function, and obtain a loss result.

[0049] Among them, the preset loss function includes the mean absolute error loss function, the root mean square error loss function or the mean absolute percentage error loss function.

[0050] Step S1033, based on the loss result, the parameters of the higher layers in the initial graph multi-head attention model are iteratively updated and optimized through the back propagation algorithm and the gradient descent algorithm to obtain the graph multi-head attention model.

[0051] In some embodiments of the present invention, the model is trained by an Adaptive Moment Estimation (Adam) optimizer, and the initial learning rate is set to 0.001.

[0052] In the trained graph multi-head attention model, the first fully connected layer is used to convert graph data into spatial feature embedding. At the same time, since spatial feature embedding only provides static representation and cannot represent the dynamic correlation of pixel nodes, based on this, Figure 3 As shown, the present invention further proposes time feature embedding, which converts P historical time step vectors and Q future time step vectors into a vector through the first fully connected layer , embed P historical time steps and Q future time steps into the time feature, and get the time feature embedding, expressed as ,in, , to provide the dynamic correlation of spatial pixel nodes. Then, the spatial feature embedding and temporal feature embedding are fused as spatiotemporal feature embedding to obtain the temporally correlated pixel node representation . Wherein, P and Q are integers greater than 1, including 11, 12 and 13, etc., and the values ​​of P and Q can be adjusted according to actual conditions. This embodiment does not limit the values ​​of P and Q; N = |V| represents the total number of pixel nodes; D is an edge set, representing the adjacent connections between pixel nodes.

[0053] Specifically, the first fully connected layer is used to convert the graph data into spatial feature embedding, and fuse the time feature embedding of the historical time step vector and the future time step vector to obtain the spatiotemporal feature embedding. The process of converting the graph data into spatial feature embedding, and fusing the time feature embedding of the historical time step vector and the future time step vector to obtain the spatiotemporal feature embedding includes: calculating the graph data through a biased second-order random walk algorithm and converting it into a spatial feature embedding matrix; mapping the joint time vector into a temporal feature embedding vector; aligning the spatial feature embedding matrix and the temporal feature embedding vector node by node, and fusing them to obtain the spatiotemporal feature embedding.

[0054] After the historical wavefront phase map is transformed into graph data through the first fully connected layer and historical time step vectors into spatiotemporal feature embeddings After that, the spatiotemporal features are embedded into the input coding layer for encoding. Specifically, the coding layer is used to encode each pixel node data of the historical wavefront phase map as grid node information into the model, encode the spatiotemporal feature embedding, and output the historical features.

[0055] In some embodiments of the present invention, Figure 3 As shown, the encoding layer includes L series-connected self-attention blocks. Wherein, L is an integer greater than 1, including 3, 4 or 5. In actual implementation, the value of L can be adjusted according to actual needs. This embodiment does not limit the value of L.

[0056] Each self-attention block includes a spatial attention submodule, a temporal attention submodule, and a gated fusion submodule. The spatial attention submodule is used to model the dynamic spatial correlation of the historical wavefront phase map; the temporal attention submodule is used to model the nonlinear temporal data correlation; and the gated fusion submodule is used to fuse spatial and temporal representations.

[0057] In the L series of self-attention blocks, The input of the self-attention block is the spatiotemporal feature embedding , the input of the 2nd to Lth self-attention blocks is the output of the previous self-attention block.

[0058] In the 2nd to Lth self-attention blocks, Take the self-attention block as an example, The input of a self-attention block is represented as , that is, -1 output of the self-attention block. In the self-attention block, the output of the spatial attention submodule is expressed as , the output of the temporal attention submodule is expressed as Among them, the time step Vertex The hidden states of and , for A collection of for A collection of .

[0059] Output of the spatial attention submodule , the output of the temporal attention submodule Afterwards, the output of the spatial attention submodule is merged into and the output of the temporal attention submodule Fusion, get The output of the self-attention block .

[0060] Since the influence between different pixel nodes is highly dynamic, the present invention adaptively captures the correlation between different pixel nodes through the spatial attention submodule. The key idea is to dynamically assign different weights to different pixel nodes at different time steps.

[0061] First Take the self-attention block as an example. In the self-attention block, for any pixel node, a weighted sum is calculated from all pixel nodes as the hidden state of the pixel node. The set of hidden states of all pixel nodes is the output of the spatial attention submodule. .

[0062] Specifically, the time step Pixel nodes For example, its weighted sum can be calculated by the following formula: In the formula, Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the set of all pixel nodes; It is a pixel node For pixel nodes The sum of the attention scores is equal to 1, expressed as .

[0063] At any time step, the current wavefront phase may affect the correlation between pixel nodes. Based on this, in this embodiment, the phase feature and the graph structure are considered to learn the attention score.

[0064] Specifically, the hidden state of the pixel node is connected with the corresponding spatiotemporal feature embedding, and the scaled dot product method is used to calculate the similarity between pixel nodes, which is expressed by the following formula: In the formula, Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the time step Mid Pixel Node The corresponding spatiotemporal feature embedding of Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the time step Mid Pixel Node The corresponding spatiotemporal feature embedding of represents the concatenation operation; <•,•> represents the inner product operator, and 2D represents The corresponding dimension.

[0065] After that, the pixel nodes are obtained through softmax normalization For pixel nodes The attention score can be expressed as follows: In the formula, , and Represents a pixel node; Represents the set of all pixel nodes; Represents a pixel node With pixel nodes The similarity between Represents a pixel node With pixel nodes The similarity between .

[0066] Getting pixel nodes Attention score Then, through the attention score and , calculated And update the pixel node After calculating the hidden states of all pixel nodes, we get The output of the spatial attention submodule of the self-attention block .

[0067] In some embodiments of the present invention, in order to stabilize the learning process, the spatial attention submodule is expanded to a multi-head mechanism, and each spatial attention submodule includes K parallel spatial attention heads, and different spatial attention heads have different learnable projections. Wherein, K is an integer greater than 1, including 8, 9 or 10. In actual implementation, the value of K can be adjusted according to actual needs, and this embodiment does not limit the value of K.

[0068] Specifically, each spatial attention head is used to take the weighted sum of the nodes output by the spatial attention submodule in the previous self-attention module as the hidden state of the pixel node, embed the connection of each pixel node hidden state with the corresponding spatiotemporal feature, and calculate the similarity between each pixel node and other pixel nodes through the scaled dot product algorithm; normalize the similarity to obtain the pixel node attention score corresponding to each pixel node; based on the pixel node attention score and the pixel node hidden state, calculate the weighted sum of the nodes corresponding to each pixel node; merge the outputs of K spatial attention heads to obtain the output result of the spatial attention submodule.

[0069] First Taking the spatial attention head as an example, the hidden state of each pixel node is embedded and connected with the corresponding spatiotemporal feature. The similarity between each pixel node and other pixel nodes is calculated by the scaled dot product algorithm, which can be expressed by the following formula: In the formula, Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the time step Mid Pixel Node The corresponding spatiotemporal feature embedding of Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the time step Mid Pixel Node The corresponding spatiotemporal feature embedding of represents the concatenation operation; <•,•> represents the inner product operator, and d represents The corresponding dimension; Indicates The nonlinear mapping of the spatial attention heads is expressed as .

[0070] In the The similarity is normalized by a spatial attention head to obtain the node attention score corresponding to each node, which can be expressed as follows: In the formula, , and Represents a pixel node; Represents the set of all pixel nodes; Indicates Pixel nodes in the spatial attention head With pixel nodes The similarity between Indicates Pixel nodes in the spatial attention head With pixel nodes The similarity between .

[0071] The outputs of K spatial attention heads are merged, which can be expressed as follows: In the formula, Indicates The nonlinear mapping of the spatial attention heads is expressed as ; Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of .

[0072] Correspondingly, each temporal attention head is used to take the node weighted sum output by the temporal attention submodule in the previous self-attention module as the pixel node hidden state, embed each pixel node hidden state with the corresponding spatiotemporal feature, and calculate the time step similarity between the time step corresponding to each pixel node and other time steps through the scaled dot product algorithm; normalize the time step similarity to obtain the time step attention score; calculate the node weighted sum corresponding to each pixel node based on the time step attention score and the pixel node hidden state; merge the outputs of K temporal attention heads to obtain the output result of the temporal attention submodule.

[0073] First Taking the spatial attention head as an example, the hidden state of each pixel node is embedded and connected with the corresponding spatiotemporal feature. The similarity between the time step corresponding to each pixel node and other pixel nodes is calculated by the scaled dot product algorithm, which can be expressed by the following formula: In the formula, Represents the time step and The similarity between Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the time step Mid Pixel Node The corresponding spatiotemporal feature embedding of Represents the time step Mid Pixel Node The hidden state of -1 time step of the self-attention block output Mid Pixel Node The weighted sum of Represents the time step Mid Pixel Node The corresponding spatiotemporal feature embedding of represents the concatenation operation; <•,•> represents the inner product operator, and d represents The corresponding dimension; and Indicates Nonlinear mapping of spatial attention heads.

[0074] In the The similarity is normalized by a spatial attention head to obtain the node attention score corresponding to each node, which can be expressed as follows: In the formula, , and represents the time step; represents the set of all time steps; Indicates time step in the temporal attention head and The attention score between Indicates time step in the temporal attention head and The attention score between .

[0075] The outputs of K time attention heads are merged, which can be expressed as follows: In the formula, Indicates Nonlinear mapping of spatial attention heads; Represents the time step Mid Pixel Node The hidden state of -1 time step of the output of the temporal attention submodule of the self-attention block Mid Pixel Node The hidden state; Represents the time step and time step The attention score between .

[0076] In the The output of the spatial attention submodule in the self-attention block , the output of the temporal attention submodule Afterwards, the gated fusion submodule is used to fuse the output results of the spatial attention submodule and the temporal attention submodule through the gated fusion mechanism and output them to obtain the first The output of the self-attention block. The output of the final encoding layer is expressed as .

[0077] In order to alleviate the error propagation effect between different prediction time steps in a long time range, in some embodiments of the present invention, a conversion attention layer is added between the encoding layer and the decoding layer to simulate the direct relationship between each future time step and each historical time step, converting the wavefront phase characteristics of the encoded output Transformed into features for future time steps , as the input of the decoding layer. Specifically, the conversion attention layer is used to model the direct relationship between historical time steps and future time steps, mapping historical features to future features corresponding to future time steps.

[0078] In some embodiments of the present invention, the conversion attention layer includes a first branch, a second branch and an output layer.

[0079] Among them, the first branch is used to perform nonlinear projection on the temporal feature embedding to obtain a nonlinear projection result; the nonlinear projection result is normalized after an inner product operation to obtain a historical-future time step attention score as the output result of the first branch; the historical-future time step attention score is used to indicate the correlation between the future time step and the historical time step.

[0080] like Figure 4 As shown, the first branch includes a first nonlinear projection layer, a first inner product operation layer and a normalization layer in sequence; the implementation of a nonlinear projection layer and a first inner product operation layer can be expressed by the following formula: In the formula, Indicates that in the conversion attention layer, The future time step corresponding to the attention head Pixel Node The spatiotemporal features of Perform nonlinear projection; Indicates that in the conversion attention layer, The historical time step corresponding to the attention head Pixel Node The spatiotemporal features of Perform nonlinear projection; <•,•> represents the inner product operator, and d represents The corresponding dimension.

[0081] The implementation of the normalization layer can be expressed as follows: In the formula, represents the future time step; , , and represents the historical time step.

[0082] The second branch is used to perform nonlinear projection on historical features to obtain the output results of the second branch; the output layer is used to perform inner product operations on the output results of the first branch and the output results of the second branch, and to perform matrix concatenation on the results of the inner product operations to obtain future features. Figure 4 As shown, the second branch includes a second nonlinear projection layer; the output layer includes a second inner product operation layer and a matrix concatenation layer in sequence. The implementation of the second branch and the output layer can be expressed by the following formula: In the formula, Represented by a nonlinear projection function Project the output historical features; represents the future time step; , , and represents the historical time step; Indicates attention heads; K represents the number of attention heads.

[0083] Output future features in the conversion attention layer Afterwards, the future features As the input of the decoding layer, the decoding layer is used to decode future features and output preliminary prediction results. Stack L spatial-temporal self-attention blocks on top and output them as preliminary prediction results Finally, the preliminary prediction results are input into the second fully connected layer, which is used to optimize the preliminary prediction results, output the predicted wavefront phase, and generate the predicted wavefront phase of Q future time steps. .

[0084] Step S104, reconstruct the predicted wavefront phase into a phase distribution form suitable for phase modulator correction, and map the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator and input it into the phase modulator, so that the phase modulator adjusts the phase structure based on the control voltage signal and performs real-time phase compensation on the actual wavefront phase of Q future time steps.

[0085] refer to Figure 5, taking the phase modulator as a deformable mirror as an example, after obtaining the predicted wavefront phase, the predicted wavefront phase is input into the controller, and the controller reconstructs the predicted wavefront phase under the conditions of integrating the spatial correlation and temporal correlation of the wavefront phase, and generates a control voltage based on a preset control algorithm and inputs it into the deformable mirror. Among them, the controller includes devices such as a computer, a digital signal processor (DSP) or a field programmable gate array (FPGA), and the preset control algorithm includes linear control (such as matrix multiplication) or nonlinear control (such as an optimization algorithm), etc. This embodiment does not limit the implementation method of the controller and the implementation method of the control algorithm. The deformable mirror works under the control of the control voltage output by the controller. By changing the parameters such as the optical path and refractive index of the wavefront transmission, the phase structure is adjusted to achieve phase compensation. The total delay that can be reduced by compensation is , which improves the real-time performance of phase compensation.

[0086] In addition, in order to improve the accuracy of phase compensation, in some embodiments of the present invention, after the phase modulator compensation is completed, the compensated wavefront phase is compared with the predicted wavefront phase, and accurate phase compensation is achieved by using feedback adjustment.

[0087] Specifically, the predicted wavefront phase is reconstructed into a phase distribution form suitable for phase modulator correction, and the reconstructed predicted wavefront phase is mapped into a control voltage signal of the phase modulator and then input into the phase modulator, and it also includes: comparing the compensated wavefront phase with the predicted wavefront phase to obtain an error result; generating a new control voltage signal based on the error result and sending it to the phase modulator, so that the phase modulator adjusts the phase structure based on the new control voltage signal.

[0088] For example: Reference Figure 5 , re-input into the CCD sensor, combined with the wavefront detection-free phase recovery technology, the phase information is transmitted back to the controller, a new control voltage signal is generated and sent to the deformable mirror, so that the deformable mirror adjusts the phase structure based on the new control voltage signal.

[0089] In summary, the adaptive optical real-time compensation method provided by the present invention extracts the historical wavefront phase map corresponding to P historical time steps before the current time step from the currently received wavefront phase map, and each historical wavefront phase map corresponds to a historical time step vector; based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase map and the connection relationship between the pixel nodes, the node features and the adjacency matrix are constructed to obtain the graph data of the historical wavefront phase map; the graph data and the historical time step vector are input into the pre-trained graph multi-head attention model, and the predicted wavefront phase of Q future time steps is output; the predicted wavefront phase is reconstructed into a phase modulator suitable for correction. The phase distribution form is obtained, and the reconstructed predicted wavefront phase is mapped to the control voltage signal of the phase modulator and then input into the phase modulator, so that the phase modulator adjusts the phase structure based on the control voltage signal, and performs real-time phase compensation on the actual wavefront phase of Q future time steps; it can solve the problem of time delay in existing wavefront phase prediction and turbulence compensation methods, accurately predict the time-space related turbulence model through the graph multi-head attention model, jointly estimate the wavefront phase data in the time dimension and the space dimension, combine the hidden state of the historical wavefront phase with the time-space feature embedding, and significantly improve the real-time and accuracy of wavefront phase compensation.

[0090] Figure 6 It is a block diagram of an adaptive optical real-time compensation system provided by an embodiment of the present application, which system includes at least the following modules: a wavefront phase receiving module 610, a historical phase extraction module 620, a graph data construction module 630, a wavefront phase prediction module 640, a voltage signal generation module 650 and a phase modulator 660.

[0091] The wavefront phase receiving module 610 is used to receive the wavefront phase image in real time and send the wavefront phase image to the historical phase extraction module 620; The historical phase extraction module 620 is used to extract the historical wavefront phase diagram corresponding to P historical time steps before the current time step from the currently received wavefront phase diagram, each historical wavefront phase diagram corresponds to a historical time step vector; and send the historical wavefront phase diagram to the graph data construction module 630; The graph data construction module 630 is used to construct node features and an adjacency matrix based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase graph and the connection relationship between the pixel nodes, and obtain the graph data of the historical wavefront phase graph; and send the graph data to the wavefront phase prediction module 640; The wavefront phase prediction module 640 is used to input the graph data and the historical time step vector into the pre-trained graph multi-head attention model, and output the predicted wavefront phase of Q future time steps; the predicted wavefront phase is sent to the voltage signal generation module 650; wherein the model includes a first fully connected layer, an encoding layer, a conversion attention layer, a decoding layer, and a second fully connected layer connected in series; the first fully connected layer is used to convert the graph data into a spatial feature embedding, and fuse the time feature embedding of the historical time step vector and the future time step vector to obtain a spatiotemporal feature embedding; the encoding layer is used to encode the spatiotemporal feature embedding and output the historical feature; the conversion attention layer is used to map the historical feature to the future feature corresponding to the future time step; the decoding layer is used to decode the future feature and output the preliminary prediction result; the second fully connected layer is used to optimize the preliminary prediction result and output the predicted wavefront phase; The voltage signal generating module 650 is used to reconstruct the predicted wavefront phase into a phase distribution form suitable for correction by the phase modulator 660, and map the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator 660 and then input it into the phase modulator 660; The phase modulator 660 is used to adjust the phase structure based on the control voltage signal and perform real-time phase compensation on the actual wavefront phase of Q future time steps.

[0092] For relevant details, refer to the above embodiments.

[0093] It should be noted that: the adaptive optical real-time compensation system provided in the above embodiment only uses the division of the above functional modules as an example when performing adaptive optical real-time compensation. In actual applications, the above functions can be assigned to different functional modules as needed to complete all or part of the functions described above. In addition, the adaptive optical real-time compensation system provided in the above embodiment and the adaptive optical real-time compensation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0094] Corresponding to the above method, another aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the adaptive optical real-time compensation method as described above.

[0095] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the adaptive optics real-time compensation method as described above when the computer program / instruction is executed by a processor.

[0096] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0097] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0098] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An adaptive optical real-time compensation method, characterized in that: The method comprises the following steps: Extracting the historical wavefront phase diagram corresponding to P historical time steps before the current time step from the currently received wavefront phase diagram, each historical wavefront phase diagram corresponds to a historical time step vector; wherein P is an integer greater than 1; Based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase map and the connection relationship between the pixel nodes, a node feature and an adjacency matrix are constructed to obtain the graph data of the historical wavefront phase map; The graph data and the historical time step vector are input into the pre-trained graph multi-head attention model, and the predicted wavefront phases of Q future time steps are output; the model includes a first fully connected layer, an encoding layer, a conversion attention layer, a decoding layer, and a second fully connected layer connected in series; the first fully connected layer is used to convert the graph data into a spatial feature embedding, and fuse the time feature embeddings of the historical time step vector and the future time step vector to obtain a spatiotemporal feature embedding; the encoding layer is used to encode the spatiotemporal feature embedding and output historical features; the conversion attention layer is used to map the historical features to future features corresponding to the future time step; the decoding layer is used to decode the future features and output a preliminary prediction result; the second fully connected layer is used to optimize the preliminary prediction result and output the predicted wavefront phase; wherein Q is an integer greater than 1; The predicted wavefront phase is reconstructed into a phase distribution form suitable for phase modulator correction, and the reconstructed predicted wavefront phase is mapped to a control voltage signal of the phase modulator and then input into the phase modulator, so that the phase modulator adjusts the phase structure based on the control voltage signal and performs real-time phase compensation on the actual wavefront phases of the Q future time steps.

2. The adaptive optical real-time compensation method according to claim 1, characterized in that: The historical time step vector is a joint time vector obtained by concatenating the week code and the time code; the graph data is converted into a spatial feature embedding, and the time feature embedding of the historical time step vector and the future time step vector is fused to obtain a spatiotemporal feature embedding, including: The graph data is calculated by a biased second-order random walk algorithm and converted into a spatial feature embedding matrix; Mapping the joint time vector into a time feature embedding vector; The spatial feature embedding matrix and the temporal feature embedding vector are aligned node by node and fused to obtain the spatiotemporal feature embedding.

3. The adaptive optical real-time compensation method according to claim 1, characterized in that: The encoding layer and the decoding layer respectively include L serially connected self-attention modules; each self-attention module includes a spatial attention submodule, a temporal attention submodule and a gated fusion submodule; the spatial attention submodule includes K parallel spatial attention heads, and the temporal attention submodule includes K parallel temporal attention heads; wherein K is an integer greater than 1; Each spatial attention head is used to take the node weighted sum output by the spatial attention submodule in the previous self-attention module as the hidden state of the pixel node, embed the connection of each pixel node hidden state with the corresponding spatiotemporal feature, and calculate the similarity between each pixel node and other pixel nodes through the scaled dot product algorithm; normalize the similarity to obtain the pixel node attention score corresponding to each pixel node; calculate the node weighted sum corresponding to each pixel node based on the pixel node attention score and the pixel node hidden state; merge the outputs of K spatial attention heads to obtain the output result of the spatial attention submodule; Each temporal attention head is used to take the node weighted sum output by the temporal attention submodule in the previous self-attention module as the hidden state of the pixel node, embed the connection of each pixel node hidden state with the corresponding spatiotemporal feature, and calculate the time step similarity between the time step corresponding to each pixel node and other time steps through the scaled dot product algorithm; normalize the time step similarity to obtain the time step attention score; calculate the node weighted sum corresponding to each pixel node based on the time step attention score and the pixel node hidden state; merge the outputs of K temporal attention heads to obtain the output result of the temporal attention submodule; The gated fusion submodule is used to fuse the output result of the spatial attention submodule and the output result of the temporal attention submodule through a gated fusion mechanism and output them.

4. The adaptive optical real-time compensation method according to claim 1, characterized in that: The conversion attention layer includes a first branch, a second branch and an output layer; The first branch is used to perform nonlinear projection on the temporal feature embedding to obtain a nonlinear projection result; the nonlinear projection result is subjected to an inner product operation and then normalized to obtain a history-future time step attention score as an output result of the first branch; the history-future time step attention score is used to indicate the correlation between the future time step and the history time step; The second branch is used to perform nonlinear projection on the historical features to obtain an output result of the second branch; The output layer is used to perform an inner product operation on the output result of the first branch and the output result of the second branch, and to perform matrix concatenation on the results of the inner product operation to obtain the future features.

5. The adaptive optical real-time compensation method according to claim 4, characterized in that: The first branch includes a first nonlinear projection layer, a first inner product operation layer and a normalization layer in sequence; The second branch includes a second nonlinear projection layer; The output layer includes a second inner product operation layer and a matrix concatenation layer in sequence.

6. The adaptive optical real-time compensation method according to claim 1, characterized in that: The training process of the graph multi-head attention model includes: Input the sample training graph data and the sample time series information into the initial graph multi-head attention model, and output the prediction result; the initial graph multi-head attention model has the same model structure as the graph multi-head attention model; the sample training graph data and the sample time series information are extracted from a wavefront phase map with time series correlation; Compare the prediction result with the sample training graph data and the real data corresponding to the sample time series information, and evaluate the prediction result by a preset loss function to obtain a loss result; the preset loss function includes a mean absolute error loss function, a root mean square error loss function, or a mean absolute percentage error loss function; Based on the loss result, the parameters of each layer in the initial graph multi-head attention model are iteratively updated and optimized through the back propagation algorithm and the gradient descent algorithm to obtain the graph multi-head attention model.

7. The adaptive optical real-time compensation method according to claim 1, characterized in that: After reconstructing the predicted wavefront phase into a phase distribution form suitable for phase modulator correction, and mapping the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator and inputting it into the phase modulator, the method further includes: Comparing the compensated wavefront phase with the predicted wavefront phase to obtain an error result; A new control voltage signal is generated based on the error result and sent to the phase modulator, so that the phase modulator adjusts the phase structure based on the new control voltage signal.

8. An electronic device comprising a processor, a memory and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. An adaptive optical real-time compensation system, characterized in that: The system comprises: A wavefront phase receiving module, used for receiving a wavefront phase image in real time and sending the wavefront phase image to a historical phase extraction module; The historical phase extraction module is used to extract the historical wavefront phase diagram corresponding to P historical time steps before the current time step from the currently received wavefront phase diagram, each historical wavefront phase diagram corresponds to a historical time step vector; and send the historical wavefront phase diagram to the graph data construction module; The graph data construction module is used to construct node features and an adjacency matrix based on the light intensity attribute corresponding to each pixel node in the historical wavefront phase graph and the connection relationship between the pixel nodes to obtain the graph data of the historical wavefront phase graph; and send the graph data to the wavefront phase prediction module; The wavefront phase prediction module is used to input the graph data and the historical time step vector into the pre-trained graph multi-head attention model, and output the predicted wavefront phase of Q future time steps; the predicted wavefront phase is sent to the voltage signal generation module; wherein the model includes a first fully connected layer, an encoding layer, a conversion attention layer, a decoding layer, and a second fully connected layer connected in series; the first fully connected layer is used to convert the graph data into a spatial feature embedding, and fuse the time feature embedding of the historical time step vector and the future time step vector to obtain a spatiotemporal feature embedding; the encoding layer is used to encode the spatiotemporal feature embedding and output the historical feature; the conversion attention layer is used to map the historical feature to the future feature corresponding to the future time step; the decoding layer is used to decode the future feature and output a preliminary prediction result; the second fully connected layer is used to optimize the preliminary prediction result and output the predicted wavefront phase; The voltage signal generating module is used to reconstruct the predicted wavefront phase into a phase distribution form suitable for phase modulator correction, and map the reconstructed predicted wavefront phase into a control voltage signal of the phase modulator and then input it into the phase modulator; The phase modulator is used to adjust the phase structure based on the control voltage signal to perform real-time phase compensation on the actual wavefront phases of the Q future time steps.

Images