Coded aperture imaging short-time image motion determination method based on ConvNeXt coarse image motion estimation network and constrained phase correlation

Abstract

The invention provides a coding aperture imaging short-time image motion determination method based on a ConvNeXt coarse image motion estimation network and constraint phase correlation, and the method comprises the steps: determining an effective matching region in a coding modulation effective field of view, inputting two adjacent frames into ConvNeXt in a paired manner, extracting cross-frame multi-scale features, generating a structure guide weight according to feature difference, and carrying out the weighted convergence, regressing the coarse image motion vector and the displacement search range and generating a structure highlighting feature map; then roughly aligning a to-be-detected frame, respectively calculating an image phase correlation graph and a structure phase correlation graph in a limited range, fusing the image phase correlation graph and the structure phase correlation graph to obtain a joint phase correlation graph, calculating displacement confidence coefficient based on peak position consistency and main peak prominence, screening displacement peaks, and performing sub-pixel refinement in a local window to obtain a refined image motion vector; finally, outputting a final image motion and a quality mark according to confidence and consistency judgment, and generating a compensation control quantity or a zero compensation control quantity according to the final image motion and the quality mark; the method can improve the image motion judgment reliability and reduce the error compensation risk.

Description

Technical Field

[0001] This invention relates to the field of image shift measurement and compensation control technology for coded aperture imaging, and in particular to a short-time image shift determination method for coded aperture imaging based on a ConvNeXt coarse image shift estimation network and constrained phase correlation. Background Technology

[0002] Coordinated aperture imaging systems, used in applications such as vehicle-mounted inspection, airborne reconnaissance, and shipborne surveillance, often operate in environments with minor vibrations and short-term disturbances, inevitably resulting in two-dimensional image shifts between consecutive short-duration imaging frames. Unlike conventional imaging, coded aperture modulators introduce specific modulation into the optical link, ensuring that the acquired coded image simultaneously contains target structural information and coded modulation texture. When short-duration image shifts occur, the superimposed displacement of these two elements alters the discriminability of inter-frame matching. In engineering practice, the consistency and non-abrupt nature of image shift results over time are paramount to ensure stable operation of subsequent imaging interpretation, use, and compensation processes.

[0003] Current short-term image shift determination is typically implemented using an image registration process: matching regions are selected between two adjacent frames, and the translation amount is estimated in the frequency domain using cross-correlation, phase correlation, or their constraints. This can be combined with a multi-scale pyramid for coarse-to-fine search, obtaining integer pixel displacements through peak detection, and then sub-pixel refinement is achieved through interpolation or local fitting. Other methods perform filtering, enhancement, and edge / gradient feature transformations on the input before correlation matching to improve relevance. In recent years, displacement regression or feature extraction schemes based on convolutional neural networks have also emerged, inputting paired frames into the network to obtain initial displacement values ​​or directly outputting displacements for subsequent registration or compensation control.

[0004] The above-mentioned schemes are easily affected by the combined effects of modulation texture and effective field of view boundary in coded aperture imaging: when relying solely on image phase correlation / cross-correlation, the correlation surface may exhibit multiple peaks, lack of prominent main peaks, or peak position drift, causing the displacement solution to switch between adjacent frames and produce jumps; when directly regressing displacement using only the network, there is a lack of constraints consistent with the correlation peak structure and interpretable judgment criteria, making it difficult to filter out unreliable results; at the same time, a fixed or excessively large search range may introduce more candidate peaks under short-term perturbation changes, resulting in insufficient stability of displacement judgment and potentially causing inconsistent output of compensation control.

[0005] Therefore, a method for determining short-term image shift in coded aperture imaging that can overcome the shortcomings of the prior art is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a short-term image shift determination method for coded aperture imaging based on a ConvNeXt coarse image shift estimation network and constrained phase correlation. The core technical problem to be solved by this application is: in coded aperture imaging scenarios with coded modulation texture interference, limited effective field of view, and frequent short-term platform disturbances, how to achieve image shift determination and consistent output with a certain degree of confidence for adjacent short-term frames, and how to reliably back off when the results are unreliable, thereby supporting stable compensation control and subsequent workflow.

[0007] The short-term image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation according to embodiments of the present invention includes:

[0008] S1. Receive continuous imaging frames from the coded aperture imaging system, select the previous frame as the reference coded aperture image and the next frame as the coded aperture image to be tested, and determine the effective matching area within the effective field of view of the coded modulation.

[0009] S2. Within the effective matching region, the reference coded aperture image and the coded aperture image to be tested are input into the ConvNeXt coarse image shift estimation network in a paired manner. The ConvNeXt backbone is used to extract cross-frame multi-scale features. Based on the cross-frame multi-scale features, the cross-frame feature differences are calculated and structural guidance information is formed. The structural guidance information is used to perform weighted convergence of the cross-frame multi-scale features to generate the reference structural salience feature map and the salience feature map to be tested. Based on the weighted converged cross-frame multi-scale features, the coarse image shift vector is obtained by regression. The displacement search range parameters are determined according to the coarse image shift vector.

[0010] S3. Based on the coarse image shift vector and displacement search range parameters, coarsely align the coded aperture image to be tested with the spurious feature map of the structure to be tested. Within the displacement range defined by the displacement search range parameters, calculate the image phase correlation map between the reference coded aperture image and the coded aperture image to be tested, and calculate the structural phase correlation map between the reference structural spurious feature map and the spurious feature map of the structure to be tested. Fuse the image phase correlation map and the structural phase correlation map in the same displacement coordinate system to obtain a joint phase correlation map. Extract the displacement peak list from the joint phase correlation map. Calculate the peak position consistency index based on the peak position deviation between the image phase correlation map and the structural phase correlation map. Calculate the main peak prominence index based on the difference between the main peak and the secondary peak in the joint phase correlation map. Calculate the displacement confidence score using the peak position consistency index and the main peak prominence index, and filter to obtain the displacement judgment peaks.

[0011] S4. Determine the local refinement window using the displacement judgment peak, perform sub-pixel localization on the joint phase correlation map within the local refinement window, and obtain the refined image shift vector;

[0012] S5. Determine the final image shift vector based on the displacement confidence level and the preset confidence threshold. If the displacement confidence level meets the confidence threshold, use the fine image shift vector. If the displacement confidence level does not meet the confidence threshold, use the coarse image shift vector and generate a quality label.

[0013] S6. Based on the quality mark, the compensation mode is determined. When the quality mark is valid, a compensation control quantity is generated according to the final image shift vector. When the quality mark is invalid, a zero compensation control quantity is generated.

[0014] Optionally, S1 is as follows:

[0015] Receive continuous imaging frames from the coded aperture imaging system, and select two adjacent frames in chronological order under the same pixel coordinate system as the processing objects;

[0016] The two adjacent frames with the earlier time are set as reference coded aperture images, and the two adjacent frames with the later time are set as coded aperture images to be tested, while keeping the height, width and number of channels of both consistent.

[0017] The effective field of view of the coded aperture modulator is determined based on the calibration parameters of the coded aperture modulator, and the effective field of view of the coded aperture modulator is mapped to the pixel coordinate system of the reference coded aperture image and the coded aperture image to be tested to form the effective field of view region;

[0018] Based on the effective field of view region and combined with the preset maximum displacement boundary, the effective field of view region is shrunken to obtain the effective matching region. The reference coded aperture image and the coded aperture image to be tested are cropped within the effective matching region as the pairing input data for the subsequent ConvNeXt coarse image shift estimation network.

[0019] Optionally, S2 is as follows:

[0020] The reference coded aperture image and the coded aperture image to be tested within the effective matching area are stitched together along the channel dimension to form a paired input tensor, which is then input into the ConvNeXt coarse image shift estimation network.

[0021] The ConvNeXt backbone is used to perform staged convolutional feature extraction on paired input tensors. In each stage, the staged convolutional feature extraction sequentially executes a downsampling layer and several ConvNeXt blocks. The ConvNeXt blocks sequentially include a depthwise convolutional layer, a normalization layer, a pointwise convolutional expansion layer, a nonlinear activation unit, a pointwise convolutional compression layer, and a residual connection, outputting cross-frame multi-scale features with multiple downsampling scales.

[0022] The cross-frame multi-scale features at each scale are input into the cross-frame feature difference calculation branch. First, the cross-frame multi-scale features are separated into reference corresponding features and test corresponding features according to the channel source of the paired input tensor. Then, position-by-position difference operation is performed on the reference corresponding features and test corresponding features to obtain the difference feature map at each scale.

[0023] The differential feature map is input into the structure guidance information to generate branches. Convolutional layers and nonlinear activation units are used to map the differential feature map, and the output structure guidance information with the same scale as the differential feature map is output. The structure guidance information is a single-channel weight map.

[0024] The cross-frame multi-scale features and structural guidance information are input into the weighted convergence module. The cross-frame multi-scale features are weighted position by position, and the features at different scales are aligned with resolution before cross-scale convergence is performed to obtain the weighted converged cross-frame multi-scale features.

[0025] The weighted and aggregated multi-scale features across frames are input into the structure salience map generation head. The number of feature channels is compressed using pointwise convolutional layers, and upsampling and fusion are used to restore the multi-scale features to the spatial resolution of the effective matching region. The output is a reference structure salience map and a test structure salience map.

[0026] The weighted and converged cross-frame multi-scale features are input into the coarse image shift vector regression head and the displacement search range parameter regression head. A global convergence layer is used to generate a fixed-length feature vector, which is then output as a coarse image shift vector and displacement search range parameter through a fully connected regression layer. Boundary constraints are applied to the displacement search range parameter based on the coarse image shift vector to ensure that the displacement search range parameter falls within the preset maximum displacement boundary.

[0027] Optionally, when the structure guidance information generation branch maps the differential feature map and outputs structure guidance information, the weight values ​​of the structure guidance information at the pixel location are calculated using a weight calculation formula:

[0028] ;

[0029] in, Represents structural guidance information At pixel position The weight value at that location, Indicates the first Differential feature maps at various scales, The structure guides the information generation branch in the first... Convolutional kernel parameters at various scales, Indicates and The corresponding bias parameter, symbol This represents the convolution operation. Indicates the reference corresponding feature at position The channel vector value at that location. Indicates the location of the corresponding feature to be tested. The channel vector value at that location, sign This represents the dot product operation of channel vectors. This represents the L2 norm operation of the channel vector. This indicates a Sigmoid nonlinear activation. This represents the similarity scaling factor. This represents a positive constant used to avoid a denominator of zero, calculated by the processing end as described above. A weight value is provided at each spatial location to indicate that the location is more likely to belong to a target structural region that moves consistently across frames.

[0030] Optionally, S3 specifically refers to:

[0031] A coarse-aligned coded aperture image is obtained by performing a translation transformation on the image to be tested based on the coarse image shift vector, and a coarse-aligned coded aperture image is obtained by performing a translation transformation on the spur feature map of the structure to be tested using the same displacement as the translation transformation, thereby limiting the data after the translation transformation to the effective matching region;

[0032] Within the displacement range defined by the displacement search range parameter, constrained phase correlation calculation is performed using the reference coded aperture image and the coarsely aligned coded aperture image to be tested as input. The input image is then subjected to discrete Fourier transform, the normalized cross-frame frequency domain correlation quantity is calculated, and an inverse transform is performed to obtain an image phase correlation map covering the displacement range.

[0033] Within the displacement range defined by the displacement search range parameter, constrained phase correlation calculation is performed using the reference structure salient feature map and the coarsely aligned salient feature map of the structure to be tested as inputs. The input feature maps are then subjected to discrete Fourier transform, normalized cross-frame frequency domain correlation quantities are calculated, and inverse transform is performed to obtain a structural phase correlation map covering the displacement range.

[0034] The image phase correlation map and the structure phase correlation map are mapped to the same displacement coordinate system, and the image phase correlation map and the structure phase correlation map are weighted and fused using a preset fusion weight to obtain a joint phase correlation map;

[0035] Local maxima detection is performed on the joint phase correlation map to obtain a list of displacement peaks. After sorting the list of displacement peaks by peak intensity, candidate displacement peaks corresponding to a preset number of peaks are retained.

[0036] For each candidate displacement peak, the peak position deviation is calculated based on the peak coordinates in the image phase correlation map and the peak coordinates in the structural phase correlation map. A peak position consistency index is generated based on the peak position deviation and a preset consistency threshold. A main peak prominence index is generated based on the main peak value and the secondary peak value in the joint phase correlation map. The peak position consistency index and the main peak prominence index are weighted by a preset weight to obtain the displacement confidence.

[0037] Select the candidate displacement peak that meets the preset screening threshold and has the highest displacement confidence from the displacement peak list as the displacement judgment peak.

[0038] Optionally, the confidence level of the displacement can be calculated using the confidence level calculation formula:

[0039] ;

[0040] in, Indicates the first Displacement confidence of each candidate displacement peak. This represents the Sigmoid non-linear activation function, which takes real numbers as input and outputs a value between zero and one. The preset weights represent the peak position consistency index. Indicator of peak position consistency This indicates the preset weight of the peak prominence index. This indicates the prominence of the main peak. , These represent the horizontal and vertical components of the image peak coordinates, respectively. , These represent the horizontal and vertical components of the structural peak coordinates, respectively. This indicates a preset consistency threshold. This represents a positive constant used to avoid a denominator of zero. This indicates taking the absolute value; the processing end will assign the displacement confidence level. With candidate displacement peak displacement coordinates Associated storage.

[0041] Optionally, S4 specifically refers to:

[0042] Centered on the displacement coordinates corresponding to the displacement judgment peak, a local refinement window is extracted from the joint phase correlation diagram according to the preset refinement window size, and the spatial range of the local refinement window is cropped to the displacement range corresponding to the displacement search range parameter.

[0043] Boundary attenuation weighting is applied to the joint phase correlation map within the local refinement window, so that the weight of the edge region of the local refinement window is less than the weight of the center region;

[0044] The joint phase correlation map within the weighted local refinement window is interpolated and upsampled using a preset upsampling factor to obtain a local refinement window with sub-pixel resolution.

[0045] Within a local refinement window at subpixel resolution, the peak coordinates are determined and converted into a subpixel displacement offset relative to the displacement determination peak. The displacement coordinates of the displacement determination peak are then superimposed with the subpixel displacement offset to obtain the refined image shift vector.

[0046] Optional, S5 specifically includes:

[0047] Read the displacement confidence level and compare it with the preset confidence threshold to obtain the confidence judgment result;

[0048] When the credibility determination result is satisfied, the absolute values ​​of the differences between the fine image shift vector and the coarse image shift vector in the horizontal and vertical components are taken respectively, and compared with the preset consistency threshold to obtain the consistency determination result.

[0049] When both the credibility and consistency judgment results are satisfied, the fine image shift vector is determined as the final image shift vector; when either the credibility or consistency judgment results are not satisfied, the coarse image shift vector is determined as the final image shift vector.

[0050] A quality marker is generated based on the source of the final image shift vector. The quality marker is set to valid when the final image shift vector is determined by the fine image shift vector, and set to invalid when the final image shift vector is determined by the coarse image shift vector.

[0051] Optionally, step S6 specifically includes:

[0052] Read the quality marker and compare it with the preset valid marker value to obtain the compensation mode determination result;

[0053] When the compensation mode determination result is valid, the final image shift vector is used as input and the compensation control quantity is generated using the preset image shift to control mapping parameters;

[0054] The compensation control quantity is subjected to amplitude limiting processing to make the compensation control quantity fall within the preset control amplitude boundary, and the amplitude-limited compensation control quantity is output to the compensation execution mechanism of the coded aperture imaging system.

[0055] When the compensation mode determination result is invalid, the compensation control quantity is set to zero and output to the compensation execution agency.

[0056] The beneficial effects of this invention are:

[0057] (1) This proposal proposes an improved method for determining short-term image shift in coded aperture imaging. By constructing an effective matching region within the effective field of view of coded modulation, and introducing the ConvNeXt coarse image shift estimation network to pair adjacent frames and regress the coarse image shift vector and displacement search range parameters, a processing link of "coarse localization first, then restricted search" is realized. Compared with the approach based solely on global phase correlation or a fixed search window, this scheme uses coarse image shift to coarsely align the test frames and limits the subsequent phase correlation calculation to the displacement range regressed by the network. At the same time, the maximum displacement boundary constraint is applied to the displacement range to prevent the correlation calculation from falling into the invalid modulation region and the false correlation response caused by the out-of-bounds region. This reduces the number of non-target peaks dominated by coded texture entering the candidate set, thereby enhancing the consistency of image shift determination in engineering scenarios from the process level.

[0058] (2) This proposal proposes a novel "structure-guided feature-joint phase correlation-confidence screening" mechanism. Addressing the multi-peak and peak position drift issues caused by the superposition of target structure and modulation texture in coded aperture images, it employs cross-frame multi-scale feature difference to generate single-channel structure-guided information. The cross-frame multi-scale features are then weighted and converged position-wise to form a reference and test structure prominence feature map. Based on this, the image phase correlation map and the structure phase correlation map are calculated separately and fused into a joint phase correlation map, extracting candidate peaks from the same displacement coordinate system. Furthermore, this scheme utilizes the peak position deviation between the image phase correlation peak and the structure phase correlation peak to construct a peak position consistency index, and combines this with the difference between the main peak and secondary peaks in the joint phase correlation map to construct a main peak prominence index. This comprehensive approach yields the displacement confidence score, which is used to screen displacement judgment peaks. This design differs from strategies that rely solely on the magnitude of a single correlation peak, ensuring that displacement judgment simultaneously satisfies both "cross-representation consistency" and "main peak distinguishability," thus providing a clearer basis for discriminating between multi-peak ambiguity and modulation texture interference.

[0059] (3) This proposal proposes an image shift output method for a compensation control closed loop. Through sub-pixel localization of a local refinement window and a "reliable judgment + consistency backoff" rule, a stable output usable for engineering control is formed. The scheme extracts a local refinement window centered on the displacement judgment peak, applies attenuation weighting to the window boundary, and interpolates and upsamples to obtain the refined image shift vector by completing the peak sub-pixel localization within the local area. Then, the usability of the refined image shift is determined by the displacement confidence level and a preset reliability threshold. Consistency verification is performed on the component differences between the refined and coarse image shifts. If the conditions are not met, the coarse image shift is used, and a quality marker is generated. The quality marker is further used to select the compensation mode: when effective, the compensation control quantity after amplitude limiting is output; when ineffective, the zero compensation control quantity is output. Compared with the method of directly driving the actuator without reliable gating, this scheme explicitly transmits the uncertainty of image shift judgment to the control side, reducing compensation chaos and state jump risks caused by unreliable displacement. Attached Figure Description

[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0061] Figure 1 This is a flowchart of a short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention;

[0062] Figure 2 This is a flowchart of the effective matching region determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention.

[0063] Figure 3 This invention presents a flowchart of the ConvNeXt coarse image shift estimation network and structure-guided generation process for a short-term image shift determination method for coded aperture imaging based on a ConvNeXt coarse image shift estimation network and constrained phase correlation.

[0064] Figure 4 This is a flowchart of the coarse alignment and constrained phase correlation joint displacement determination method for short-time image shift determination in coded aperture imaging based on ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention;

[0065] Figure 5 This is a flowchart of the joint phase correlation map subpixel refinement localization method for short-term image shift determination in coded aperture imaging based on ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention;

[0066] Figure 6 This is a flowchart of the final image shift vector determination and quality labeling based on displacement confidence in a short-time image shift determination method for coded aperture imaging based on ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention.

[0067] Figure 7 The flowchart shows the quality-label-based compensation mode determination and control output of a short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention.

[0068] Figure 8 This is a schematic diagram showing a more prominent peak in the joint phase correlation map of a short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention;

[0069] Figure 9This is a diagram of the ConvNeXt network architecture for a short-term image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation proposed in this invention. Detailed Implementation

[0070] In Example 1, reference Figures 1 to 9 A method for determining short-term image shift in coded aperture imaging based on a ConvNeXt coarse image shift estimation network and constrained phase correlation includes:

[0071] S1. Receive continuous imaging frames from the coded aperture imaging system, select the previous frame as the reference coded aperture image and the next frame as the coded aperture image to be tested, and determine the effective matching area within the effective field of view of the coded modulation.

[0072] S2. Within the effective matching region, the reference coded aperture image and the coded aperture image to be tested are input into the ConvNeXt coarse image shift estimation network in a paired manner. The ConvNeXt backbone is used to extract cross-frame multi-scale features. Based on the cross-frame multi-scale features, the cross-frame feature differences are calculated and structural guidance information is formed. The structural guidance information is used to perform weighted convergence of the cross-frame multi-scale features to generate the reference structural salience feature map and the salience feature map to be tested. Based on the weighted converged cross-frame multi-scale features, the coarse image shift vector is obtained by regression. The displacement search range parameters are determined according to the coarse image shift vector.

[0073] S3. Based on the coarse image shift vector and displacement search range parameters, coarsely align the coded aperture image to be tested with the spurious feature map of the structure to be tested. Within the displacement range defined by the displacement search range parameters, calculate the image phase correlation map between the reference coded aperture image and the coded aperture image to be tested, and calculate the structural phase correlation map between the reference structural spurious feature map and the spurious feature map of the structure to be tested. Fuse the image phase correlation map and the structural phase correlation map in the same displacement coordinate system to obtain a joint phase correlation map. Extract the displacement peak list from the joint phase correlation map. Calculate the peak position consistency index based on the peak position deviation between the image phase correlation map and the structural phase correlation map. Calculate the main peak prominence index based on the difference between the main peak and the secondary peak in the joint phase correlation map. Calculate the displacement confidence score using the peak position consistency index and the main peak prominence index, and filter to obtain the displacement judgment peaks.

[0074] S4. Determine the local refinement window using the displacement judgment peak, perform sub-pixel localization on the joint phase correlation map within the local refinement window, and obtain the refined image shift vector;

[0075] S5. Determine the final image shift vector based on the displacement confidence level and the preset confidence threshold. If the displacement confidence level meets the confidence threshold, use the fine image shift vector. If the displacement confidence level does not meet the confidence threshold, use the coarse image shift vector and generate a quality label.

[0076] S6. Based on the quality mark, the compensation mode is determined. When the quality mark is valid, a compensation control quantity is generated according to the final image shift vector. When the quality mark is invalid, a zero compensation control quantity is generated.

[0077] In this embodiment, step S1 specifically includes:

[0078] The coded aperture imaging system outputs continuous imaging frames at a fixed frame period. The processing end receives the continuous imaging frames sequentially by frame number or timestamp to form a coded aperture image sequence. ,in, Indicates the first Frame-coded aperture image, High ,width With the number of channels The number of channels The processing end will be consistent with the image channel configuration output by the coded aperture imaging system. These are considered as two-dimensional sampling results in the same pixel coordinate system, thus providing a consistent coordinate basis for subsequent cross-frame pairing inputs;

[0079] Upon receiving the coded aperture image sequence Then, the processing unit selects two adjacent frames in chronological order as the processing objects. and Forming a pair of adjacent frames, the processing end will select the frame with the earlier time frame. Set as reference coded aperture image The later the time Set as the image of the coded aperture to be tested In this embodiment, the reference coded aperture image is used. Image of the coded aperture to be tested Keep the same height ,width With the number of channels This ensures that the ConvNeXt coarse image shift estimation network does not introduce size inconsistency issues when stitching the two frames along the channel dimension.

[0080] Subsequently, the processing end uses the calibration parameters of the coded aperture modulator. Determine the effective field of view for coded modulation, the calibration parameters Obtained through an offline calibration process, which includes: placing a preset calibration target within the field of view of the coded aperture imaging system; controlling the coded aperture modulator to output a preset coding state and acquiring multiple frames of coded aperture images; detecting target corner points or edges in the multiple frames of coded aperture images as pixel reference points; solving the mapping relationship between the pixel coordinate system and the coded aperture modulator coordinate system based on the pixel reference points and the geometric layout of the coded aperture modulator; and writing the mapping relationship, alignment offset, and effective coverage area of ​​the coded aperture modulator in the sensor pixel coordinate system into the data. The processing end is based on Reference coded aperture image Image of the coded aperture to be tested Generate the effective field of view in the pixel coordinate system The effective field of view area This represents the set of pixel coordinates or its corresponding binary mask where the coded modulation texture appears effectively in both frames. The processing end will then define the effective field of view region. Simultaneously applied to the reference coded aperture image Image of the coded aperture to be tested This ensures that the subsequent cropped areas of the two frames completely overlap in the same pixel coordinate system.

[0081] Furthermore, the processing end is based on the effective field of view area. And combined with the preset maximum displacement boundary For the effective field of view Boundary contraction is performed to obtain an effective matching region. ,in, These are preset parameters used to characterize the maximum allowable range of short-term image shift. It includes the maximum displacement boundary in the horizontal direction and the maximum displacement boundary in the vertical direction. The boundary contraction is performed as follows: within the effective field of view... The left, right, top, and bottom boundaries contract inwards respectively. The corresponding boundary width is used to avoid subsequent coarse alignment and phase correlation calculations covering invalid regions beyond the effective field of view of the coded modulation. When provided in the form of a binary mask, the processing end first determines the axis-aligned circumscribed rectangle of the mask, and then obtains the effective matching region of the axis alignment according to the boundary shrinkage rule mentioned above. This allows for direct use in image cropping and network input construction;

[0082] Finally, the processing end will refer to the coded aperture image. In the valid matching area Obtain the reference valid matching image by inner cut. The image of the coded aperture to be tested In the valid matching area The inner section obtains the valid matching image to be tested. superscript Indicates the valid matching region The reference valid matching image is obtained by inner cropping. Valid matching image to be tested Having the same spatial resolution and number of channels, and corresponding to the same pixel coordinate range, allows subsequent steps to... and The paired input data of the ConvNeXt coarse image shift estimation network are concatenated according to the channel dimension, and the cross-frame multi-scale features extracted by the ConvNeXt backbone spatially cover the superimposed area of ​​the target structure and the coded modulation texture, providing input support for the subsequent output of reference structure salient feature map, test structure salient feature map, coarse image shift vector and displacement search range parameters.

[0083] In this embodiment, step S2 specifically includes:

[0084] Step S2 is performed within the valid matching region, and the processing end obtains the reference valid matching image. Valid matching image to be tested ,in This represents the cropping result of the reference coded aperture image within the effective matching region. This represents the cropping result of the coded aperture image under test within the effective matching region. and With the same spatial resolution and number of channels, the processing end will and Paired input tensors are formed by concatenating along the channel dimension. The paired input tensor The channel order is fixed with the reference coded aperture image channels first and the coded aperture image channels to be tested last, so that subsequent separation based on channel source has a definite channel index basis. The processing end will pair the input tensors. Input ConvNeXt coarse image shift estimation network The ConvNeXt coarse image shift estimation network It consists of the ConvNeXt backbone, cross-frame feature difference calculation branch, structural guidance information generation branch, weighted convergence module, structural protrusion feature map generation head, coarse image shift vector regression head, and displacement search range parameter regression head;

[0085] Prior to deployment, the ConvNeXt coarse image shift estimation network was tested. Offline training is performed, during which training sample pairs are constructed. Each training sample pair consists of a reference coded aperture image and a coded aperture image to be tested within the effective matching region. A coarse image shift vector supervision label and a displacement search range parameter supervision label are provided for each training sample pair. The supervision labels are obtained by applying a known pixel-level displacement to the coded aperture imaging system using a displacement-controllable translation platform, or by applying a known pixel-level translation to the reference coded aperture image and re-superimposing the coded modulation texture. During training, paired input tensors are used. Forward inference yields predicted values ​​of the coarse image shift vector and displacement search range parameters. A regression loss is used to measure the difference between the predicted values ​​and the supervision labels. Backpropagation updates the network parameters in the ConvNeXt backbone, cross-frame feature difference calculation branch, structure guidance information generation branch, weighted convergence module, structure saliency feature map generation head, coarse image shift vector regression head, and displacement search range parameter regression head, ensuring the optimal convolutional kernel parameters used in subsequent inference stages. With bias parameters To fix the parameters after training;

[0086] The ConvNeXt backbone uses paired input tensors The input is used for staged convolutional feature extraction. Each stage first performs a downsampling layer, followed by concatenated ConvNeXt blocks. The downsampling layer uses stride convolution to reduce spatial resolution and increase the number of feature channels. Each ConvNeXt block sequentially executes a depthwise convolutional layer, a normalization layer, a pointwise convolutional expansion layer, a non-linear activation unit, a pointwise convolutional compression layer, and a residual connection. The depthwise convolutional layer independently convolves each channel to expand the receptive field and preserve the spatial context of the encoded modulation texture. The pointwise convolutional expansion layer and the pointwise convolutional compression layer are mixed in the channel dimension to fuse the joint representation of the target structure and the encoded modulation texture. The ConvNeXt backbone outputs multi-scale features across frames at multiple downsampling scales. ,in Indicates the first To support subsequent cross-frame multi-scale features, the ConvNeXt backbone organizes the output channels at each scale into two sets of consecutive channels. The first set of channels corresponds to the reference corresponding features, and the second set of channels corresponds to the target corresponding features, thus enabling cross-frame multi-scale features. It can be deterministically separated into reference corresponding features along the channel dimension. Corresponding features to be tested ;

[0087] Cross-frame feature difference calculation branch receives cross-frame multi-scale features at each scale. First, based on the channel index, Separate into and And then and Perform position-by-position difference operations to obtain the difference feature map. Position-by-position difference operations calculate the difference between the reference corresponding feature and the target corresponding feature at the same spatial location and the same channel index, resulting in a difference feature map. The directionality and consistency of cross-frame grayscale changes are expressed in the form of spatial distribution, and the structural guidance information generation branch uses differential feature maps. Corresponding features to the reference , corresponding features to be tested As input, generate structural guidance information. Structural guidance information For a single-channel weighted graph, the weight calculation formula is used for each position:

[0088] ;

[0089] in, Represents structural guidance information At pixel position The weight value at that location, Indicates the first Differential feature maps at various scales, The structure guides the information generation branch in the first... Convolutional kernel parameters at various scales, Indicates and The corresponding bias parameter, symbol This represents the convolution operation. Indicates the reference corresponding feature at position The channel vector value at that location. Indicates the location of the corresponding feature to be tested. The channel vector value at that location, sign This represents the dot product operation of channel vectors. This represents the L2 norm operation of the channel vector. This indicates a Sigmoid nonlinear activation. This represents the similarity scaling factor. This represents a positive constant used to avoid a denominator of zero, calculated by the processing end as described above. A weight value is provided at each spatial location to indicate that the location is more likely to belong to a target structure region that moves consistently across frames;

[0090] The weighted convergence module receives multi-scale features across frames. With structural guidance information The weighted aggregation module first sets the structural guidance information. Broadcast to the channel Consistent channel count, for The weighted features are obtained by performing position-by-position multiplication. Subsequently, the weighted aggregation module combines weighted features at different scales. Aligning to the same spatial resolution, interpolation upsampling is performed on low-resolution scales, while maintaining the original resolution on high-resolution scales. After resolution alignment, the weighted convergence module aligns the weighted features in scale order and performs cross-scale convergence. Cross-scale convergence generates weighted converged cross-frame multi-scale features by concatenating channel dimensions and then performing pointwise convolutional compression. The Used to simultaneously drive the structural salience feature map generation head, the coarse image shift vector regression head, and the displacement search range parameter regression head;

[0091] Structural salient feature map generation head As input, a pointwise convolutional layer is first used to compress the number of channels to a preset number. ,in This represents the number of channels in the structural salience map. Subsequently, the structural salience map generation head restores the multi-scale information to the spatial resolution of the effective matching region. It upsamples the features from different scales participating in the convergence and fuses them at the same resolution. Then, it uses convolutional mapping to form two output feature maps, one being the reference structural salience map. Highlighting features of the structure under test ,in, This represents a feature map highlighting the target structure in the reference coded aperture image. This represents a feature map that highlights the target structure in the coded aperture image under test. and The spatial dimensions are consistent with the effective matching area, and the number of channels is also consistent. This is used as the input for calculating the subsequent structural phase correlation map;

[0092] The coarse image shift vector regression head and the displacement search range parameter regression head share the same parameters. A global description, for the processing end Perform global convergence to generate fixed-length feature vectors ,in The coarse image shift vector regression head represents the overall cross-frame joint representation of the effective matching region. As input, a coarse image shift vector is output through a fully connected regression layer. ,in The vectors are two-dimensional, representing the horizontal and vertical coarse image shift components, respectively. The displacement search range parameter regression head uses... As input, the displacement search range parameter is output through a fully connected regression layer. ,in This is a two-dimensional range vector, representing the horizontal and vertical displacement search radii, respectively. The displacement search range parameters are used to... With respect to the preset maximum displacement boundary Consistent, the processing end is based on the coarse image shift vector Displacement search range parameters Apply boundary constraints, The two components are limited by the Within a given maximum value range, and replacing components less than zero with zero, the displacement search range parameter is set. This generates numerical inputs that can be directly used for subsequent displacement range limiting and coarse alignment. Finally, the ConvNeXt coarse image displacement estimation network... Output reference structure salient feature map 1. Prominent feature diagram of the structure under test Coarse image shift vector With displacement search range parameters This provides the necessary input for subsequent steps to calculate the image phase correlation map and the structure phase correlation map within the displacement range defined by the displacement search range parameter.

[0093] In this embodiment, step S3 specifically includes:

[0094] In one implementation, step S3 uses the coarse image shift vector output in step S2. Displacement search range parameters Reference structure salient feature diagram 1. Prominent feature diagram of the structure under test As input, and with reference valid matching images within the valid matching region. Valid matching image to be tested As the object of phase correlation calculation, among which, and The cropping results of the reference coded aperture image and the coded aperture image to be tested within the effective matching area are used as a reference. It is a two-dimensional coarse image shift vector. It is a two-dimensional displacement range vector. and A multi-channel feature map whose spatial dimensions are consistent with the effective matching region;

[0095] First, coarse alignment is performed, and the processor reads the coarse image shift vector. The horizontal and vertical components will shift the coarse image vector. Displacement converted to translation transformation The displacement Using separate methods The pixel-level translation amount is obtained by taking the nearest integer from the horizontal and vertical components, and the rounded residual is retained as the compensation amount for subsequent sub-pixel localization. The processing end is used to obtain the effective matching image to be tested. Execute by displacement A translation transformation of the parameters yields a coarsely aligned image of the coded aperture to be tested. The processing end displays the prominent feature map of the structure under test. Perform a displacement consistent with the translation transformation to obtain a coarsely aligned salient feature map of the structure under test. The processing end will and Within the coordinate range corresponding to the valid matching area, out-of-bounds areas are removed by clipping, and the empty areas appearing after clipping are filled with a preset fill value, wherein the preset fill value is zero.

[0096] Subsequently, the image phase correlation map is calculated, and the processing unit operates within the displacement search range parameters. Within a defined displacement range, the effective matching image for the reference. Coarsely aligned with the image of the coded aperture to be tested Perform constraint phase correlation calculations, which include: for and The reference spectrum is obtained by performing discrete Fourier transforms on each sample. With the spectrum to be measured ,right and Calculate the cross-frame frequency domain correlation, which is achieved by... and The conjugate multiplication yields the complex correlation spectrum. The complex correlation spectrum is then normalized according to frequency point amplitudes to obtain a normalized cross-frame frequency domain correlation quantity. A positive constant is added to the normalized denominator amplitude to avoid the denominator being zero. An inverse discrete Fourier transform is then performed on the normalized cross-frame frequency domain correlation quantity to obtain the image phase correlation map. The processing end is based on the displacement search range parameter From image phase correlation map Extracting the displacement range sub-map Displacement range subplot Coverage centered on zero displacement, with a horizontal range of... The horizontal component is determined, and the vertical range is determined by... The set of displacement coordinates determined by the vertical component;

[0097] Next, the structural phase correlation diagram is calculated, with the processing end using the same displacement search range parameters. Under certain constraints, highlight the feature map based on the reference structure. Highlighting feature map of the structure under test with coarse alignment Perform constraint phase correlation calculations on the input to obtain the structural phase correlation map. ,when and When generating multi-channel feature maps, the processing unit performs discrete Fourier transform and normalized cross-frame frequency domain correlation calculations on each channel, and performs position-by-position summation or position-by-position averaging on the inverse transform results of each channel to obtain a single-channel structure-phase correlation map. The processing end is based on the displacement search range parameter From the structural phase correlation diagram Extracting the displacement range sub-map and make the displacement range subgraph With displacement range subgraph They have the same origin of displacement coordinates and the same sampling interval of displacement coordinates;

[0098] Then a joint phase correlation map is generated, and the processing end will generate the displacement range submap. With displacement range subgraph Mapped to the same displacement coordinate system, with the coarsely aligned zero displacement as the origin, the processing end uses a preset fusion weight. and right and Weighted fusion is performed to obtain the joint phase correlation map. ,in and It is a non-negative real number and satisfies the preset normalization constraint that the sum of the fusion weights is one;

[0099] Subsequently, a list of displacement peaks is extracted and displacement confidence is calculated. The processing unit then analyzes the joint phase correlation plot. Perform local maximum detection, using a preset neighborhood window size. For each pixel location, compare it with the pixel values ​​in its neighborhood and retain those that satisfy the local maximum condition and have a peak intensity greater than a preset peak threshold. The position is used as the displacement peak to form a displacement peak list. The processing end lists the displacement peaks according to their peak intensity. Sort and retain the preset number of peaks. The corresponding set of candidate displacement peaks The displacement coordinates of the candidate displacement peaks are marked as follows: ,in Index of candidate displacement peaks;

[0100] For each candidate displacement peak, the processing end uses the displacement coordinates of the candidate displacement peak. Centered on the image phase correlation map displacement range submap The search radius is preset. The search yielded the image peak coordinates. And in the displacement range subplot of the structural phase correlation diagram The same preset search radius is used within. The search yielded the coordinates of the structural peaks. The processing end is based on and Calculate peak position deviation Peak position deviation The peak position deviation is obtained by taking the absolute value of the difference between the horizontal and vertical coordinates of the image peak coordinates and summing the two values. Consistency threshold with preset Comparison, peak position deviation Satisfying the consistency threshold Consistency index of peak position Set to 1, peak position deviation The consistency threshold is not met. Consistency index of peak position Set to zero;

[0101] Processing end in joint phase correlation map The peak value of the candidate displacement peak is taken as the candidate peak value. and from the candidate displacement peak set The highest peak value other than the current candidate displacement peak is taken as the second-highest peak value. A positive constant is introduced at the processing end. As a normalized stable term, the difference component is first calculated as a candidate peak value. Subtract the peak value of the second peak Then divide the difference component by the candidate peak value. With the positive constant The sum of these values ​​yields the peak prominence index. The processing end uses preset weights and Consistency index of peak position With the main peak prominence index Weighting is performed, and a normalization penalty for peak position deviation is introduced when the peak positions are inconsistent, to obtain the displacement confidence score. The confidence level is calculated using the following formula:

[0102] ;

[0103] in, Indicates the first Displacement confidence of each candidate displacement peak. This represents the Sigmoid non-linear activation function, which takes real numbers as input and outputs a value between zero and one. The preset weights represent the peak position consistency index. Indicator of peak position consistency This indicates the preset weight of the peak prominence index. This indicates the prominence of the main peak. , These represent the horizontal and vertical components of the image peak coordinates, respectively. , These represent the horizontal and vertical components of the structural peak coordinates, respectively. This indicates a preset consistency threshold. This represents a positive constant used to avoid a denominator of zero. This indicates taking the absolute value; the processing end will assign the displacement confidence level. With candidate displacement peak displacement coordinates Associated storage;

[0104] Finally, displacement determination peaks are selected, and the processing unit considers the candidate displacement peak set. The confidence level of the internal screening displacement meets the preset screening threshold. The candidate shift peaks form a credible candidate set. When the set of credible candidates When not empty, the processing end selects from the set of trusted candidates. The candidate displacement peak with the highest displacement confidence level is selected as the displacement determination peak. and output displacement determination peak The displacement coordinates are used as the center input for subsequent sub-pixel localization, when the reliable candidate set When empty, the processing end selects from the candidate displacement peak set. The candidate displacement peak with the highest displacement confidence level is selected as the displacement determination peak. and the displacement determination peak Inputs marked as low confidence are used for subsequent confidence threshold determination.

[0105] In this embodiment, step S4 specifically includes:

[0106] In one implementation, step S4 is the joint phase correlation map output in step S3. Displacement determination peak displacement coordinates and displacement search range parameters As input, the output is the fine image shift vector. ,in, Includes horizontal displacement search radius Search radius of displacement in the vertical direction Joint phase correlation plot The displacement coordinate system takes the zero displacement after coarse alignment as its origin. shaft and The axes correspond to horizontal and vertical displacements, respectively.

[0107] First, a local refinement window is captured, and the processing end determines the displacement coordinates of the peak based on the displacement. As the center, based on the preset refined window size In the joint phase correlation diagram The boundary index of the local refinement window is determined in the middle, and the preset refinement window size is... Used to limit the local refinement window. direction and The window width in the direction is determined by the processing end based on the displacement search range parameter. Determine the boundary index of the allowable displacement range; the horizontal boundary of the allowable displacement range is... The boundary in the vertical direction is The processing unit clips the boundary indices of the local refinement window to within the boundary indices of the allowable displacement range, thus obtaining the clipped local refinement window. This allows for localized window refinement. Parameters that cover the neighborhood of the displacement determination peak and do not cross the displacement search range Limited displacement range;

[0108] Subsequently, boundary attenuation weighting is performed, and the processing end performs local refinement within the window. Generate boundary attenuation weight map in the displacement coordinate system The boundary attenuation weight map With local refinement window With the same spatial dimensions, the processing end sets the weight corresponding to the center position of the local refinement window to one, and sets the weights corresponding to the four boundary positions of the local refinement window to zero. The processing end calculates the horizontal and vertical distances from each position within the local refinement window to the center position, and normalizes these horizontal and vertical distances according to the half-window width of the local refinement window in the corresponding direction to obtain normalized distances. The processing end uses linear decay to map the normalized distances to weight values, so that the weight values ​​monotonically decrease as the normalized distance increases. The processing end then maps the boundary decay weights. Multiply position by position to the local refinement window Obtain the weighted local refinement window This makes the contribution of the edge region of the local refinement window to peak localization smaller than that of the central region;

[0109] Next, interpolation upsampling is performed, and the processing end refines the weighted local window. Use preset upsampling factor Interpolation upsampling is performed to obtain a local refinement window with sub-pixel resolution. The preset upsampling factor For integers greater than one, the processing end uses bilinear interpolation or cubic interpolation. direction and The direction generates denser sampling points, enabling local refinement windows at sub-pixel resolution. The sampling interval in the displacement coordinate system becomes the original sampling interval. And maintain the displacement coordinates corresponding to the center of the local refinement window and the displacement coordinates of the displacement judgment peak. Consistent;

[0110] Finally, peak coordinate determination and fine image shift vector generation are performed, with the processing end operating within a sub-pixel resolution local refinement window. The location of the maximum value within the search is used as the peak coordinate. ,in This represents the peak coordinate in the subpixel resolution displacement coordinate system. When multiple maximum values ​​exist, the maximum value position with the smallest distance from the center of the local refinement window is selected as the peak coordinate. The processing end will use the peak coordinates Converted to displacement coordinates relative to the displacement determination peak subpixel displacement offset The subpixel displacement offset The index offset of the peak position relative to the center of the local refinement window divided by the preset upsampling factor The processing end obtains the sub-pixel displacement offset. The horizontal component is denoted as Subpixel displacement offset The vertical component is denoted as The processing end will determine the displacement coordinates of the displacement judgment peak. With subpixel displacement offset Superposition yields the fine image shift vector. Among them, the fine image shift vector The horizontal component is The vertical component is and the fine image shift vector The output is used in subsequent trusted threshold determination steps.

[0111] In this embodiment, step S5 specifically includes:

[0112] In one implementation, step S5 uses the displacement confidence score output in step S3 and the coarse image shift vector output in step S2. The fine image shift vector output in step S4 As input, the final output is the image shift vector. With quality mark ,in, This represents the coarse image shift vector, which includes the horizontal component. With vertical component , This represents the fine image shift vector, which includes the horizontal component. With vertical component , This represents the final image shift vector, which includes the horizontal component. With vertical component The displacement confidence level is denoted as , The confidence scores of candidate displacement peaks corresponding to the displacement determination peaks are taken, and the preset confidence threshold is denoted as . The preset consistency threshold is denoted as , Includes horizontal consistency threshold With vertical consistency threshold Quality Marker Binary label;

[0113] First, a confidence level determination is performed, and the processor reads the displacement confidence level. , displacement confidence Compared with the preset confidence threshold When comparing, Greater than or equal to At that time, the credibility determination result will be... Set to satisfy, when Less than At that time, the credibility determination result will be... If the condition is not met, the processing end will determine the reliability of the result. This serves as a gating condition for subsequent consistency determination and final image shift vector selection;

[0114] Subsequently, in the credibility determination results To ensure consistency during execution, the processing unit calculates the fine image shift vector. With coarse image shift vector The absolute value of the difference between the horizontal and vertical components is used to obtain the absolute value of the horizontal difference. absolute value of vertical difference ,in, Through the Take the absolute value. Through the The absolute value is obtained, and the processing end will... With horizontal consistency threshold Compare and With vertical consistency threshold When comparing, Not greater than and Not greater than At that time, the consistency determination result will be... Set to satisfy, when Greater than or Greater than At that time, the consistency determination result will be... If the condition is not met, the processing end will determine the reliability of the result. If the consistency requirement is not met, the consistency determination will not be performed, and the consistency determination result will be directly passed to the public. Set it to not satisfy, so that subsequent selection links do not depend on unexecuted intermediate data;

[0115] Next, the final image shift vector is determined, and the processing end uses the confidence judgment result. To satisfy and consistent determination results To satisfy the condition, the fine image shift vector is... Determined as the final image shift vector and order , The processing end determines the credibility of the result. For results that do not meet or are inconsistent If the condition is not met, the coarse image shift vector will be adjusted. Determined as the final image shift vector and order , The processing end will ultimately shift the image vector. It maintains synchronized output with its identified source, enabling subsequent users to interpret the reliability of the image shift output based on the identified source;

[0116] Finally, a quality marker is generated, and the processing unit uses the final image shift vector... Determining the source generates quality tags When the final image shift vector From the fine image shift vector When determined, mark the quality. Set as valid when the final image shift vector From coarse image shift vector When determined, mark the quality. If set to invalid, the processor will change the final image shift vector. With quality mark The final output of the same frame pair is written to the buffer or output interface for subsequent imaging compensation or trajectory stabilization control.

[0117] In this embodiment, step S6 specifically includes:

[0118] In one implementation, step S6 uses the final image shift vector output in step S5. With quality mark As input, generate and output compensation control quantity. To the compensation actuator of the coded aperture imaging system, wherein the final image shift vector Includes horizontal components With vertical component Quality Marker It is a binary tag, and the preset valid tag value is denoted as . The result of the compensation mode determination is recorded as The preset image shift to the control mapping parameter is denoted as Compensation control quantity It is a two-dimensional control vector that includes a horizontal control component. With vertical control component The preset control amplitude boundary is the component level boundary, including... , , , The zero-compensation control quantity is denoted as Both of its components are zero.

[0119] First, a compensation mode determination is performed, and the processing unit reads the quality flag. quality mark Compared with the preset valid flag value When comparing, At that time, the compensation mode determination result will be... Set to valid when At that time, the compensation mode determination result will be... Set to invalid, and the processing end will determine the compensation mode result. As a gating condition for the generation and output of the compensation control quantity, the compensation actuator receives non-zero control input only when the quality mark is valid;

[0120] Subsequently, the compensation mode determination result To generate a compensation control quantity when effective, the processing end reads the final image shift vector. horizontal components With vertical component And use preset image shift to control mapping parameters The image shift is mapped to a control quantity, and the preset image shift to control mapping parameters are used. Including linear mapping matrix With control bias vector , where the linear mapping matrix for A real matrix is ​​used to convert pixel displacements into two-axis control values ​​for the compensation actuator, controlling the bias vector. This is a two-dimensional real vector used to compensate for the zero-position drift of the actuator. The processing end first performs a linear mapping to obtain the intermediate control vector. Reading the linear mapping matrix The two coefficients in the first row are respectively with , Multiply and sum to get The horizontal component is read from the linear mapping matrix. The two coefficients in the second row are respectively with , Multiply and sum to get The vertical component, the processing end then controls the bias vector Superimposed on intermediate control vector Obtain the compensation control quantity and will compensate control amount Decomposed into horizontal control components With vertical control component The preset image shift to control mapping parameters The system calibration process involves applying multiple sets of known control variables to the compensation actuator and collecting corresponding image shift measurements. A linear mapping matrix is ​​then fitted based on the correspondence between the control variables and the image shift measurements. Image shift measurements are acquired while the compensation actuator maintains zero control input to estimate the control bias vector. ;

[0121] Next, amplitude limiting processing is performed and output, and the processing end controls the compensation quantity. Perform component-level amplitude limiting processing: when When Replace with ,when When Replace with ,when When Replace with ,when When Replace with The processing end will limit the amplitude. Recombined into compensating control quantities and will compensate control amount The control interface outputs to the compensation actuator of the coded aperture imaging system, causing the compensation actuator to perform image shift compensation according to a defined amplitude.

[0122] Finally, the result was determined based on the compensation mode. When the output is invalid, a zero-compensation control quantity is output, and the processing end sets the compensation control quantity to the zero-compensation control quantity. and zero-compensation control quantity The output is sent to the compensation actuator, which keeps the compensation actuator stationary or returns to the zero control state when the quality mark is invalid, thus avoiding the introduction of unreliable final image shift vectors into the compensation closed loop.

[0123] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for determining short-time image shift in coded aperture imaging based on a ConvNeXt coarse image shift estimation network and constrained phase correlation, characterized in that, include: S1. Receive continuous imaging frames from the coded aperture imaging system, select the previous frame as the reference coded aperture image and the next frame as the coded aperture image to be tested, and determine the effective matching area within the effective field of view of the coded modulation. S2. Within the effective matching region, the reference coded aperture image and the coded aperture image to be tested are input into the ConvNeXt coarse image shift estimation network in a paired manner. The ConvNeXt backbone is used to extract cross-frame multi-scale features. Based on the cross-frame multi-scale features, the cross-frame feature differences are calculated and structural guidance information is formed. The structural guidance information is used to perform weighted convergence of the cross-frame multi-scale features to generate the reference structural salience feature map and the salience feature map to be tested. Based on the weighted converged cross-frame multi-scale features, the coarse image shift vector is obtained by regression. The displacement search range parameters are determined according to the coarse image shift vector. S3. Based on the coarse image shift vector and displacement search range parameters, coarsely align the coded aperture image to be tested with the spurious feature map of the structure to be tested. Within the displacement range defined by the displacement search range parameters, calculate the image phase correlation map between the reference coded aperture image and the coded aperture image to be tested, and calculate the structural phase correlation map between the reference structural spurious feature map and the spurious feature map of the structure to be tested. Fuse the image phase correlation map and the structural phase correlation map in the same displacement coordinate system to obtain a joint phase correlation map. Extract the displacement peak list from the joint phase correlation map. Calculate the peak position consistency index based on the peak position deviation between the image phase correlation map and the structural phase correlation map. Calculate the main peak prominence index based on the difference between the main peak and the secondary peak in the joint phase correlation map. Calculate the displacement confidence score using the peak position consistency index and the main peak prominence index, and filter to obtain the displacement judgment peaks. S4. Determine the local refinement window using the displacement judgment peak, perform sub-pixel localization on the joint phase correlation map within the local refinement window, and obtain the refined image shift vector; S5. Determine the final image shift vector based on the displacement confidence level and the preset confidence threshold. If the displacement confidence level meets the confidence threshold, use the fine image shift vector. If the displacement confidence level does not meet the confidence threshold, use the coarse image shift vector and generate a quality label. S6. Based on the quality mark, the compensation mode is determined. When the quality mark is valid, a compensation control quantity is generated according to the final image shift vector. When the quality mark is invalid, a zero compensation control quantity is generated.

2. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 1, characterized in that, S1 specifically refers to: Receive continuous imaging frames from the coded aperture imaging system, and select two adjacent frames in chronological order under the same pixel coordinate system as the processing objects; The two adjacent frames with the earlier time are set as reference coded aperture images, and the two adjacent frames with the later time are set as coded aperture images to be tested, while keeping the height, width and number of channels of both consistent. The effective field of view of the coded aperture modulator is determined based on the calibration parameters of the coded aperture modulator, and the effective field of view of the coded aperture modulator is mapped to the pixel coordinate system of the reference coded aperture image and the coded aperture image to be tested to form the effective field of view region; Based on the effective field of view region and combined with the preset maximum displacement boundary, the effective field of view region is shrunken to obtain the effective matching region. The reference coded aperture image and the coded aperture image to be tested are cropped within the effective matching region as the pairing input data for the subsequent ConvNeXt coarse image shift estimation network.

3. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 1, characterized in that, S2 specifically refers to: The reference coded aperture image and the coded aperture image to be tested within the effective matching area are stitched together along the channel dimension to form a paired input tensor, which is then input into the ConvNeXt coarse image shift estimation network. The ConvNeXt backbone is used to perform staged convolutional feature extraction on paired input tensors. In each stage, the staged convolutional feature extraction sequentially executes a downsampling layer and several ConvNeXt blocks. The ConvNeXt blocks sequentially include a depthwise convolutional layer, a normalization layer, a pointwise convolutional expansion layer, a nonlinear activation unit, a pointwise convolutional compression layer, and a residual connection, outputting cross-frame multi-scale features with multiple downsampling scales. The cross-frame multi-scale features at each scale are input into the cross-frame feature difference calculation branch. First, the cross-frame multi-scale features are separated into reference corresponding features and test corresponding features according to the channel source of the paired input tensor. Then, position-by-position difference operation is performed on the reference corresponding features and test corresponding features to obtain the difference feature map at each scale. The differential feature map is input into the structure guidance information to generate branches. Convolutional layers and nonlinear activation units are used to map the differential feature map, and the output structure guidance information with the same scale as the differential feature map is output. The structure guidance information is a single-channel weight map. The cross-frame multi-scale features and structural guidance information are input into the weighted convergence module. The cross-frame multi-scale features are weighted position by position, and the features at different scales are aligned with resolution before cross-scale convergence is performed to obtain the weighted converged cross-frame multi-scale features. The weighted and aggregated multi-scale features across frames are input into the structure salience map generation head. The number of feature channels is compressed using pointwise convolutional layers, and upsampling and fusion are used to restore the multi-scale features to the spatial resolution of the effective matching region. The output is a reference structure salience map and a test structure salience map. The weighted and converged cross-frame multi-scale features are input into the coarse image shift vector regression head and the displacement search range parameter regression head. A global convergence layer is used to generate a fixed-length feature vector, which is then output as a coarse image shift vector and displacement search range parameter through a fully connected regression layer. Boundary constraints are applied to the displacement search range parameter based on the coarse image shift vector to ensure that the displacement search range parameter falls within the preset maximum displacement boundary.

4. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 3, characterized in that, When the structure guidance information generation branch maps the differential feature map and outputs structure guidance information, the weight values ​​of the structure guidance information at the pixel positions are calculated using the weight calculation formula: ； in, Represents structural guidance information At pixel position The weight value at that location, Indicates the first Differential feature maps at various scales, The structure guides the information generation branch in the first... Convolutional kernel parameters at various scales, Indicates and The corresponding bias parameter, symbol This represents the convolution operation. Indicates the reference corresponding feature at position The channel vector value at that location. Indicates the location of the corresponding feature to be tested. The channel vector value at that location, sign This represents the dot product operation of channel vectors. This represents the L2 norm operation of the channel vector. This indicates a Sigmoid nonlinear activation. This represents the similarity scaling factor. This represents a positive constant used to avoid a denominator of zero, calculated by the processing end as described above. A weight value is provided at each spatial location to indicate that the location is more likely to belong to a target structural region that moves consistently across frames.

5. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 1, characterized in that, S3 specifically refers to: A coarse-aligned coded aperture image is obtained by performing a translation transformation on the image to be tested based on the coarse image shift vector, and a coarse-aligned coded aperture image is obtained by performing a translation transformation on the spur feature map of the structure to be tested using the same displacement as the translation transformation, thereby limiting the data after the translation transformation to the effective matching region; Within the displacement range defined by the displacement search range parameter, constrained phase correlation calculation is performed using the reference coded aperture image and the coarsely aligned coded aperture image to be tested as input. The input image is then subjected to discrete Fourier transform, the normalized cross-frame frequency domain correlation quantity is calculated, and an inverse transform is performed to obtain an image phase correlation map covering the displacement range. Within the displacement range defined by the displacement search range parameter, constrained phase correlation calculation is performed using the reference structure salient feature map and the coarsely aligned salient feature map of the structure to be tested as inputs. The input feature maps are then subjected to discrete Fourier transform, normalized cross-frame frequency domain correlation quantities are calculated, and inverse transform is performed to obtain a structural phase correlation map covering the displacement range. The image phase correlation map and the structure phase correlation map are mapped to the same displacement coordinate system, and the image phase correlation map and the structure phase correlation map are weighted and fused using a preset fusion weight to obtain a joint phase correlation map; Local maxima detection is performed on the joint phase correlation map to obtain a list of displacement peaks. After sorting the list of displacement peaks by peak intensity, candidate displacement peaks corresponding to a preset number of peaks are retained. For each candidate displacement peak, the peak position deviation is calculated based on the peak coordinates in the image phase correlation map and the peak coordinates in the structural phase correlation map. A peak position consistency index is generated based on the peak position deviation and a preset consistency threshold. A main peak prominence index is generated based on the main peak value and the secondary peak value in the joint phase correlation map. The peak position consistency index and the main peak prominence index are weighted by a preset weight to obtain the displacement confidence. Select the candidate displacement peak that meets the preset screening threshold and has the highest displacement confidence from the displacement peak list as the displacement judgment peak.

6. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 5, is characterized in that, The confidence level of displacement is calculated using the confidence level calculation formula: ； in, Indicates the first Displacement confidence of each candidate displacement peak. This represents the Sigmoid non-linear activation function, which takes real numbers as input and outputs a value between zero and one. The preset weights represent the peak position consistency index. Indicator of peak position consistency This indicates the preset weight of the peak prominence index. This indicates the prominence of the main peak. , These represent the horizontal and vertical components of the image peak coordinates, respectively. , These represent the horizontal and vertical components of the structural peak coordinates, respectively. This indicates a preset consistency threshold. This represents a positive constant used to avoid a denominator of zero. This indicates taking the absolute value; the processing end will assign the displacement confidence level. With candidate displacement peak displacement coordinates Associated storage.

7. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 1, characterized in that, S4 specifically refers to: Centered on the displacement coordinates corresponding to the displacement judgment peak, a local refinement window is extracted from the joint phase correlation diagram according to the preset refinement window size, and the spatial range of the local refinement window is cropped to the displacement range corresponding to the displacement search range parameter. Boundary attenuation weighting is applied to the joint phase correlation map within the local refinement window, so that the weight of the edge region of the local refinement window is less than the weight of the center region; The joint phase correlation map within the weighted local refinement window is interpolated and upsampled using a preset upsampling factor to obtain a local refinement window with sub-pixel resolution. Within a local refinement window at subpixel resolution, the peak coordinates are determined and converted into a subpixel displacement offset relative to the displacement determination peak. The displacement coordinates of the displacement determination peak are then superimposed with the subpixel displacement offset to obtain the refined image shift vector.

8. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 1, characterized in that, S5 specifically refers to: Read the displacement confidence level and compare it with the preset confidence threshold to obtain the confidence judgment result; When the credibility determination result is satisfied, the absolute values ​​of the differences between the fine image shift vector and the coarse image shift vector in the horizontal and vertical components are taken respectively, and compared with the preset consistency threshold to obtain the consistency determination result. When both the credibility and consistency judgment results are satisfied, the fine image shift vector is determined as the final image shift vector; when either the credibility or consistency judgment results are not satisfied, the coarse image shift vector is determined as the final image shift vector. A quality marker is generated based on the source of the final image shift vector. The quality marker is set to valid when the final image shift vector is determined by the fine image shift vector, and set to invalid when the final image shift vector is determined by the coarse image shift vector.

9. The short-time image shift determination method for coded aperture imaging based on the ConvNeXt coarse image shift estimation network and constrained phase correlation as described in claim 1, characterized in that, Step S6 is as follows: Read the quality marker and compare it with the preset valid marker value to obtain the compensation mode determination result; When the compensation mode determination result is valid, the final image shift vector is used as input and the compensation control quantity is generated using the preset image shift to control mapping parameters; The compensation control quantity is subjected to amplitude limiting processing to make the compensation control quantity fall within the preset control amplitude boundary, and the amplitude-limited compensation control quantity is output to the compensation execution mechanism of the coded aperture imaging system. When the compensation mode determination result is invalid, the compensation control quantity is set to zero and output to the compensation execution agency.

Images