Low altitude aircraft visual blind area reconstruction system based on adaptive image extrapolation

Abstract

The application discloses a low-altitude aircraft visual blind area reconstruction system based on adaptive image deduction, and relates to the technical field of low-altitude aircraft visual image processing. The system aims at the dynamic occlusion blind area formed by the structure of the aircraft entering the field of view of the airborne camera, carries out attitude compensation on the static mask of the aircraft body occlusion according to the current frame and the flight attitude vector, obtains the dynamic occlusion mask, the dynamic occlusion boundary and the blind area to be reconstructed, projects the boundary local flow vector to the normal vector in the blind area, obtains the inflow type or the divergence type determination result, executes the historical frame time reversal sampling when the inflow type is adopted, executes the boundary neighborhood sampling when the divergence type or the sampling fails, and modifies the next frame local deduction main direction, the historical frame extrapolation weight and the boundary neighborhood priority reconstruction boundary section according to the direction consistency error, the brightness continuity error and the time reversal failure rate respectively, and finally outputs the final frame.

Description

Technical Field

[0001] This invention relates to the field of visual image processing technology for low-altitude aircraft, and more specifically, to a visual blind spot reconstruction system for low-altitude aircraft based on adaptive image inference. Background Technology

[0002] With the increasing application of low-altitude aircraft in scenarios such as inspection, aerial photography, emergency reconnaissance, near-ground mapping, and low-altitude operations, airborne vision systems typically rely on cameras to continuously acquire environmental images during flight. These images are then combined with attitude sensors, inertial measurement units, historical frame images, or image processing algorithms to stabilize the current field of view, identify targets, and assist in perception. In practical applications, the wings, arms, landing gear, gimbal supports, fuselage edges, or external sensors of low-altitude aircraft may enter the camera's field of view, creating localized occlusion areas. To address image occlusion or partial loss, existing technologies typically employ fixed occlusion area removal, neighbor pixel filling, historical frame reference compensation, optical flow estimation, image registration, video restoration, or deep learning-based image completion methods to repair or visually enhance the occluded areas, maintaining the continuity and readability of the video stream as much as possible. These methods are effective in general image restoration, video stabilization, or static occlusion completion scenarios and have become common image processing techniques in visual perception systems.

[0003] However, the visual blind zone of low-altitude aircraft is not fixed; its occlusion range dynamically changes with pitch, roll, yaw, vibration, and maneuvering flight states. Simultaneously, directional textures such as power lines, runway lines, road edges, and building outlines in low-altitude scenes often enter or leave the occlusion boundary from the truly visible area as the aircraft moves. Existing processing methods based on fixed masks, ordinary neighborhood completion, or conventional historical frame references often struggle to accurately distinguish whether boundary textures are entering or leaving the blind zone, and they also find it difficult to establish stable texture extension criteria for dynamic occlusion boundaries. This easily leads to problems such as edge breaks, directional misalignments, texture trailing, abrupt brightness changes, or incorrectly extrapolating historical textures that have already left the boundary into the blind zone during reconstruction. Therefore, under the dynamic attitude change conditions of low-altitude aircraft, this invention proposes a technical solution that can adaptively reconstruct the visual blind zone by combining dynamic occlusion boundaries and boundary texture movement trends, thereby improving the directional continuity, temporal consistency, and output stability of the blind zone completion results. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a visual blind spot reconstruction system for low-altitude aircraft based on adaptive image inference, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: The low-altitude aircraft visual blind zone reconstruction system based on adaptive image inference includes the following modules: a boundary calibration module, which compensates for the static mask of fuselage occlusion based on the current frame and flight attitude vector to obtain the dynamic occlusion mask, dynamic occlusion boundary and blind zone to be reconstructed; The inflow determination module uses the dynamic occlusion boundary sampling point as the determination unit. Based on the current frame and historical frames, it obtains the local optical flow vector and the normal vector in the blind zone, calculates the projection of the local optical flow vector onto the normal vector, and obtains the inflow type or deviation type local inflow determination result. Based on the blind zone inflow projection amount, it obtains the historical frame extrapolation weight corresponding to the boundary sampling point, and uses the normalization direction of the optical flow vector as the main direction of local inference. The directional synthesis module determines the boundary starting point of the pixel to be reconstructed and associates it with the corresponding boundary sampling point. It selects the pixel reconstruction path based on the local inflow determination result and the historical frame extrapolation weight. When the local inflow determination result supports historical frame extrapolation and the historical frame inversion sampling is valid, the pixel channel value of the real visible sampled pixel in the historical frame is read along the opposite direction of the local optical flow vector as the pixel reconstruction value. Otherwise, the pixel reconstruction value is generated from the visible pixels in the neighborhood of the boundary starting point to form the blind zone reconstruction enhancement layer image. The closed-loop correction module obtains the directional consistency error, brightness continuity error, and temporal inversion failure rate based on the continuity error between the blind zone reconstruction enhancement layer image and the real visible side region of the dynamic occlusion boundary. When the directional consistency error exceeds the limit, it corrects the main direction of local extrapolation in the next frame. When the temporal inversion failure rate exceeds the limit, it reduces the extrapolation weight of the historical frame in the next frame. When the brightness continuity error exceeds the limit, it marks the boundary neighborhood and prioritizes the reconstruction of the boundary segment. The final output frame is obtained by fusion.

[0006] In a preferred embodiment, the boundary calibration module obtains the current frame's dynamic occlusion mask and dynamic occlusion boundary as follows: It calculates the time difference between the camera sampling time and the inertial measurement unit (IMU) sampling time. When the time difference is not greater than the time synchronization threshold, it directly uses the flight attitude vector as the attitude label for the current frame. When the time difference is greater than the time synchronization threshold, it performs linear interpolation on the pitch, roll, and yaw angles of two adjacent IMU sampling points before and after the camera sampling time to obtain the flight attitude vector corresponding to the current frame. Based on the change in the flight attitude vector relative to the standard attitude, it interpolates the horizontal displacement, vertical displacement, and boundary angle correction of the fuselage occlusion static mask using adjacent attitude grid points in the airframe structure calibration table corresponding to the current aircraft model, camera installation position, lens distortion correction status, and mounting structure. It then performs image coordinate transformation on the fuselage occlusion static mask using these displacements and corrections to obtain the current frame's dynamic occlusion mask. The airframe structure calibration table also records... When there is a boundary point coordinate offset, the boundary point coordinate offset corresponding to the current frame is obtained by interpolation according to the adjacent attitude grid points, and the boundary is corrected for the dynamic occlusion mask of the current frame based on the interpolated boundary point coordinates. In the dynamic occlusion mask of the current frame obtained by compensating the static occlusion mask of the fuselage through the fuselage structure calibration table, the outer contour representing the fuselage occlusion area is extracted as a candidate dynamic occlusion boundary. When the average displacement of the candidate dynamic occlusion boundary relative to the dynamic occlusion boundary of the previous frame is not greater than the boundary stability threshold, the candidate dynamic occlusion boundary is used as the current dynamic occlusion boundary. When the average displacement is greater than the boundary stability threshold, the boundary points of the candidate dynamic occlusion boundary and the dynamic occlusion boundary of the previous frame are resampled and weighted and fused to obtain the current dynamic occlusion boundary. The connected region representing the fuselage occlusion area in the dynamic occlusion mask of the current frame is determined as the blind area to be reconstructed, and on the side of the dynamic occlusion boundary away from the blind area to be reconstructed, a real visible pixel area is selected according to the preset boundary reference width to generate the boundary reference area.

[0007] In a preferred embodiment, the inflow determination module obtains the local optical flow vector as follows: A single outer contour pixel on the dynamic occlusion boundary is used as a boundary sampling point; a preset local spatiotemporal sampling window is set with the boundary sampling point as the center. The preset local spatiotemporal sampling window includes a local spatial window size and a local temporal window length, with the local temporal window length corresponding to the current frame and several historical frames preceding the current frame; the current frame and historical frames covered by the preset local spatiotemporal sampling window are preprocessed to obtain a brightness map for gradient calculation; pixels located in the corresponding frame's truly visible area, not exceeding the image range, and with unsaturated brightness are used as valid pixels; horizontal gradient, vertical gradient, and temporal gradient are calculated at the valid pixels; the accumulated correlation components between the horizontal gradient, vertical gradient, and temporal gradient are weighted and summed to obtain the critical region spatiotemporal gradient tensor matrix; a local minimum error equation is established based on the spatial structure component and spatiotemporal coupling component in the critical region spatiotemporal gradient tensor matrix; the horizontal and vertical velocities are solved to obtain the horizontal and vertical velocities, and the horizontal and vertical velocities are combined into a local optical flow vector.

[0008] In a preferred embodiment, the inflow determination module obtains the blind zone inflow projection amount and the historical frame extrapolation weight in the following manner: the local tangent direction at the boundary sampling point is determined by the positional change of adjacent outer contour pixels before and after the boundary sampling point; the side perpendicular to the local tangent direction and pointing to the inside of the current frame dynamic occlusion mask is determined as the normal vector in the blind zone; the dot product of the local optical flow vector and the normal vector in the blind zone is divided by the sum of the vector norm of the local optical flow vector and the preset stability constant to obtain the blind zone inflow projection amount; the sum of the absolute values ​​of the blind zone inflow projection amount and the blind zone inflow projection amount is divided by 2 to obtain the historical frame extrapolation weight.

[0009] In a preferred embodiment, the inflow determination module outputs the blind zone inflow determination result as follows: when the blind zone inflow projection amount is greater than the blind zone inflow determination threshold, the vector norm of the local optical flow vector is not less than the minimum velocity modulus threshold, and the spatial gradient intensity of the effective pixels within the preset local spatiotemporal sampling window is not lower than the low texture determination threshold, an inflow-type blind zone inflow determination result is output, and the pixel to be reconstructed associated with the boundary sampling point is allowed to enter the historical frame time inversion sampling path; when the blind zone inflow projection amount is less than or equal to the blind zone inflow determination threshold, or the vector norm of the local optical flow vector is less than the minimum velocity modulus threshold, or the spatial gradient intensity of the effective pixels within the preset local spatiotemporal sampling window is lower than the low texture determination threshold, a divergence-type blind zone inflow determination result is output, the historical frame extrapolation weight is updated to zero, and the pixel to be reconstructed associated with the boundary sampling point is prohibited from entering the historical frame time inversion sampling path, causing it to switch to the boundary starting point neighborhood sampling path.

[0010] In a preferred embodiment, the directional synthesis module determines the boundary start point and direction extrapolation channel corresponding to the pixel to be reconstructed as follows: For any pixel to be reconstructed within the blind zone, perform pixel-level backtracking from the pixel to be reconstructed along the opposite direction of the local extrapolation main direction. The intersection point with the dynamic occlusion boundary is taken as the boundary start point, and the boundary sampling point closest to the boundary start point is taken as the associated boundary sampling point. The local inflow determination result, local extrapolation main direction, and historical frame extrapolation weight corresponding to the associated boundary sampling point are read. If no valid intersection point can be obtained, the closest point on the dynamic occlusion boundary to the pixel to be reconstructed is selected. The boundary sampling point is used as the boundary starting point, and the pixel to be reconstructed is marked as the nearest neighbor matching pixel. The distance from the starting point of the boundary to the blind zone of the pixel to be reconstructed is measured along the main direction of local inference as the longitudinal projection depth. The deviation of the pixel to be reconstructed from the main direction of local inference is measured along the vertical direction perpendicular to the main direction of local inference as the lateral offset. When the longitudinal projection depth is not less than zero and the lateral offset is not greater than the sum of the boundary basic channel width and the channel lateral growth coefficient multiplied by the longitudinal projection depth, the pixel to be reconstructed is determined to belong to the direction inference channel. Otherwise, the pixel to be reconstructed is determined not to belong to the direction inference channel.

[0011] In a preferred embodiment, the directional synthesis module locates historical frame sampling pixels as follows: when the local inflow determination result corresponding to the associated boundary sampling point is inflow type, the historical frame extrapolation weight is greater than the preset extrapolation weight threshold, and the pixel to be reconstructed belongs to the directional extrapolation channel, historical frame time inversion sampling is allowed; the projection modulus of the local optical flow vector in the local extrapolation main direction is taken as the effective inflow velocity, and the longitudinal projection depth is divided by the effective inflow velocity and rounded up to obtain the time inversion step number; when the effective inflow velocity is less than the minimum effective inflow velocity threshold, or the time inversion step number is greater than the maximum number of frames supported by the historical frame buffer, The pixel to be reconstructed is marked as a failed inversion pixel. When the effective inflow velocity is not less than the minimum effective inflow velocity threshold and the number of time inversion steps is not greater than the maximum number of frames supported by the historical frame buffer, the displacement corresponding to the number of time inversion steps is traced back from the image coordinates of the current pixel to be reconstructed along the opposite direction of the local optical flow vector to obtain the historical frame sampled pixel in the time inversion sampling frame. If the historical frame sampled pixel is located in the occluded area of ​​the time inversion sampling frame or exceeds the image range, the current pixel to be reconstructed is marked as a failed inversion pixel. If the historical frame sampled pixel is located in the truly visible area, the pixel channel value of the historical frame sampled pixel in the truly visible state is read.

[0012] In a preferred embodiment, the directional synthesis module performs boundary start point neighborhood sampling on pixels to be reconstructed whose local inflow determination result is divergent, pixels to be reconstructed whose historical frame extrapolation weight is not greater than a preset extrapolation weight threshold, pixels to be reconstructed outside the directional inference channel, nearest neighbor matching pixels, and pixels that failed inversion. The boundary start point neighborhood sampling includes: selecting real visible and unsaturated effective pixels in the boundary reference region within a preset boundary sampling radius, centered on the boundary start point corresponding to the pixel to be reconstructed; taking the arithmetic mean of the pixel channel values ​​of the effective pixels as the pixel reconstruction value of the pixel to be reconstructed; when the current frame is a color image frame, performing neighborhood sampling on the luminance channel value and chrominance channel value respectively; generating the pixel reconstruction value of each pixel to be reconstructed in the blind zone in order of increasing vertical projection depth, and writing the pixel reconstruction value into an image buffer of the same size as the dynamic occlusion mask of the current frame according to its position to obtain the blind zone reconstruction enhancement layer image.

[0013] In a preferred embodiment, the closed-loop correction module obtains the continuity error as follows: Ring-shaped verification windows with a preset verification window width are set on both sides of the dynamic occlusion boundary. The verification window inside the blind zone is located on one side of the blind zone to be reconstructed and is used to read local information of the reconstruction boundary in the blind zone reconstruction enhancement layer image. The verification window on the truly visible side is located on one side of the boundary reference region and is used to read local information of the true boundary in the current frame. The horizontal and vertical gradients are calculated for the boundary sampling points within each verification window. The gradient directions of each pixel are weighted and summed according to their gradient magnitudes and then normalized. The blind zone inner reconstruction direction vector and the truly visible side verification window are obtained. The direction vector of the truly visible side is obtained, and the magnitude of the difference between the reconstructed direction vector inside the blind zone and the true visible side direction vector is taken as the direction consistency error. The reconstructed brightness inside the blind zone is obtained by arithmetically averaging the brightness channel values ​​of the reconstructed enhancement layer image inside the blind zone within the verification window. The true visible side brightness is obtained by arithmetically averaging the brightness channel values ​​of the current frame within the verification window. The brightness continuity error is obtained by dividing the absolute value of the difference between the reconstructed brightness inside the blind zone and the true visible side brightness by the sum of the true visible side brightness and the preset stability constant. The temporal inversion failure rate is obtained by statistically analyzing the proportion of inversion failure pixels in the total pixels of the blind zone to be reconstructed.

[0014] In a preferred embodiment, the closed-loop correction module performs write-back correction on the local extrapolation main direction, historical frame extrapolation weights, and boundary segment priority reconstruction of the boundary neighborhood in the next frame in the following manner: When the direction consistency error is greater than the preset direction error upper limit, the current local inference main direction is fused with the actual visible side direction vector and then normalized to obtain the local inference main direction of the next frame; when the direction consistency error is not greater than the preset direction error upper limit, the current local inference main direction is used in the next frame. When the time inversion failure rate is greater than the preset failure rate limit, the extrapolation weight of the current historical frame is divided by the time inversion failure rate to obtain the extrapolation weight of the next historical frame; when the time inversion failure rate is not greater than the preset failure rate limit, the extrapolation weight of the current historical frame is maintained in the next frame. When the brightness continuity error exceeds the preset brightness error upper limit, the corresponding boundary sampling point is marked as a brightness excess sampling point, and adjacent brightness excess sampling points are connected and grouped according to the sampling order of the dynamic occlusion boundary to obtain a brightness excess boundary segment; when the number of sampling points in the brightness excess boundary segment is not less than the preset minimum number of continuous points, the brightness excess boundary segment is marked as a boundary neighborhood priority reconstruction boundary segment, so that the pixels to be reconstructed associated with the boundary segment in the next frame will preferentially use the visible pixels in the neighborhood of the boundary start point to generate pixel reconstruction values; when the brightness continuity error does not exceed the preset brightness error upper limit, the current reconstruction path is maintained in the next frame; After completing the above write-back correction, the current frame dynamic occlusion mask is used as the fusion basis. The original pixels of the current frame in the real visible area are retained. The occlusion area of ​​the machine is replaced with the reconstructed pixel values ​​of the same coordinates in the blind zone reconstruction enhancement layer image. The pixels at the dynamic occlusion boundary are linearly transitioned according to the boundary distance to obtain the final output frame.

[0015] The technical effects and advantages of this invention's low-altitude aircraft visual blind zone reconstruction system based on adaptive image inference are as follows: This invention uses a boundary calibration module to perform attitude compensation on the static mask of fuselage occlusion based on the current frame and flight attitude vector. This allows the position of the visual blind zone to be dynamically updated with the pitch, roll, and yaw changes of the low-altitude aircraft, avoiding the problem of inconsistency between the blind zone boundary and the actual occlusion position caused by using a fixed occlusion area. Furthermore, this application uses an inflow determination module to project the local optical flow vector near the dynamic occlusion boundary onto the normal vector within the blind zone, and determines whether the truly visible texture flows into or away from the blind zone. This allows subsequent reconstruction to no longer rely solely on optical flow amplitude, neighborhood smoothing, or historical frame replication, but to select a reconstruction strategy based on the actual movement trend of the texture relative to the blind zone boundary. When the blind zone inflow determination result is inflow type, this application uses a directional synthesis module to locate the historical frame sampling pixel of the texture carried by the pixel to be reconstructed in the historical frame along the opposite direction of the local optical flow vector, and then samples its truly visible state. The pixel channel values ​​under the current state are used as the basis for reconstruction. When the current frame is a single-channel image, the pixel channel value is the luminance channel value. When the current frame is a color image frame, the pixel channel value includes the luminance channel value and the chrominance channel value, so that directional textures such as wires, runway lines, road edges, and building outlines can continue in the blind zone along their true entry direction, reducing edge breaks, directional misalignment, and texture trailing. When the judgment result is a deviation type, the pixel to be reconstructed does not belong to the directional inference channel, or the historical frame sampling fails, the reconstruction value is generated based on the real visible pixel channel values ​​in the boundary reference area around the boundary starting point corresponding to the pixel to be reconstructed, avoiding the incorrect extrapolation of historical frame textures that deviate from the dynamic occlusion boundary into the blind zone to be reconstructed. Finally, this application uses a closed-loop correction module to write back and correct the local inference main direction and historical frame extrapolation weight of the next frame according to the continuity error between the reconstruction enhancement layer and the real visible areas on both sides of the dynamic occlusion boundary, thereby improving the directional continuity, temporal consistency, and output stability of blind zone reconstruction in continuous video streams. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the visual blind spot reconstruction system for low-altitude aircraft based on adaptive image inference according to the present invention. Figure 2 This is a schematic diagram of the dynamic occlusion boundary calibration of the present invention; Figure 3 This is a schematic diagram of the inflow determination based on the projection of the normal vector within the blind zone according to the present invention; Figure 4 This is a schematic diagram of blind zone reconstruction based on historical frame time inversion sampling according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example This embodiment discloses a visual blind spot reconstruction system for low-altitude aircraft based on adaptive image inference. It is applicable to scenarios where low-altitude aircraft are experiencing dynamic occlusion blind spots due to the entry of wings, arms, landing gear, fuselage edges, gimbal supports, or external sensors into the field of view of the onboard camera during near-ground flight, turning, pitching / climbing, or descent obstacle avoidance. The blind spot to be reconstructed is a dynamically occluded area formed by changes in the aircraft's own structure due to pitch angle, roll angle, yaw angle, aircraft vibration, or flight maneuvering. Subsequent boundary calibration, inflow determination, directional synthesis, and closed-loop correction are all performed around this dynamically occluded area.

[0019] Please see Figure 1 This invention discloses a visual blind spot reconstruction system for low-altitude aircraft based on adaptive image inference, comprising the following modules: Please see Figure 2 The boundary calibration module is used to convert the pre-calibrated static mask of fuselage occlusion into a dynamic occlusion region under the current flight attitude based on the current frame and flight attitude vector. This module obtains the dynamic occlusion mask, dynamic occlusion boundary, blind zone to be reconstructed, and boundary reference region of the current frame through time synchronization, attitude compensation, and boundary stabilization processing. The output of the boundary calibration module is used to provide accurate boundary sampling positions for the subsequent inflow determination module, so that subsequent image inference is no longer processed for a fixed occlusion template, but for the real occlusion boundary that changes with the flight attitude. In this embodiment, the current frame, historical frame and final output frame are all image frames in a continuous video stream. The current frame represents the image frame captured at the current moment, the historical frame represents the image frame captured before the current frame, and the final output frame represents the image frame written into the continuous video stream after blind zone reconstruction is completed. The boundary calibration module first reads the current frame captured by the airborne camera. Simultaneously, the flight attitude vector output by the inertial measurement unit is read. Current frame This represents the image frame captured by the airborne camera at time t in the video stream. The current frame includes the pixel matrix and the camera sampling time. Camera calibration parameters; flight attitude vector Including the sampling time of the inertial measurement unit And the corresponding pitch, roll and yaw angles. Pitch is used to indicate the degree to which the aircraft's nose pitches up or down relative to the horizontal plane, roll is used to indicate the degree to which the aircraft tilts left or right around the forward and backward axis of the aircraft, and yaw is used to indicate the degree to which the aircraft turns around the vertical axis. The system reads the camera sampling time. and inertial measurement unit sampling time And calculate the time difference between the two; when the time difference is not greater than the time synchronization threshold At that time, directly input the corresponding flight attitude vector As the current frame The attitude label; when the time difference is greater than the time synchronization threshold At that time, read the camera sampling time. The sampling points of two adjacent inertial measurement units are used, and linear interpolation is performed on the pitch, roll, and yaw angles to obtain the result relative to the current frame. Corresponding flight attitude vector Through this process, the boundary calibration module subsequently obtains a dynamic occlusion mask. Able to work with the current frame Maintaining time consistency; in this embodiment, the time synchronization threshold The initial value is set to 1ms; this initial value is obtained based on the frame rate of commonly used cameras and the sampling frequency of inertial measurement units for low-altitude aircraft. Its function is to limit the timing deviation between attitude data and image data and avoid dynamic occlusion of boundaries during rapid maneuvers. Location mismatch occurred; Then read the static mask obstructing the fuselage. ; static mask for fuselage obstruction This represents the fixed occlusion areas of the wings, arms, landing gear, fuselage edges, gimbal supports, or external sensors in an image when the low-altitude aircraft is in its standard attitude; a static mask representing fuselage occlusion. Generated during the system's factory calibration phase; during calibration, the aircraft camera is aligned with a uniform background, and reference images are acquired under standard attitude. A binary mask is obtained through occlusion region segmentation; in the binary mask, a pixel value of 1 represents the occluded area of ​​the aircraft, and a pixel value of 0 represents the truly visible area; static mask of the aircraft occlusion. Stored in accordance with aircraft model, camera installation location, and mounting structure; To avoid the dynamic occlusion boundary being interpreted as a pre-fixed boundary, in this embodiment, the fuselage occlusion static mask, the fuselage structure calibration table, and the current frame dynamic occlusion mask have a sequential input-output relationship: the fuselage occlusion static mask represents the reference occlusion position of the fuselage structure in the image under standard attitude; the fuselage structure calibration table records the horizontal displacement, vertical displacement, boundary angle correction, and / or boundary point coordinate offset corresponding to changes in flight attitude; the current frame dynamic occlusion mask is calculated by the boundary calibration module based on the output results of the fuselage occlusion static mask and the fuselage structure calibration table. Therefore, the current frame dynamic occlusion mask is not an arbitrary image defect mask directly detected, nor is it a manually specified fixed occlusion area, but rather a dynamic occlusion area of ​​the fuselage structure obtained by transforming the reference occlusion area under standard attitude with the current flight attitude.

[0020] After obtaining the flight attitude vector and fuselage obstruction static mask Subsequently, the boundary calibration module calls the airframe structure calibration table for attitude compensation; the airframe structure calibration table records the changes in pitch, roll, and yaw angles on the static mask obstructing the fuselage. The horizontal displacement, vertical displacement, and boundary angle offset in the image coordinate system have an impact; the system is based on the current flight attitude vector. The horizontal displacement, vertical displacement, and boundary angle correction are read from the airframe structure calibration table relative to the change in standard attitude, and the static mask for airframe occlusion is applied accordingly. Perform image coordinate transformation to generate a dynamic occlusion mask for the current frame. ; Specifically, the airframe structure calibration table Tcal is generated during the system calibration phase; during calibration, the low-altitude aircraft is fixed to an adjustable attitude platform, and the airborne camera is aligned with a uniform background or checkerboard calibration board; a static mask corresponding to the standard attitude A0 is used for fuselage occlusion. As a reference mask, calibration frames are acquired at preset pitch, roll, and yaw angle grid points. The outer contour of the airframe occlusion area at each attitude grid point is extracted and matched with the outer contour of the reference mask to obtain the horizontal displacement Δx, vertical displacement Δy, and boundary angle correction Δθ corresponding to that attitude grid point. Each record in the airframe structure calibration table Tcal includes attitude changes Δpitch, Δroll, Δyaw, and the corresponding Δx, Δy, and Δθ. When the current flight attitude vector... When the change in attitude relative to the standard attitude A0 falls on an attitude grid point in the fuselage structure calibration table Tcal, the corresponding Δx, Δy, and Δθ are directly read. When the change is between adjacent attitude grid points, the adjacent attitude grid points surrounding the change are selected, and linear or trilinear interpolation is performed on Δx, Δy, and Δθ to obtain the horizontal displacement, vertical displacement, and boundary angle correction for the current frame. This is then applied to the fuselage occlusion static mask. When performing image coordinate transformation, the center of the mask or the principal point of the camera in the image coordinate system is used as the center of rotation. First, rotation is performed according to the boundary angle correction Δθ, and then translation is performed according to the horizontal displacement Δx and the vertical displacement Δy to obtain the dynamic occlusion mask Mb(t) for the current frame. The airframe structure calibration table is stored in a one-to-one correspondence with the aircraft model, airborne camera model, camera installation position, camera installation extrinsic parameters, lens distortion correction status, gimbal lock attitude or gimbal movable range, external sensor assembly status, and airframe structure combination status. Upon system startup, the corresponding airframe structure calibration table is read based on the current aircraft configuration identifier. When the camera installation angle is adjusted, the lens is replaced, the gimbal bracket position is changed, external sensors are added or removed, the landing gear shape is changed, or the arm structure is modified, the original airframe structure calibration table is no longer directly used for generating the dynamic occlusion mask for the current frame. The airframe structure calibration should be re-executed, or the airframe structure calibration table corresponding to the current assembly status should be called. If the current flight attitude vector exceeds the range of attitude grid points covered by the airframe structure calibration table, no extrapolation compensation will be performed for the out-of-range data. Instead, the corresponding dynamic occlusion boundary sampling point will be marked as the calibration outer boundary sampling point, and the sampling path in the neighborhood of the boundary starting point will be used preferentially in the current frame.

[0021] In one implementation, the camera intrinsic parameter correction and lens distortion correction have been completed before the current frame enters the boundary calibration module. At this time, the attitude compensation of the static mask of the body occlusion can be achieved by using the horizontal displacement, vertical displacement and boundary angle correction to realize the dynamic occlusion mask update.

[0022] In another implementation, when the airborne camera is a wide-angle or fisheye camera, or when the occlusion area of ​​the aircraft exhibits significant nonlinear deformation at the image edges, the aircraft structure calibration table also records the boundary point coordinate offsets at each attitude grid point. The boundary calibration module first performs lens distortion correction on the current frame, then reads or interpolates the corresponding boundary point coordinate offsets according to the boundary point sequence number, and generates a dynamic occlusion mask for the current frame based on the interpolated boundary point coordinates. If the average difference between the boundary point coordinate offset and the boundary position obtained from the horizontal displacement, vertical displacement, and boundary angle correction is greater than a preset nonlinear compensation threshold, the boundary point coordinate offset is used to generate the dynamic occlusion boundary for the current frame.

[0023] For example, with a standard pose of pitch=0°, roll=0°, and yaw=0°, when in the calibration table roll=5° corresponds to Δx=12 pixels, Δy=3 pixels, and Δθ=1.5°, and roll=10° corresponds to Δx=24 pixels, Δy=6 pixels, and Δθ=3°, if the current roll=7° and pitch and yaw have not changed, then Δx=16.8 pixels, Δy=4.2 pixels, and Δθ=2.1° can be obtained through linear interpolation, and this can be used to create a static mask for the fuselage occlusion. Perform attitude compensation.

[0024] To avoid instantaneous noise from the inertial measurement unit causing dynamic occlusion of the mask in the current frame. The boundary flickering is detected, and the boundary calibration module further performs boundary stabilization processing; the system extracts the dynamic occlusion mask of the current frame. The outer contour is used as the candidate dynamic occlusion boundary, and the dynamic occlusion boundary relative to the previous frame's dynamic occlusion boundary is calculated. The average displacement; this embodiment sets a boundary stability threshold. Boundary stability threshold The initial value is set to 8 pixels; this initial value is calibrated based on the camera resolution, the amplitude of body vibration and the statistical results of the boundary displacement of adjacent frames in the low-altitude inspection scene, and is used to distinguish between real attitude changes and short-term measurement jitter. When the candidate dynamic occlusion boundary is relative to the dynamic occlusion boundary of the previous frame The average displacement is not greater than the boundary stability threshold. When the candidate dynamic occlusion boundary is used as the current dynamic occlusion boundary, the candidate dynamic occlusion boundary is used as the current dynamic occlusion boundary. When the average displacement is greater than the boundary stability threshold In this process, the boundary calibration module first resamples the candidate dynamic occlusion boundary and the dynamic occlusion boundary of the previous frame into the same number of boundary points according to the boundary arc length, denoted as Pcand,k and Ppre,k, where k is the boundary point index; then, it determines the fusion coefficient αedge based on the average displacement dedge, where αedge = Tedge / dedge; when αedge is less than the preset minimum fusion coefficient αmin, it takes αmin, and the initial value of αmin is set to 0.25; then, it obtains the current frame fusion boundary point Pcur,k according to Pcur,k = αedge·Pcand,k + (1-αedge)·Ppre,k, and the current dynamic occlusion boundary is composed of all Pcur,k. ; Furthermore, to avoid short-term mismatch of dynamic occlusion boundaries caused by rapid attitude changes, this embodiment sets attitude change protection conditions and preset change displacement thresholds. The preset change displacement threshold is determined based on the average displacement distribution of adjacent frames of candidate dynamic occlusion boundaries in the rapid maneuver samples. As one implementation, its initial value is set to twice the boundary stability threshold. The attitude change threshold is determined based on the attitude change distribution of adjacent frames in the samples of rapid pitch, roll, and yaw of the low-altitude aircraft. As one implementation, the initial values ​​of the attitude change thresholds corresponding to pitch angle, roll angle, and yaw angle are all set to the high quantile values ​​of the attitude change statistics of adjacent frames. When the average displacement of the candidate dynamic occlusion boundary relative to the dynamic occlusion boundary of the previous frame is greater than the boundary stability threshold but not greater than the preset abrupt displacement threshold, the aforementioned boundary point resampling and coordinate weighted fusion are performed; when the average displacement is greater than the preset abrupt displacement threshold, or when the pitch angle change, roll angle change, or yaw angle change of the current frame flight attitude vector relative to the previous frame flight attitude vector exceeds the corresponding attitude abrupt threshold, the corresponding boundary sampling point is marked as an abruptly unstable boundary sampling point, and it is prohibited from entering the historical frame time inversion sampling path in the current frame, and the boundary starting point neighborhood sampling path is used first; after the average displacement of the boundary in several consecutive frames recovers to within the boundary stability threshold, the normal calculation of the historical frame extrapolation weight is resumed.

[0025] After completing the above processing, the system determines the connected regions representing the occlusion area in the current frame's dynamic occlusion mask as the blind zone to be reconstructed, and determines the continuous, real, visible region within the preset boundary reference width on the side of the current dynamic occlusion boundary away from the blind zone to be reconstructed, which does not belong to the current frame's dynamic occlusion mask, as the boundary reference region. The boundary sampling points used by the subsequent inflow determination module are derived from the current dynamic occlusion boundary, and the pixels to be reconstructed processed by the directional synthesis module are derived from the blind zone to be reconstructed.

[0026] For example, when a low-altitude aircraft performs a power line inspection, a left roll occurs, and the occluded area of ​​the left arm in the image shifts towards the image center; the boundary calibration module adjusts the calibration based on the flight attitude vector. In Changes are made by reading the displacement of the obstructed area from the fuselage structure calibration table and applying it to the static mask obstructing the fuselage. Perform pose compensation to obtain the dynamic occlusion mask for the current frame. And extract the dynamic occlusion boundary. If the candidate dynamic occlusion boundary exhibits an abnormal jump due to machine vibration, the boundary calibration module uses the boundary stability threshold. Triggering smooth blending ultimately outputs a stable dynamic occlusion boundary. and blind spots to be reconstructed .

[0027] This module outputs the dynamic occlusion mask for the current frame. Dynamic occlusion boundary Blind spots to be reconstructed Boundary reference area and flight attitude vector The results are then transmitted to the inflow determination module.

[0028] Please see Figure 3 The inflow determination module is used to analyze the motion trend of real visible textures near the dynamic occlusion boundary based on the current frame and historical frames. This module performs local analysis using boundary sampling points on the dynamic occlusion boundary, constructs the spatiotemporal gradient tensor matrix of the critical region, and obtains the local optical flow vector accordingly. Then, the local optical flow vector is projected onto the normal vector direction in the blind zone to obtain the blind zone inflow projection amount, and outputs the inflow-type or outflow-type blind zone inflow determination result according to the relationship between the blind zone inflow projection amount and the blind zone inflow determination threshold. The output of this module includes the local optical flow vector, the blind zone inflow determination result, the local extrapolation main direction, and the historical frame extrapolation weight, which are used to control whether the orientation synthesis module performs historical frame temporal inversion sampling. The inflow determination module receives the current frame output by the boundary calibration module. Dynamic occlusion boundary Blind spots to be reconstructed and boundary reference area It reads several historical frames corresponding to the preset historical frame buffer depth in the historical frame buffer; the initial value of the preset historical frame buffer depth is set to 2 frames, which is calibrated according to the onboard storage capacity of the low-altitude aircraft, the image frame resolution and the real-time requirements; when the initial value of the preset historical frame buffer depth is 2 frames, the several historical frames stored in the historical frame buffer are the previous frame F(t−1) and the previous two frames F(t−2); To perform local analysis near the dynamic occlusion boundary, this embodiment defines a single outer contour pixel on the dynamic occlusion boundary as a boundary sampling point Pb. The boundary sampling point Pb is used to determine both the center position of the preset local spatiotemporal sampling window and the normal vector within the blind zone pointing from the real visible area to the inside of the blind zone B to be reconstructed. For each boundary sampling point Pb, the system performs one construction of the critical region spatiotemporal gradient tensor matrix M, one solution of the local optical flow vector V, and one blind zone inflow determination. The determination results of multiple boundary sampling points Pb together constitute the inflow state distribution near the dynamic occlusion boundary. It should be noted that the blind zone inflow determination result is a local determination result at the boundary sampling point level, rather than a single global result generated for the entire blind zone to be reconstructed. When the directional synthesis module processes any pixel to be reconstructed, it first determines its boundary start point, and then calls the local inflow determination result, local extrapolation main direction, and historical frame extrapolation weight corresponding to the boundary sampling point closest to the boundary start point to determine the reconstruction path of the pixel to be reconstructed.

[0029] At dynamic occlusion boundaries Nearby, the system sets a preset local spatiotemporal sampling window; the preset local spatiotemporal sampling window includes the local spatial window size. and local time window length Local space window size The initial value is set to dynamically occlude the boundary. A 5×5 pixel window centered on the upper boundary sampling point; this initial value is determined based on the minimum visible width of slender structures such as power lines, runway lines, and road edges in low-altitude inspection scenarios; local time window length. For the current frame and several historical frames preceding the current frame, the initial value of the local time window length is set to 3 consecutive frames. This initial value is used to capture short-term texture motion trends while maintaining real-time performance. When the initial value of the local time window length is 3 frames, the frames covered by the local time window length are the current frame F(t), the previous frame F(t−1), and the two previous frames F(t−2). When the local time window length is expanded to other values ​​according to the application scenario, the frames covered by the local time window length are expanded to the current frame and the corresponding number of historical frames preceding the current frame. Within a preset local spatiotemporal sampling window, the inflow determination module first preprocesses the frames covered by the preset local spatiotemporal sampling window to obtain a brightness map for subsequent gradient calculation; the preprocessing is used to reduce the influence of isolated bright spots caused by sensor noise, compression noise and flight vibration. As one implementation method, the preprocessing includes two stages: color space conversion and image denoising. The color space conversion stage converts the color image frames in the current frame and historical frames into grayscale images. The image denoising stage performs median filtering on the grayscale images, and the initial value of the sliding window size for median filtering is set to 3×3 pixels. As another implementation, the color space conversion stage of the preprocessing can also retain the original brightness channel of the image frame without performing additional grayscale conversion; the image noise reduction stage of the preprocessing can also use any one of mean filtering, bilateral filtering or Gaussian filtering. Those skilled in the art can select according to the computing power constraints and real-time requirements of the airborne computing platform. All of the above noise reduction methods can achieve the same technical effect of reducing sensor noise, compression noise and the influence of isolated bright spots caused by flight vibration. Subsequently, the system calculates the horizontal gradient of effective pixels based on the preprocessed luminance map within a preset local spatiotemporal sampling window. Vertical gradient and time gradient Horizontal gradient The vertical gradient is obtained by comparing the brightness of adjacent pixels on the left and right sides of the same frame's brightness map. The temporal gradient is obtained by the brightness difference between adjacent pixels in the same frame's brightness map. The effective pixel is obtained by measuring the brightness change of the same pixel position between adjacent frame brightness maps. In this embodiment, an effective pixel is a pixel that simultaneously meets the following conditions: its image coordinates are located within the image range of the current video frame or a historical frame; it does not belong to the occlusion area of ​​the machine in the dynamic occlusion mask of the corresponding frame; its pixel brightness has not reached the upper and lower limits of saturation; and its location is capable of performing horizontal gradient, vertical gradient, and temporal gradient calculations. Pixels that do not meet the above conditions are not included in the accumulation calculation of the critical region spatiotemporal gradient tensor matrix M.

[0030] For any valid pixel q, the horizontal gradient The vertical gradient is obtained from the brightness difference between adjacent pixels on the left and right sides within the same frame. The temporal gradient is obtained from the brightness difference between adjacent pixels within the same frame. It is obtained from the brightness difference of the same pixel position in adjacent frames or the corresponding pixel position after motion compensation. In order to reduce the influence of edge noise on the motion trend at the boundary, this embodiment sets a spatial attenuation weight wq, which decreases as the distance between the effective pixel q and the boundary sampling point Pb increases; as one implementation, wq=1 / (1+dq), where dq is the pixel distance between the effective pixel q and the boundary sampling point Pb.

[0031] The spatial structure component and spatiotemporal coupling component in the spatiotemporal gradient tensor matrix M of the critical region are obtained by weighted accumulation of effective pixels: M11 is The accumulated values, M12 and M21 are The accumulated value, M22 is The accumulated values, M13 and M31 are The accumulated values, M23 and M32 are The accumulated value, M33 is The accumulated value. Thus, the spatiotemporal gradient tensor matrix M of the critical region can simultaneously reflect the spatial structural intensity of the texture near the boundary and the tendency to move towards the boundary over time.

[0032] To avoid random noise in low-texture areas being misjudged as deducible structures, this embodiment sets a low-texture determination threshold. After the effective pixels are filtered, the input to the determination module is based on the horizontal gradient of the effective pixels. and vertical gradient Calculating spatial gradient intensity; as one implementation method, spatial gradient intensity The average gradient magnitude of the effective pixels, i.e. ,in This represents the arithmetic mean of all valid pixels within the valid pixel set. Low-texture determination threshold. The initial value is set to 10 gray levels, which is determined based on the statistical results of gray level gradients in the sky, fog, water, and uniform ground areas from low-altitude aerial photography. When the spatial gradient intensity is below the low-texture determination threshold... When the system determines that there is a lack of stable traceable structures near the dynamic occlusion boundary, it does not initiate strong directional extrapolation and marks the corresponding boundary sampling point Pb as a low-texture boundary sampling point; when the spatial gradient intensity is not lower than the low-texture determination threshold... At that time, the inflow determination module solves for the local optical flow vector V based on the spatiotemporal gradient tensor matrix M of the critical region. The local optical flow vector V is used to represent the direction and intensity of movement of the real visible texture near the dynamic occlusion boundary in the image plane. The local optical flow vector V includes the horizontal velocity vx and the vertical velocity vy. Among them, the spatial structure components M11, M12, M21 and M22 are used to form the two-dimensional velocity solution relationship, the spatiotemporal coupling components M13 and M23 are used to form the velocity driving quantity, and the time variation component M33 is used to characterize the strength of the overall brightness change with time within the preset local spatiotemporal sampling window.

[0033] Specifically, the inflow determination module solves for the local optical flow vector V based on the criterion of minimizing the local brightness constant error. For the effective pixel set, a local error function is established. Where vx is the horizontal velocity, vy is the vertical velocity, and Σ represents the accumulation of valid pixels within the valid pixel set. Minimizing E(vx,vy) with respect to vx and vy respectively yields the following system of linear equations: (M11+εg)·vx+M12·vy=−M13; M21·vx+(M22+εg)·vy=−M23; Wherein, εg is the gradient stability constant, initially set to 0.001, to avoid instability in the equation solution due to excessively small spatial structure components. The inflow determination module obtains vx and vy based on the above linear equations, and uses V=(vx,vy) as the local optical flow vector corresponding to the boundary sampling point Pb.

[0034] When the determinant of the aforementioned linear equation system is less than a preset stability threshold, or the number of effective pixels is less than a preset minimum number of effective pixels, the inflow determination module does not output a strongly directional local optical flow vector and marks the boundary sampling point Pb as a low-texture boundary sampling point. When the vector norm of the local optical flow vector V is less than the minimum velocity modulus threshold described later, the inflow determination module does not use the boundary sampling point as an inflow-type boundary sampling point. Therefore, the system only uses the local optical flow vector V as input for subsequent blind zone inflow determination and direction inference channel construction when there is a stable traceable texture near the dynamic occlusion boundary and the texture has sufficient motion amplitude.

[0035] To avoid static textures or weak image noise from participating in blind zone inference, this embodiment sets a minimum velocity modulus threshold. Minimum velocity modulus threshold The initial value is set to 0.2 pixels per frame, which is calibrated based on the camera frame rate, low-altitude flight speed, and static texture jitter amplitude. When the vector norm of the local optical flow vector V is less than the minimum velocity magnitude threshold... When the system determines that there is no effective texture motion available for blind zone inference near the dynamic occlusion boundary, it marks the boundary sampling point Pb as a low-velocity boundary sampling point and does not treat it as an inflow-type boundary sampling point. The system also extrapolates the weights of the historical frames at the corresponding locations. Updated to 0; when the vector norm of the local optical flow vector V is not less than the minimum velocity magnitude threshold. At that time, the system inputs the local optical flow vector V into the calculation process of the projection amount flowing into the subsequent blind zone.

[0036] After obtaining the effective local optical flow vector V, the inflow determination module determines the current dynamic occlusion boundary. Determine the normal vector within the blind zone Normal vector within the blind zone This represents the unit direction from the truly visible area to the interior of the blind zone B to be reconstructed. In specific implementation, the system centers on the boundary sampling point Pb, within the current dynamic occlusion boundary. The adjacent outer contour pixels before and after the boundary sampling point Pb are selected, and the local tangent direction at the boundary sampling point Pb is determined based on the positional changes of the adjacent outer contour pixels. Subsequently, the unit direction perpendicular to this local tangent direction and pointing to the side of the blind zone B to be reconstructed in the current frame dynamic occlusion mask with a pixel value of 1 is determined as the normal vector in the blind zone. Therefore, the normal vector within the blind zone A one-to-one correspondence is formed with the boundary sampling point Pb, which is used to determine whether the local optical flow vector V near the boundary sampling point Pb is oriented towards the interior of the blind region B to be reconstructed.

[0037] To determine whether the truly visible texture is moving towards the blind area to be reconstructed. Internal inflow; in this embodiment, the local optical flow vector... Projected onto the normal vector within the blind zone Direction, to obtain the projected amount of blind zone inflow. And based on the amount of projection flowing into the blind zone Generate historical frame extrapolation weights Among them, the amount of projection flowing into the blind zone. This is used to represent the magnitude of the component of the local optical flow vector V that faces into the blind zone B to be reconstructed.

[0038] In one implementation, the blind zone inflow projection amount The calculation formula is expressed as: In the formula, The local optical flow vector represents the dynamic occlusion boundary. The direction and intensity of motion of nearby real-world visible textures within the image plane. The normal vector within the blind zone represents the direction from the truly visible area to the blind zone to be reconstructed. Internal unit direction, Let V be the vector norm of the local optical flow vector. A preset stability constant is used to avoid division by zero. The initial value is set to 0.0001, which is set according to the stability requirements of floating-point calculation; Then based on the obtained blind zone inflow projection amount Historical frame extrapolation weights The calculation formula is expressed as: Historical frame extrapolation weights This represents the base strength of the historical frame texture participating in blind zone reconstruction; when the blind zone flows into the projection amount When greater than 0, the historical frame extrapolation weight Projected amount flowing into the blind spot Increases as the blind zone flows into the projected amount. When less than or equal to 0, the extrapolation weight of historical frames Setting it to 0 prevents textures that deviate from the blind zone B to be reconstructed from participating in strong directional extrapolation.

[0039] This embodiment sets a threshold for determining blind zone inflow. Blind spot inflow determination threshold The initial value is set to 0; the technical meaning of this initial value is: only when the local optical flow vector is... Normal vector within the blind zone Only when the projection in the direction is positive is the truly visible texture considered to have entered the blind zone to be reconstructed. The trend; when the blind spot flows into the projection amount Greater than the blind zone inflow determination threshold And the local optical flow vector The vector norm is not less than the minimum velocity magnitude threshold. Furthermore, the spatial gradient intensity is not lower than the low texture determination threshold. At that time, the inflow determination module outputs the inflow determination result for the blind zone. and the local optical flow vector The normalization direction is determined as the main direction of local deduction. When the blind zone flows into the projected amount Less than or equal to the blind zone inflow determination threshold or local optical flow vector The vector norm is less than the minimum velocity magnitude threshold. Or the spatial gradient intensity is lower than the low texture determination threshold. At that time, the inflow determination module outputs the inflow determination result of the deviation type blind zone. The historical frame extrapolation weight is updated to 0, so that the directional synthesis module no longer performs strong directional extrapolation of historical frames along the direction corresponding to the boundary sampling point Pb. For example, when a low-altitude aircraft inspects power lines, the power lines extend from the visible area to near the area obstructed by the aircraft arm; the inflow determination module detects this at the dynamic obstruction boundary. Construct the spatiotemporal gradient tensor matrix of the critical region nearby And calculate the local optical flow vector If the local optical flow vector Projected onto the normal vector within the blind zone Blind spot flow into projection after direction A positive value indicates that the wire texture is moving towards the blind area to be reconstructed. Internal extension, the system outputs the inflow-type blind zone inflow determination result. And this direction of motion is determined as the main direction of local deduction. Conversely, when the aircraft turns, causing the ground texture to shift from the dynamic occlusion boundary... When the point is moved away, the local optical flow vector Deviation from the normal vector within the blind zone Blind spot inflow projection Not greater than the blind zone inflow determination threshold The system outputs the result of the blind zone inflow determination for the deviation type. The subsequent targeted compositing module will no longer forcibly push historical frame textures into the blind area to be reconstructed. ; This module outputs the spatiotemporal gradient tensor matrix of the critical region. Local optical flow vector Blind spot inflow projection Blind spot inflow determination results Local deduction main direction and historical frame extrapolation weights The results are then passed to the directional synthesis module.

[0040] The directional synthesis module generates pixel reconstruction values ​​for pixels within the blind zone to be reconstructed based on the blind zone inflow determination result output by the inflow determination module. When the blind zone inflow determination result is inflow type, this module determines the directional inference channel based on the local inference main direction and locates the historical frame sampled pixel corresponding to the texture carried by the pixel to be reconstructed in the historical frame along the opposite direction of the local optical flow vector. When the historical frame sampled pixel is located in the real visible area of ​​the time inversion sampled frame, the pixel channel value at that position is used as the pixel reconstruction value of the pixel to be reconstructed. When the current frame is a single-channel image, the pixel channel value is the luminance channel value. When the current frame is a color image frame, the pixel channel value includes the luminance channel value and the chrominance channel value. When the local inflow determination result corresponding to the associated boundary sampling point When the pixel is a divergent type, or the historical frame extrapolation weight is 0, or the pixel to be reconstructed does not belong to the directional inference channel, or the pixel to be reconstructed is a nearest neighbor matching pixel or a pixel that failed time inversion, the directional synthesis module does not start the historical frame time inversion sampling path, but instead uses the boundary starting point neighborhood sampling path to generate the pixel reconstruction value; among which, the local inference main direction corresponding to the divergent type boundary sampling point. The boundary sampling point is marked as a low-confidence direction, preventing the pixels to be reconstructed associated with it from being strongly extrapolated along the texture direction of the historical frame. This process ensures that textures deviating from dynamic occlusion boundaries or unable to be reliably inverted will not be incorrectly pulled into the blind zone B to be reconstructed.

[0041] When the local inflow determination result corresponding to the associated boundary sampling point When the input is inflow type, the historical frame extrapolation weight is greater than 0, and the pixel to be reconstructed belongs to the direction inference channel, the orientation synthesis module will associate the local inference main direction corresponding to the boundary sampling point. As the primary control direction, initiate the direction deduction channel. The process is constructed, and the historical frame time-reversal sampling path is entered. The directional synthesis module finally writes the pixel reconstruction value of each pixel to be reconstructed into the image buffer according to its image coordinates in the blind zone B to be reconstructed, forming the blind zone reconstruction enhancement layer image.

[0042] For blind spots to be reconstructed Any pixel to be reconstructed within The directional synthesis module first determines its corresponding boundary starting point. Specifically, from the pixels to be reconstructed Along the main direction of local deduction Perform pixel-level backtracking in the opposite direction until reaching the dynamic occlusion boundary. If they intersect, the point of intersection shall be taken as the starting point of the boundary. If due to dynamic occlusion boundary If local bumps or pixel discrepancies prevent the acquisition of effective intersection points, then a dynamic occlusion boundary is selected. Upper distance to the pixel to be reconstructed The nearest boundary sampling point is used as the boundary starting point. and the pixel to be reconstructed Pixels are marked as nearest-neighbor matching pixels; the intensity of directional extrapolation is reduced in subsequent synthesis to reduce erroneous extension at complex boundaries; To clarify the pixels to be reconstructed Does it belong to the directional deduction channel? In this embodiment, the first step is to reconstruct the pixels to be reconstructed. Boundary starting point and the main direction of local deduction Determine the longitudinal projection depth Specifically, the system uses the boundary starting point... Starting from this point, proceed along the main direction of local deduction. Measure the pixels to be reconstructed Entering the blind zone to be reconstructed The distance is used as the longitudinal projection depth. When the pixel to be reconstructed Located in the main direction of local deduction When in front, longitudinal projection depth The value is non-negative, when the pixel to be reconstructed is... Located in the main direction of local deduction When on the opposite side, the longitudinal projection depth It is a negative value; At the same time, the system determines and locally extrapolates the main direction. Vertical direction and along the vertical direction Measure the pixels to be reconstructed Relative to the main direction of local deduction The degree of deviation is taken as the lateral offset. ; lateral offset The smaller the value, the more pixels are to be reconstructed. The closer the texture is to the real visible texture, the more likely it is to follow the main direction of local extrapolation. The resulting extended path; lateral offset The larger the value, the more pixels are to be reconstructed. The further it deviates from the extended path; Obtaining the longitudinal projection depth and lateral offset Afterwards, the direction deduction channel The formula for determining ownership is expressed as: ; in, Indicates the pixel to be reconstructed It belongs to the directional deduction channel Participate in strong directional reconstruction; Indicates the pixel to be reconstructed Not part of the directional deduction channel Only conservative completion is performed; boundary base channel width Indicates the direction deduction channel At dynamic occlusion boundaries The base width of the surrounding area is initially set to 2 pixels; the channel horizontal growth factor. Indicates the direction deduction channel With longitudinal projection depth The degree to which horizontal expansion is allowed is increased, with an initial value set at 0.15; Satisfying the direction deduction channel Pixels to be reconstructed based on the criteria Classified into the directional deduction channel This indicates that the pixel is located along the principal direction of the locally inferred texture in the actual visible texture. Entering the blind zone to be reconstructed Within the effective channel; not satisfying the direction deduction channel Pixels to be reconstructed based on the criteria Instead of performing strong directional extrapolation, pixel values ​​are generated using only a conservative completion method. Through this process, directional textures such as wires, runway lines, road edges, or building outlines can extend along their true direction of entry without spreading into irrelevant background areas. Subsequently, the orientation synthesis module based on the longitudinal projection depth Determine the pixels to be reconstructed directional projection hierarchy; projection layer spacing Indicates the distance between adjacent projection layers, projection layer spacing The initial value is set to 1 pixel; this initial value is set according to the pixel-level reconstruction accuracy requirements to ensure proximity to the dynamic occlusion boundary. The pixels are first reconstructed and then used as a reference source for deeper pixels; the system follows the vertical projection depth. Pixel compositing is performed in ascending order to enable directional deduction channels. The pixels within the local extrapolation principal direction Generate layer by layer; After determining the generation order of the directional projection hierarchy, the orientation synthesis module reconstructs the blind areas to be reconstructed in ascending order of the directional projection hierarchy. Execute pixel reconstruction value for each pixel to be reconstructed. The synthesis; when determined to be of the inflow type, the truly visible texture is along the local optical flow vector. Entering the blind zone to be reconstructed Therefore, the current frame Located in the blind spot to be reconstructed Any pixel to be reconstructed within The textures it carries have not yet entered the blind zone to be reconstructed in historical frames several frames ago. It remains within the truly visible region; the local optical flow vector Extending backwards in the time dimension, this module determines the historical frame sampled pixel corresponding to the current pixel P to be reconstructed within the truly visible area of ​​the historical frame. Based on the above relationship, this module uses the historical frame time-reverse sampling path for the pixels to be reconstructed within the directional extrapolation channel, and uses the pixel channel value at that location as the pixel reconstruction value of the pixel to be reconstructed. When the current frame is a single-channel image, the pixel channel value is the luminance channel value; when the current frame is a color image frame, the pixel channel value includes both luminance and chrominance channel values. For pixels to be reconstructed, nearest-neighbor matching pixels, and pixels that failed time inversion outside the direction inference channel, a boundary starting point neighborhood sampling path is used to avoid weighting multiple types of reference brightness. Please see Figure 4 For the current pixel to be reconstructed in the direction deduction channel The directional synthesis module calculates the local optical flow vector. In the main direction of local deduction The projection modulus is used as the effective inflow velocity along the locally extrapolated principal direction. Effective inflow velocity This indicates that the real visible texture is along the principal direction of local inference. Each frame enters the blind zone to be reconstructed. The distance; Subsequently, based on the current pixel to be reconstructed Longitudinal projection depth and effective inflow velocity Determine the number of time inversion steps Divide the longitudinal projected depth by the effective inflow velocity. Round up to get the number of time reversal steps. Time reversal steps The physical meaning is the pixel to be reconstructed. The texture it carries is in The frame before that has not yet entered the reconstruction blind zone. It is still located in the actual visible area; To avoid effective inflow speed Too small a number of time reversal steps The abnormal increase in flow rate is addressed by setting a minimum effective inflow velocity threshold in this embodiment. Minimum effective inflow velocity threshold The initial value is set to 0.2 pixels per frame; when the effective inflow rate... Less than the minimum effective inflow velocity threshold When the historical frame time inversion sampling is not performed, the current pixel P to be reconstructed is marked as an inversion failure pixel, and the sampling path of the neighborhood of the boundary starting point is switched; when the effective inflow velocity Not less than the minimum effective inflow velocity threshold At that time, based on the longitudinal projection depth and effective inflow velocity Determine the number of time inversion steps ; Furthermore, based on the number of time inversion steps Select time-reversal sampling frames from the historical frame buffer. The history frame buffer is used to store image frame data of several frames preceding the current frame in a continuous video stream. The maximum number of frames supported by the history frame buffer is the preset history frame buffer depth. The specific method for selecting the temporal inversion sampling frame is as follows: the k-th historical frame in the historical frame buffer that precedes the current frame is used as the temporal inversion step. The time-reversal sampling frame corresponding to k is equal to k. When the time reversal steps When the depth exceeds the preset historical frame buffer depth, it exceeds the range of the historical frame buffer. The current pixel to be reconstructed is marked as a failed inversion pixel and transferred to the neighborhood sampling path of the boundary starting point. As one implementation method, when the initial value of the preset historical frame buffer depth is 2 frames, the selection relationship of the time inversion sampling frame is as follows: when the time inversion step is equal to 1, the previous frame F(t−1) is selected as the time inversion sampling frame; when the time inversion step is equal to 2, the previous two frames F(t−2) are selected as the time inversion sampling frames; when the time inversion step is greater than 2, it exceeds the range of the historical frame buffer. The specific value of the preset historical frame buffer depth can be expanded according to the onboard storage capacity. For example, when the onboard storage capacity allows, the preset historical frame buffer depth can be set to 3 frames, 5 frames or more, and the upper limit of the time inversion step that can be processed is correspondingly expanded, thereby supporting blind zone reconstruction in scenarios with small effective inflow velocity or large vertical projection depth. Subsequently, in the time-reversal sampling frame Mid-positioning historical frame sampling pixels : From the current pixel to be reconstructed Starting from the image coordinates, along the local optical flow vector Reverse backtracking time reversal steps The corresponding displacement yields the time-reversal sampling frame. Historical frame sampling pixels Historical frame sampling pixels Time-reversal sampling frames The corresponding pixel to be reconstructed in the middle The actual visible source location of the carried texture; Based on historical frame sampling pixels Time-reversal sampling frames The validity of time-reversal sampling for attribution determination: Specifically, the system reads time-reversal sampling frames. Corresponding dynamic occlusion mask ,in For the number of time reversal steps; when Located within the image range, and middle When the pixel value at the corresponding position represents the actual visible area, the time-reversed sampling frame is used. middle The pixel brightness at a given location is directly used as the pixel to be reconstructed. Pixel reconstruction values; when lie in When the corresponding occluded area is outside the image range, the directional synthesis module will use the current pixel to be reconstructed. Pixels marked as inversion failures are then moved to the sampling path in the neighborhood of the boundary starting point. For pixels to be reconstructed outside the directional inference channel, pixels to be reconstructed marked as nearest-neighbor matching pixels in the first half, and the aforementioned inversion failure pixels, the directional synthesis module performs boundary starting point neighborhood sampling. Specifically, it takes the boundary starting point corresponding to the pixel to be reconstructed as the center and operates within a preset boundary sampling radius. Boundary reference area within the range The arithmetic mean of the actual visible pixel brightness is taken as the pixel reconstruction value for the pixel to be reconstructed. Preset boundary sampling radius The initial value is set to 2 pixels. This initial value is determined based on the local consistency range of the truly visible texture near the boundary in the low-altitude inspection scene. This is used to avoid introducing irrelevant textures far from the boundary starting point while maintaining statistical stability. Among them, the nearest neighbor matching pixel is determined because its boundary starting point is the dynamically occluded boundary. The nearest boundary sampling point to the pixel to be reconstructed, rather than along the principal direction of local inference. The reverse backtracking intersection point indicates that the time reversal direction is no longer accurate. Therefore, we directly switch to the sampling path in the neighborhood of the boundary starting point to avoid erroneous extension. The blind areas to be reconstructed will be completed sequentially according to the directional projection hierarchy, from smallest to largest. Pixel reconstruction values ​​of all pixels to be reconstructed within the range Generate; when the pixel reconstruction values ​​of all pixels to be reconstructed in this frame are... All are written to the dynamic occlusion mask of the current frame according to their position. After using an image buffer of the same size, the blind zone reconstruction enhancement layer image is obtained. ; It should be noted that when the current frame and historical frames are color image frames, the system first converts the color image frames into luma and chroma channels. The luma channel is used for local optical flow calculation, blind zone inflow determination, and pixel luma reconstruction. For inflow-type pixels to be reconstructed, if the sampled pixel from the historical frame is located in the truly visible area, the chroma channel value at the sampled pixel in the historical frame is simultaneously read and combined with the reconstructed luma channel value to obtain the color pixel value of the pixel to be reconstructed. For out-of-range pixels, pixels outside the direction inference channel, nearest neighbor matching pixels, or pixels that failed to be inverted, neighborhood sampling is performed on the luma and chroma channel values ​​within the boundary reference area to obtain the corresponding color pixel value. Therefore, the blind zone reconstruction enhancement layer image can be either a luma image or a color image containing both luma and chroma channels.

[0043] For example, taking an initial value of 2 frames for the preset historical frame buffer depth, when a low-altitude aircraft inspects power lines, the power lines follow the locally extrapolated main direction. Extending into the blind spot to be reconstructed For a pixel to be reconstructed within the directional deduction channel Its vertical projection depth is 1.5 pixels, corresponding to an effective inflow velocity. The frame size is 1.2 pixels, from which the number of time reversal steps is obtained. The value is 2; the directional synthesis module starts from the first two frames. Mid-line local optical flow vector The historical frame sampled pixels are obtained by tracing back two steps of displacement. and directly middle The brightness of the wire pixels at the location is used as the pixel reconstruction value. The power lines are in the blind spot to be rebuilt. The inner edge continues layer by layer along the true entry direction; for pixels outside the channel that are far from the direction of the wire extension, sampling of the neighborhood of the boundary starting point is directly performed, and the wire is not laterally diffused into a thick line; when the aircraft turns, the local optical flow vector is affected. Reduced, longitudinal projection depth exceeds effective inflow velocity When accumulating coverage distance within the historical frame buffer, deep pixels are marked as inversion failure pixels and transferred to the sampling path of the neighborhood of the boundary starting point; This module outputs pixel reconstruction values. and blind zone reconstruction enhancement layer image The results are then passed to the closed-loop correction module.

[0044] The closed-loop correction module is used to perform boundary continuity verification and next-frame parameter write-back on the blind-area reconstruction enhancement layer image generated by the directional synthesis module. This module sets verification windows on both sides of the dynamic occlusion boundary to calculate the directional consistency error, brightness continuity error, and temporal inversion failure rate between the blind-area reconstruction enhancement layer image and the truly visible area in the current frame. When the directional consistency error exceeds a preset upper limit for directional error, the main direction of local extrapolation in the next frame is corrected. When the temporal inversion failure rate exceeds a preset upper limit for failure rate, the extrapolation weight of historical frames in the next frame is reduced. When the brightness continuity error exceeds a preset upper limit for brightness error, the boundary neighborhood is marked for priority reconstruction of boundary segments to suppress erroneous extrapolation, boundary bending, and brightness abrupt changes in consecutive frames. Finally, the closed-loop correction module fuses the blind-area reconstruction enhancement layer image with the current frame according to the current frame's dynamic occlusion mask to obtain the final output. The closed-loop correction module performs the following steps on the blind-area reconstruction enhancement layer image: With dynamic occlusion boundary The continuity of the visible areas on both sides is verified, and different types of continuity errors are written back to the local inference main direction of the next frame. Historical frame extrapolation weights Prioritize the reconstruction of boundary segments within the boundary neighborhood to form a closed-loop adaptive correction in the continuous video stream. Specifically: Receive the blind zone reconstruction enhancement layer image output by the directional synthesis module. And inversion failure pixel markers, and receive the current frame output by the boundary calibration module. Current frame dynamic occlusion mask Dynamic occlusion boundary and boundary reference area Receive the local inference main direction output by the inflow determination module and historical frame extrapolation weights ; At dynamic occlusion boundaries Circular verification windows are set on both sides, and the width of the circular verification windows is the preset verification window width. Preset verification window width The initial value is set to 4 pixels, which is obtained by calibrating the statistically effective range of local textures at the boundary in the low-altitude inspection scene; among them, the verification window inside the blind zone is located in the blind zone to be reconstructed. One side is used to read the image of the blind zone reconstruction enhancement layer. The reconstructed boundary local information; the actual visible side verification window is located in the boundary reference area. One side is used to read the current frame. Real boundary local information in the middle; For each boundary sampling point within the verification window, calculate the horizontal and vertical gradients respectively. Then, weight the gradient directions of each pixel according to their gradient magnitudes, sum them, and normalize them to obtain the reconstructed direction vector inside the blind zone. The direction vector of the truly visible side is obtained. Reconstruct the direction vector inside the blind zone With the actual visible side direction vector The difference in modulus is used as the directional consistency error. ; Reconstruct the enhancement layer image of the blind zone within the verification window inside the blind zone. The pixel brightness is obtained by taking the arithmetic mean to obtain the reconstructed brightness inside the blind zone. Within the actual visible side verification window, the current frame is checked. The pixel brightness is taken as the arithmetic mean to obtain the true visible side brightness. Reconstruct brightness inside the blind spot Compared to the actual visible side brightness The absolute value of the difference divided by the actual visible side brightness With the preset stability constant The sum of these values ​​yields the brightness continuity error. ; The proportion of inversion failure pixels marked by the directional synthesis module in the total number of pixels in the blind zone to be reconstructed is used to obtain the temporal inversion failure rate. The higher the temporal inversion failure rate, the lower the availability of the temporal inversion sampling path of the current historical frame. This may be caused by insufficient effective inflow velocity, insufficient coverage of historical frame buffer, historical frame sampling pixels being in occluded areas, sampling pixels going out of bounds, or the local inference main direction deviating from the real texture entry direction. It should be noted that the closed-loop correction in this embodiment does not use a unified write-back correction amount synthesized from the direction consistency error, brightness continuity error, and time inversion failure rate, nor does it trigger a unified correction after the cumulative error exceeds the consistency threshold. The three types of errors are compared with their corresponding thresholds respectively, and the correction targets for the next frame are determined accordingly. Specifically, the direction consistency error is used to trigger the correction of the main direction of local extrapolation in the next frame, the time inversion failure rate is used to trigger the reduction of the extrapolation weight of historical frames in the next frame, and the brightness continuity error is used to trigger the priority reconstruction of boundary segment markers in the boundary neighborhood.

[0045] When the direction consistency error Greater than the preset directional error limit When, explain the main direction of the current local simulation. With the actual visible side direction vector If a deviation exists, the closed-loop correction module will extrapolate the current local main direction. With the actual visible side direction vector After fusion and normalization, the principal direction of local inference for the next frame, Epush_new, is obtained, specifically as follows: Where Normalize represents vector normalization. When the direction consistency error... Not greater than the preset upper limit of directional error At that time, the next frame follows the current main direction of local deduction. .

[0046] When time reversal failure rate Greater than the preset failure rate limit This indicates that during time-reversal sampling along historical frames, many pixels to be reconstructed cannot be located to the actual visible sampled pixels. The closed-loop correction module reduces the extrapolation weight Whist_new of the next historical frame, thus reducing the next frame's dependence on the strong directional extrapolation of historical frames. As one implementation method, the extrapolation weight Whist_new of the next historical frame is adjusted according to... Received, among which The extrapolation weights for the current historical frames. When the time reversal failure rate... No greater than the preset failure rate limit At that time, the next frame retains the extrapolation weight of the current historical frame. .

[0047] When brightness continuity error Greater than the preset brightness error limit This indicates that there is a brightness discontinuity between the blind zone reconstruction enhancement layer image and the truly visible side region at the dynamic occlusion boundary, and the closed-loop correction module will generate this brightness continuity error. The dynamically occluded boundary segments are marked as boundary neighborhood priority reconstruction boundary segments; among them, the brightness continuity error is generated. The dynamic occlusion boundary segment is the set of continuous boundary sampling points covered by the annular verification window where the brightness continuity error exceeds the limit. Specifically, the closed-loop correction module calculates the brightness continuity error for each boundary sampling point according to the contour sampling order of the dynamic occlusion boundary. When the brightness continuity error corresponding to a certain boundary sampling point is greater than the preset upper limit of brightness error, the boundary sampling point is marked as a brightness excess sampling point. Subsequently, adjacent brightness excess sampling points are merged into the same brightness excess boundary segment along the sampling order of the dynamic occlusion boundary. If two brightness excess sampling points are separated by only non-excess sampling points not exceeding the preset merging interval, the brightness excess sampling points on both sides are merged into the same brightness excess boundary segment to avoid isolated sampling noise causing the boundary segment to be incorrectly split. The initial value of the preset merging interval is set to 1 boundary sampling point, and the initial value of the preset minimum number of continuous points is set to 3 boundary sampling points. After the brightness exceeding the limit boundary segment is formed, the closed-loop correction module expands both ends of the segment outwards by a preset number of boundary sampling points along the dynamic occlusion boundary sampling sequence, obtaining the boundary neighborhood priority reconstruction boundary segment. The initial value of the preset segment expansion number is set to 2 boundary sampling points to cover the transition effect of the boundary brightness abrupt change to adjacent areas. In the next frame, the pixels to be reconstructed associated with the boundary start point corresponding to this boundary neighborhood priority reconstruction boundary segment preferentially adopt the sampling path of the boundary start point's neighborhood. When the brightness continuity error... Not greater than the preset brightness error limit In this case, the current reconstruction path is maintained in the next frame.

[0048] Preset upper limit of direction error Preset brightness error upper limit and preset failure rate limit The upper limit is determined based on the distribution of orientation consistency error, brightness continuity error, and time inversion failure rate in normal and abnormal reconstruction samples, respectively. As one implementation method, the high quantile value of the corresponding error distribution in normal reconstruction samples or the low quantile value of the corresponding error distribution in abnormal reconstruction samples is selected as the initial upper limit.

[0049] After completing the above sub-corrections, the closed-loop correction module writes the next frame's local extrapolation main direction `Epush_new`, the next frame's historical frame extrapolation weight `Whist_new`, and the boundary segment prioritized for reconstruction in the boundary neighborhood into the historical state cache. This cache serves as the pre-state for the next frame's inflow determination module, directional synthesis module, and closed-loop correction module. Through this process, directional deviation, sampling failure, and brightness abrupt changes trigger different correction actions, ensuring that the closed-loop correction corresponds to the reconstruction anomalies caused by the dynamic occlusion blind spots of the low-altitude aircraft's structure.

[0050] Finally, the closed-loop correction module uses the current frame's dynamic occlusion mask as the fusion basis, retains the original pixels of the current frame in the truly visible area, replaces the occluded area of ​​the machine with the reconstructed pixel values ​​of the same coordinates in the blind zone reconstruction enhancement layer image, and performs a linear transition on the pixels at the dynamic occlusion boundary according to the boundary distance to obtain the final output frame. The final output frame is written to the continuous video stream; the current dynamic occlusion boundary, the current frame's dynamic occlusion mask, the main direction of local inference for the next frame, the extrapolation weight of the next frame's historical frames, and the boundary neighborhood priority reconstruction boundary segment are written to the historical state cache as the pre-state for the next frame processing.

[0051] For example, consider the inflow-type reconstruction of power line textures. When a low-altitude aircraft performs power line inspection, its arm enters the field of view of the onboard camera, forming a dynamic occlusion blind zone on the upper left side of the current frame. The boundary calibration module performs attitude compensation on the static mask of the fuselage occlusion corresponding to the arm based on the current frame and flight attitude vector, obtaining the dynamic occlusion boundary of the current frame. The inflow determination module calculates the local optical flow vector near the boundary sampling point of this dynamic occlusion boundary. When the power line texture extends from the real visible area to the arm occlusion area, the projection of the local optical flow vector onto the normal vector in the blind zone is greater than the blind zone inflow determination threshold, and the inflow-type local inflow determination result is output. The directional synthesis module assigns the pixels to be reconstructed associated with the boundary sampling point to the directional inference channel, and performs historical frame time inversion sampling in the opposite direction of the local optical flow vector. It locates the sampling pixels when the power line texture is in a real visible state in the historical frame, and writes them as pixel reconstruction values ​​into the blind zone reconstruction enhancement layer image. Therefore, the wire texture can continue along the true direction of its entry into the blind zone of the arm, avoiding the interruption of the wire at the blind zone caused by the use of ordinary boundary mean completion.

[0052] Another example is the reconstruction of road edge deviation. When a low-altitude aircraft flies laterally along a road and makes a turn, the landing gear or fuselage edge enters the field of view of the airborne camera, forming a dynamic occlusion blind zone of the aircraft structure. At the same time, the road edge line moves from the dynamic occlusion boundary to the outside of the truly visible area in consecutive frames. The inflow determination module calculates the local optical flow vector near the dynamic occlusion boundary and projects it onto the normal vector in the blind zone. When the projection amount is not greater than the blind zone inflow determination threshold, it outputs the deviation-type local inflow determination result and updates the corresponding historical frame extrapolation weight to zero. The directional synthesis module does not perform historical frame temporal inversion sampling on the pixels to be reconstructed associated with the boundary sampling point. Instead, it selects truly visible and unsaturated neighboring pixels in the boundary reference area with the boundary starting point as the center to generate pixel reconstruction values. Thus, the system will not erroneously extrapolate the road edge line that has deviated from the dynamic occlusion boundary into the blind zone to be reconstructed, which can reduce directional misalignment and texture ghosting.

[0053] The parameters in the above formulas are used to define the computational relationships between image frames, attitude data, dynamic occlusion boundaries, local optical flow vectors, direction extrapolation channels, and closed-loop corrections. Each parameter can be determined using calibration samples before system deployment. Calibration samples include videos of low-altitude aircraft hovering, videos of aircraft arms or landing gear entering the field of view, videos of power lines or road edges extending into occlusion areas, low-texture videos of uniform skies or water surfaces, and videos of aircraft turning causing textures to deviate from occlusion boundaries.

[0054] The time synchronization threshold is determined based on the camera frame rate and the sampling frequency of the inertial measurement unit; the boundary stability threshold is determined based on the average displacement statistics of the candidate dynamic occlusion boundary relative to the dynamic occlusion boundary of the previous frame in the hovering state; the low texture judgment threshold is determined based on the spatial gradient intensity distribution of low texture samples and directional texture samples; the minimum velocity modulus threshold and the minimum effective inflow velocity threshold are determined based on the local optical flow modulus statistics of the static background samples; the boundary basic channel width, channel lateral growth coefficient, and boundary sampling radius are determined based on the minimum visible width of the object to be reconstructed and the local consistency range of the real visible texture near the dynamic occlusion boundary; the preset upper limit of orientation error, the preset upper limit of brightness error, and the preset upper limit of failure rate are determined based on the distribution of orientation consistency error, brightness continuity error, and time inversion failure rate in normal reconstruction samples and abnormal reconstruction samples, respectively.

[0055] In one calibration method, the percentile statistical value of the average displacement of the candidate dynamic occlusion boundary in the hovering state is used as the initial value of the boundary stability threshold; the upper quantile statistical value of the local optical flow modulus in the static background sample is used as the initial value of the minimum velocity modulus threshold; and the spatial gradient intensity separation point between low-texture samples and directional texture samples is used as the initial value of the low-texture judgment threshold. After the initial calibration is completed, the dynamic occlusion boundary jitter, historical frame inversion failure rate, and boundary continuity error are verified in the low-altitude inspection video stream to see if they are within the preset range. If they are not within the preset range, the corresponding parameters are adjusted and the verification is repeated. Thus, those skilled in the art can determine the parameters of this system according to the above calibration process without designing a new algorithm.

[0056] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0057] Those skilled in the art will recognize that the modules and algorithm modules of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0058] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0059] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0060] In conclusion, the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual blind spot reconstruction system for low-altitude aircraft based on adaptive image inference, characterized in that, It includes the following modules: a boundary calibration module, which compensates for the static mask of fuselage occlusion based on the current frame and flight attitude vector to obtain the dynamic occlusion mask, dynamic occlusion boundary and blind area to be reconstructed; The inflow determination module uses the dynamic occlusion boundary sampling point as the determination unit. Based on the current frame and historical frames, it obtains the local optical flow vector and the normal vector in the blind zone, calculates the projection of the local optical flow vector onto the normal vector, and obtains the inflow type or deviation type local inflow determination result. Based on the blind zone inflow projection amount, it obtains the historical frame extrapolation weight corresponding to the boundary sampling point, and uses the normalization direction of the optical flow vector as the main direction of local inference. The directional synthesis module determines the boundary starting point of the pixel to be reconstructed and associates it with the corresponding boundary sampling point. It selects the pixel reconstruction path based on the local inflow determination result and the historical frame extrapolation weight. When the local inflow determination result supports historical frame extrapolation and the historical frame inversion sampling is valid, the pixel channel value of the real visible sampled pixel in the historical frame is read along the opposite direction of the local optical flow vector as the pixel reconstruction value. Otherwise, pixel reconstruction values ​​are generated from the visible pixels in the neighborhood of the boundary starting point to form the blind zone reconstruction enhancement layer image; The closed-loop correction module obtains the directional consistency error, brightness continuity error, and temporal inversion failure rate based on the continuity error between the blind zone reconstruction enhancement layer image and the real visible side region of the dynamic occlusion boundary. When the directional consistency error exceeds the limit, it corrects the main direction of local extrapolation in the next frame. When the temporal inversion failure rate exceeds the limit, it reduces the extrapolation weight of the historical frame in the next frame. When the brightness continuity error exceeds the limit, it marks the boundary neighborhood and prioritizes the reconstruction of the boundary segment. The final output frame is obtained by fusion.

2. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 1, characterized in that, The boundary calibration module obtains the dynamic occlusion mask and dynamic occlusion boundary of the current frame as follows: It calculates the time difference between the camera sampling time and the inertial measurement unit (IMU) sampling time. When the time difference is not greater than the time synchronization threshold, the flight attitude vector is directly used as the attitude label of the current frame. When the time difference is greater than the time synchronization threshold, linear interpolation is performed on the pitch, roll, and yaw angles of the two adjacent IMU sampling points before and after the camera sampling time to obtain the flight attitude vector corresponding to the current frame. Based on the change in the flight attitude vector relative to the standard attitude, the horizontal displacement, vertical displacement, and boundary angle correction of the fuselage occlusion static mask are obtained by interpolation using adjacent attitude grid points in the airframe structure calibration table corresponding to the current aircraft model, camera installation position, lens distortion correction status, and mounting structure. Image coordinate transformation is then performed on the fuselage occlusion static mask using these displacements and corrections to obtain the dynamic occlusion mask of the current frame. If the airframe structure calibration table also records the boundary point coordinates… When calculating the offset, the boundary point coordinate offset corresponding to the current frame is obtained by interpolation based on adjacent attitude grid points, and the boundary is corrected for the current frame dynamic occlusion mask based on the interpolated boundary point coordinates. In the current frame dynamic occlusion mask obtained by compensating the static mask of the fuselage occlusion with the fuselage structure calibration table, the outer contour representing the fuselage occlusion area is extracted as the candidate dynamic occlusion boundary. When the average displacement of the candidate dynamic occlusion boundary relative to the dynamic occlusion boundary of the previous frame is not greater than the boundary stability threshold, the candidate dynamic occlusion boundary is used as the current dynamic occlusion boundary. When the average displacement is greater than the boundary stability threshold, the boundary points of the candidate dynamic occlusion boundary and the dynamic occlusion boundary of the previous frame are resampled and weighted and fused to obtain the current dynamic occlusion boundary. The connected region representing the fuselage occlusion area in the current frame dynamic occlusion mask is determined as the blind area to be reconstructed, and on the side of the dynamic occlusion boundary away from the blind area to be reconstructed, a real visible pixel area is selected according to the preset boundary reference width to generate the boundary reference area.

3. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 1, characterized in that, The inflow determination module obtains the local optical flow vector as follows: A single outer contour pixel on the dynamic occlusion boundary is used as a boundary sampling point. A preset local spatiotemporal sampling window is set with the boundary sampling point as the center. The preset local spatiotemporal sampling window includes a local spatial window size and a local temporal window length. The local temporal window length corresponds to the current frame and several historical frames preceding the current frame. The current frame and historical frames covered by the preset local spatiotemporal sampling window are preprocessed to obtain a brightness map for gradient calculation. Pixels located in the corresponding frame's truly visible area, not exceeding the image range, and with unsaturated brightness are taken as valid pixels. Horizontal gradient, vertical gradient, and temporal gradient are calculated at the valid pixels. The accumulated correlation components between the horizontal gradient, vertical gradient, and temporal gradient are weighted and summed to obtain the critical region spatiotemporal gradient tensor matrix. A local minimum error equation is established based on the spatial structure component and spatiotemporal coupling component in the critical region spatiotemporal gradient tensor matrix. The horizontal and vertical velocities are solved to obtain the horizontal and vertical velocities, and then combined into a local optical flow vector.

4. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 3, characterized in that, The inflow determination module obtains the blind zone inflow projection amount and historical frame extrapolation weight in the following way: the local tangent direction at the boundary sampling point is determined by the position change of adjacent outer contour pixels before and after the boundary sampling point, and the side that is perpendicular to the local tangent direction and points to the inside of the current frame dynamic occlusion mask is determined as the normal vector in the blind zone; the dot product of the local optical flow vector and the normal vector in the blind zone is divided by the sum of the vector norm of the local optical flow vector and the preset stability constant to obtain the blind zone inflow projection amount; the sum of the absolute values ​​of the blind zone inflow projection amount and the blind zone inflow projection amount is divided by 2 to obtain the historical frame extrapolation weight.

5. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 4, characterized in that, The inflow determination module outputs the blind zone inflow determination result in the following manner: When the blind zone inflow projection amount is greater than the blind zone inflow determination threshold, the vector norm of the local optical flow vector is not less than the minimum velocity modulus threshold, and the spatial gradient intensity of the effective pixels within the preset local spatiotemporal sampling window is not lower than the low texture determination threshold, an inflow-type blind zone inflow determination result is output, and the pixels to be reconstructed associated with the boundary sampling point are allowed to enter the historical frame time inversion sampling path; when the blind zone inflow projection amount is less than or equal to the blind zone inflow determination threshold, or the vector norm of the local optical flow vector is less than the minimum velocity modulus threshold, or the spatial gradient intensity of the effective pixels within the preset local spatiotemporal sampling window is lower than the low texture determination threshold, a divergence-type blind zone inflow determination result is output, the historical frame extrapolation weight is updated to zero, and the pixels to be reconstructed associated with the boundary sampling point are prohibited from entering the historical frame time inversion sampling path, causing them to switch to the boundary starting point neighborhood sampling path.

6. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 1, characterized in that, The directional synthesis module determines the boundary start point and direction extrapolation channel corresponding to the pixel to be reconstructed in the following way: For any pixel to be reconstructed within the blind zone, perform pixel-level backtracking from the pixel to be reconstructed along the opposite direction of the local extrapolation main direction. Take the intersection point with the dynamic occlusion boundary as the boundary start point, and take the boundary sampling point closest to the boundary start point as the associated boundary sampling point. Read the local inflow judgment result, local extrapolation main direction, and historical frame extrapolation weight corresponding to the associated boundary sampling point. If no effective intersection point can be obtained, select the boundary sampling point on the dynamic occlusion boundary closest to the pixel to be reconstructed. The pixel to be reconstructed is marked as the nearest neighbor matching pixel, starting from the boundary starting point. The distance from the pixel to be reconstructed into the blind zone is measured along the main direction of local deduction, starting from the boundary starting point, as the vertical projection depth. The deviation of the pixel to be reconstructed from the main direction of local deduction is measured along the vertical direction perpendicular to the main direction of local deduction as the lateral offset. When the vertical projection depth is not less than zero and the lateral offset is not greater than the sum of the boundary base channel width and the channel lateral growth coefficient multiplied by the vertical projection depth, the pixel to be reconstructed is determined to belong to the direction deduction channel; otherwise, the pixel to be reconstructed is determined not to belong to the direction deduction channel.

7. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 6, characterized in that, The directional synthesis module locates historical frame sampling pixels in the following manner: when the local inflow determination result corresponding to the associated boundary sampling point is inflow type, the historical frame extrapolation weight is greater than the preset extrapolation weight threshold, and the pixel to be reconstructed belongs to the directional inference channel, historical frame time inversion sampling is allowed. The projection modulus of the local optical flow vector in the local extrapolation principal direction is taken as the effective inflow velocity. The longitudinal projection depth is divided by the effective inflow velocity and rounded up to obtain the time inversion step number. When the effective inflow velocity is less than the minimum effective inflow velocity threshold, or the time inversion step number is greater than the maximum number of frames supported by the historical frame buffer, the current pixel to be reconstructed is marked as an inversion failure pixel. When the effective inflow velocity is not less than the minimum effective inflow velocity threshold and the time inversion step number is not greater than the maximum number of frames supported by the historical frame buffer, the displacement corresponding to the time inversion step number is traced back from the image coordinates of the current pixel to be reconstructed along the opposite direction of the local optical flow vector to obtain the historical frame sampled pixel in the time inversion sampling frame. If the historical frame sampled pixel is located in the occluded area of ​​the time inversion sampling frame or exceeds the image range, the current pixel to be reconstructed is marked as an inversion failure pixel. If the historical frame sampled pixel is located in the truly visible area, the pixel channel value of the historical frame sampled pixel in the truly visible state is read.

8. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 7, characterized in that, The directional synthesis module performs boundary starting point neighborhood sampling on pixels to be reconstructed that are determined to be divergent by local inflow, pixels to be reconstructed whose historical frame extrapolation weight is not greater than the preset extrapolation weight threshold, pixels to be reconstructed outside the directional inference channel, nearest neighbor matching pixels, and pixels that fail to be inverted. The boundary starting point neighborhood sampling includes: taking the boundary starting point corresponding to the pixel to be reconstructed as the center, selecting real visible and unsaturated effective pixels in the boundary reference region within the preset boundary sampling radius, and taking the arithmetic mean of the pixel channel values ​​of the effective pixels as the pixel reconstruction value of the pixel to be reconstructed; when the current frame is a color image frame, performing neighborhood sampling on the luminance channel value and the chrominance channel value respectively; generating the pixel reconstruction value of each pixel to be reconstructed in the blind area in the order of vertical projection depth from small to large, and writing the pixel reconstruction value into the image buffer of the same size as the dynamic occlusion mask of the current frame according to the position to obtain the blind area reconstruction enhancement layer image.

9. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 1, characterized in that, The closed-loop correction module obtains the continuity error as follows: Ring-shaped verification windows with a preset verification window width are set on both sides of the dynamic occlusion boundary. The verification window inside the blind zone is located on one side of the blind zone to be reconstructed and is used to read the local information of the reconstruction boundary in the enhancement layer image of the blind zone reconstruction. The verification window on the truly visible side is located on one side of the boundary reference region and is used to read the local information of the true boundary in the current frame. The horizontal and vertical gradients are calculated for the boundary sampling points within each verification window. The gradient directions of each pixel are weighted and summed according to their gradient magnitudes and then normalized. The reconstruction direction vector inside the blind zone is obtained, and the true visible side is obtained. The visible side direction vector is used as the magnitude of the difference between the reconstructed direction vector inside the blind zone and the true visible side direction vector, which is taken as the direction consistency error. The reconstructed brightness inside the blind zone is obtained by arithmetically averaging the brightness channel values ​​of the reconstructed enhancement layer image inside the blind zone within the verification window. The true visible side brightness is obtained by arithmetically averaging the brightness channel values ​​of the current frame within the verification window. The brightness continuity error is obtained by dividing the absolute value of the difference between the reconstructed brightness inside the blind zone and the true visible side brightness by the sum of the true visible side brightness and the preset stability constant. The temporal inversion failure rate is obtained by statistically analyzing the proportion of inversion failure pixels in the total pixels of the blind zone to be reconstructed.

10. The low-altitude aircraft visual blind spot reconstruction system based on adaptive image inference according to claim 9, characterized in that, The closed-loop correction module performs write-back corrections on the local extrapolation main direction, historical frame extrapolation weights, and boundary segment priority reconstruction of the boundary neighborhood in the next frame in the following manner: When the direction consistency error is greater than the preset direction error upper limit, the current local inference main direction is fused with the actual visible side direction vector and then normalized to obtain the local inference main direction of the next frame; when the direction consistency error is not greater than the preset direction error upper limit, the current local inference main direction is used in the next frame. When the time inversion failure rate is greater than the preset failure rate limit, the extrapolation weight of the current historical frame is divided by the time inversion failure rate to obtain the extrapolation weight of the next historical frame. When the time reversal failure rate is not greater than the preset failure rate limit, the next frame retains the extrapolation weight of the current historical frame. When the brightness continuity error exceeds the preset brightness error upper limit, the corresponding boundary sampling point is marked as a brightness over-limit sampling point, and adjacent brightness over-limit sampling points are connected and grouped according to the sampling order of the dynamic occlusion boundary to obtain a brightness over-limit boundary segment; when the number of sampling points in the brightness over-limit boundary segment is not less than the preset minimum number of continuous points, the brightness over-limit boundary segment is marked as a boundary neighborhood priority reconstruction boundary segment, so that the pixels to be reconstructed associated with the boundary segment in the next frame will preferentially use the visible pixels in the neighborhood of the boundary starting point to generate pixel reconstruction values; When the brightness continuity error is not greater than the preset brightness error upper limit, the next frame maintains the current reconstruction path; After completing the above write-back correction, the current frame dynamic occlusion mask is used as the fusion basis. The original pixels of the current frame in the real visible area are retained. The occlusion area of ​​the machine is replaced with the reconstructed pixel values ​​of the same coordinates in the blind zone reconstruction enhancement layer image. The pixels at the dynamic occlusion boundary are linearly transitioned according to the boundary distance to obtain the final output frame.

Images