Image data processing method and system based on heterogeneous computing platform

Abstract

This invention relates to the field of image data processing technology, specifically to an image data processing method and system based on a heterogeneous computing platform. The method includes acquiring multiple extravehicular image streams, head posture data, and aircraft posture data; determining the target observation viewpoint based on the aircraft posture data and head posture data; determining the corresponding imaging viewpoint based on the spatial installation parameters of each imaging device; mapping image frames to an imaging frustum to obtain the image field of view, and extracting image sub-regions; calculating viewpoint deviation parameters within the overlapping region based on the imaging spatial position, imaging direction, and the spatial difference between the observation spatial position and observation direction; and performing spatial reprojection processing and weighted fusion processing to obtain a composite image. This invention can improve the consistency between the composite image and the target observation viewpoint and enhance the fusion stability of the overlapping region.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, specifically to image data processing methods and systems based on heterogeneous computing platforms. Background Technology

[0002] With the development of airborne image sensing, helmet-mounted displays, and airborne computing platforms, acquiring external images using imaging devices installed at different locations on the aircraft and combining this with the pilot's head posture information to generate a display screen that matches the current observation direction has become an important technological direction in the field of airborne graphics and image data processing. The pilot's head posture changes in real time, and the display screen needs to be updated rapidly according to the observation direction. Therefore, the image processing process involves not only the spatial geometric relationships between multiple external images but also the viewpoint transformation relationships caused by changes in the observation direction.

[0003] Existing methods include two types: one type performs geometric mapping and stitching of multiple extravehicular images based on the installation parameters of the imaging device, and completes the composite display by determining the overlapping area, seam area or projection area of ​​the images; the other type calculates the current observation direction based on head posture data, then extracts the corresponding field of view from the extravehicular images, and adjusts the displayed image through delay compensation.

[0004] In airborne helmet-mounted display scenarios, multiple external images typically pass through different image acquisition, transmission, and processing links, with latency differences between each link. Simultaneously, head and aircraft postures continuously change, dynamically altering the viewpoint. When there are differences between the image acquisition time, posture acquisition time, and display time, even with conventional geometric stitching or overall latency compensation, the composite image may still exhibit localized drift, jumps, or double images in overlapping areas or adjacent seams that are inconsistent with the current viewpoint. Summary of the Invention

[0005] The purpose of this invention is to provide an image data processing method and system based on a heterogeneous computing platform to solve the problem that, in airborne dynamic observation scenarios, when multiple external images are displayed in a composite manner around the current observation viewpoint, the content of local images is difficult to keep consistent with the current observation viewpoint.

[0006] To achieve the above objectives, in one aspect, the present invention provides an image data processing method based on a heterogeneous computing platform, the method comprising: Step S1: Acquire multiple extravehicular image streams output by imaging devices at different fixed installation positions, as well as the spatial installation parameters of each imaging device; acquire head attitude data and body attitude data.

[0007] Step S2: The central processing unit determines the target observation viewpoint based on the body posture data and head posture data. The target observation viewpoint includes the observation spatial position and observation direction in the geographic coordinate system. The imaging spatial position and imaging direction of the corresponding imaging viewpoint in the geographic coordinate system are determined based on the spatial installation parameters of each imaging device.

[0008] Step S3: The graphics processor maps the image frames in the multi-channel extravehicular image stream to the corresponding imaging frustum in the geographic coordinate system to obtain the image field of view corresponding to each imaging viewpoint; the graphics processor extracts the corresponding image sub-region from each image field of view according to the target observation viewpoint; the central processing unit determines the overlapping region according to the spatial overlap relationship between the image sub-regions.

[0009] Step S4: The central processing unit calculates the viewpoint deviation parameters of each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint; the graphics processing unit performs spatial reprojection processing on each image sub-region based on the viewpoint deviation parameters to obtain reprojected image sub-regions; the graphics processing unit performs weighted fusion processing on each reprojected image sub-region within the overlapping region based on the viewpoint deviation parameters of each reprojected image sub-region, and retains the corresponding reprojected image sub-regions in the non-overlapping region to obtain a composite image.

[0010] Furthermore, the method for determining the target observation viewpoint based on body posture data and head posture data includes: Head posture data sequences and body posture data sequences are constructed based on head posture data and body posture data, with each head posture data and each body posture data corresponding to a timestamp.

[0011] A unified target time is determined based on the timestamp; the attitude change between adjacent times is calculated based on the head attitude data sequence and the body attitude data sequence; time interpolation is performed on the head attitude data and body attitude data based on the attitude change to obtain aligned attitude data corresponding to the unified target time.

[0012] The observation spatial position and observation direction are determined based on the alignment posture data; the target observation viewpoint is determined based on the observation spatial position and observation direction.

[0013] Furthermore, the method for determining the observation spatial position and observation direction based on the alignment posture data includes: Calculate the head posture change rate based on the head posture data sequence; calculate the body posture change rate based on the body posture data sequence; obtain the time compensation amount from the posture data acquisition time to the image display time.

[0014] Based on the head posture change rate, body posture change rate, and time compensation, the alignment posture data is extrapolated along the time axis to obtain the predicted posture data; the observation spatial position and observation direction are determined based on the predicted posture data.

[0015] Furthermore, the method for extracting corresponding image sub-regions from each image field of view based on the target viewing point includes: The target observation field of view is determined according to the observation direction of the target observation viewpoint; the target observation field of view is mapped to each image field of view to obtain the target projection area; and overlapping candidate areas are determined within the target projection area according to the spatial overlap relationship between the image fields of view.

[0016] Based on the spatial distance between the overlapping candidate regions and the boundaries of each image field of view, the regions in the overlapping candidate regions whose spatial distance from the boundaries of the image field of view is less than a preset spatial distance threshold are determined as the initial seam adjacent regions.

[0017] The target projection region is divided according to the initial seam adjacent region to obtain the seam adjacent region and the non-seam adjacent region; the image sub-region includes the seam adjacent region and the non-seam adjacent region.

[0018] Furthermore, the method for dividing the target projection region based on the initial seam adjacent region to obtain the seam adjacent region and the non-seam adjacent region includes: The initial seam adjacent region position sequence is determined based on the initial seam adjacent region at consecutive time points; the change in region position between adjacent time points is calculated based on the initial seam adjacent region position sequence.

[0019] Based on the change in the region's location, the initial seam adjacent region at the current moment is smoothly adjusted to obtain the seam adjacent region; the region in the target projection region other than the seam adjacent region is taken as the non-seam adjacent region.

[0020] Furthermore, the method for calculating the viewpoint deviation parameter of each image sub-region within the overlapping region based on the spatial difference relationship between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint includes: Based on the spatial location of the image sub-region in the geographic coordinate system, and combined with the imaging spatial location, imaging direction, observation spatial location, and observation direction, the imaging line-of-sight vector and the observation line-of-sight vector corresponding to the image sub-region are constructed respectively.

[0021] Based on the angle between the imaging line-of-sight vector and the observation line-of-sight vector, as well as the scene distance information corresponding to the image sub-region, the pixel offset of the image sub-region is calculated as the basic viewpoint deviation.

[0022] The image sub-region is divided into multiple distance layers based on the scene distance information; the basic viewpoint deviation is weighted and normalized within each distance layer to obtain the layered deviation corresponding to each distance layer; the image sub-region is subjected to deviation mapping processing based on the layered deviation to obtain the viewpoint deviation parameter of the image sub-region.

[0023] Furthermore, the method for weighted normalizing the basic viewpoint deviation within each distance layer to obtain the layer deviation corresponding to each distance layer includes: The relative positional distance between the image sub-region and the center position of the adjacent seam region is determined based on the position of the image sub-region in the overlapping region.

[0024] Within the same distance layer, the image sub-region is divided into a seam center region and a seam edge region based on the relative position distance; the local deviation adjustment coefficients corresponding to the seam center region and the seam edge region are calculated respectively.

[0025] Based on the local deviation adjustment coefficient, the basic viewpoint deviation corresponding to the center region of the joint is subjected to high-precision deviation constraint processing, and the basic viewpoint deviation corresponding to the edge region of the joint is subjected to low-precision deviation constraint processing, so as to obtain the layered deviation amount after joint sensitivity refinement.

[0026] Furthermore, the method for weighted fusion processing of each reprojected image sub-region within the overlapping region based on the viewpoint deviation parameter of each reprojected image sub-region includes: The corresponding pixel position distribution is determined based on the spatial position of the reprojected image sub-region in the overlapping region; a spatial weight distribution function is constructed based on the pixel position distribution; a deviation weight coefficient is calculated based on the viewpoint deviation parameter corresponding to each reprojected image sub-region; and a deviation weight distribution function is constructed based on the deviation weight coefficient.

[0027] Based on the spatial weight distribution function and the bias weight distribution function, the pixels in the overlapping region of the reprojected image sub-region are jointly weighted to obtain an initial fused image; an initial fused image sequence is obtained; the pixel change between adjacent time steps is calculated based on the initial fused image sequence; and the initial fused image at the current time step is temporally smoothed based on the pixel change to obtain a fused image.

[0028] Based on the same inventive concept, this invention also provides an image data processing system based on a heterogeneous computing platform, the system comprising: The data acquisition module is used to acquire multiple extravehicular image streams output by imaging devices at different fixed installation positions, as well as the spatial installation parameters of each imaging device; and to acquire head attitude data and body attitude data.

[0029] The viewpoint construction module is used by the central processing unit to determine the target observation viewpoint based on the body posture data and head posture data. The target observation viewpoint includes the observation spatial position and observation direction in the geographic coordinate system. The module also determines the imaging spatial position and imaging direction of the corresponding imaging viewpoint in the geographic coordinate system based on the spatial installation parameters of each imaging device.

[0030] The region extraction module is used by the graphics processor to map image frames from multiple extravehicular image streams to corresponding imaging frustums in the geographic coordinate system, thereby obtaining the image field of view for each imaging viewpoint; the graphics processor extracts corresponding image sub-regions from each image field of view based on the target observation viewpoint; and the central processing unit determines the overlapping region based on the spatial overlap relationship between the image sub-regions.

[0031] The fusion processing module is used by the central processing unit to calculate the viewpoint deviation parameters of each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint; the graphics processing unit performs spatial reprojection processing on each image sub-region based on the viewpoint deviation parameters to obtain reprojected image sub-regions; and the graphics processing unit performs weighted fusion processing on each reprojected image sub-region within the overlapping region based on the viewpoint deviation parameters of each reprojected image sub-region, retaining the corresponding reprojected image sub-regions in the non-overlapping region to obtain a composite image.

[0032] Compared with the prior art, the beneficial effects of the present invention are: 1. By determining the target observation viewpoint based on the aircraft attitude data and head attitude data, and combining the imaging spatial position and imaging direction of each imaging device, multiple external images are mapped to the image field of view and image sub-regions are extracted. In the overlapping area, the viewpoint deviation parameter is calculated based on the spatial difference relationship between the imaging viewpoint and the target observation viewpoint, and spatial reprojection and weighted fusion processing are performed to enable the composite image to be constructed around the target observation viewpoint, thereby reducing image misalignment and jump phenomena in the overlapping area and improving the spatial consistency of the composite image.

[0033] 2. Based on scene distance information, the image sub-regions are divided into distance layers. Within each distance layer, the basic viewpoint deviation is weighted and normalized by combining the positional relationship of the seam adjacent regions to obtain the layered deviation amount. Deviation mapping is then performed to create differentiated control between the seam adjacent regions and the non-seam adjacent regions in terms of deviation constraint, thereby further improving the fusion stability of the seam adjacent regions and reducing local discontinuities. Attached Figure Description

[0034] Figure 1 This is a flowchart of the image data processing method based on a heterogeneous computing platform according to the present invention; Figure 2This is a block diagram of the image data processing system based on a heterogeneous computing platform according to the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Before providing examples, it's necessary to describe the application scenario of this invention. This invention is applicable to airborne helmet-mounted displays, where multiple imaging devices are fixedly installed at different locations on the aircraft, acquiring external images from different perspectives. The pilot's head posture changes in real time, and the observation point dynamically changes accordingly, requiring the displayed image to be continuously updated around the current observation point. In this scenario, the airborne computing platform employs a heterogeneous computing architecture where a central processing unit (CPU) and a graphics processing unit (GPU) work collaboratively. The CPU performs attitude data processing, observation point determination, and spatial relationship calculations, while the GPU performs image mapping, image sub-region extraction, spatial reprojection, and fusion processing. Multiple external images, head posture data, and aircraft posture data enter the heterogeneous computing platform through different data acquisition, transmission, and processing links, resulting in differences in task allocation and processing timing between different processing units. When the observation point continuously changes, the content of multiple images is more prone to local deviations from the current observation point in overlapping areas or adjacent seams.

[0037] Example 1: As Figure 1 As shown, this embodiment provides an image data processing method based on a heterogeneous computing platform, the method comprising: The system acquires multiple extravehicular image streams from imaging devices at different fixed installation locations, along with the spatial installation parameters of each imaging device; it also acquires head attitude data and body attitude data. The heterogeneous computing platform includes a central processing unit (CPU) and a graphics processing unit (GPU).

[0038] For example, in the airborne environment of a certain type of aircraft, six external images are acquired by six high-definition cameras installed at different locations on the fuselage. Each camera outputs an image stream with a resolution of 1920×1080 and a frame rate of 60 frames per second. Each frame of the image is accompanied by a microsecond-level acquisition timestamp synchronized based on the 1588 protocol. The spatial installation parameters of the six cameras include an intrinsic parameter matrix and an extrinsic parameter matrix. The intrinsic parameter matrix represents the focal length and principal point coordinates, while the extrinsic parameter matrix represents the installation spatial position and imaging direction of the camera in the aircraft coordinate system. The spatial installation parameters are determined during the equipment calibration phase and stored in the non-volatile memory of the airborne processing platform.

[0039] Head attitude data is acquired by an inertial measurement unit integrated into the pilot's helmet at a sampling frequency of 200 Hz. Each set of head attitude data includes pitch, roll, and yaw angles, and is accompanied by a timestamp synchronized based on the 1588 protocol. Aircraft attitude data is output from the aircraft's inertial navigation system (INS) and transmitted via the ARINC429 bus at a frequency of 100 Hz. Each set of aircraft attitude data includes pitch, roll, and yaw angles, corresponding to the acquisition time within the INS. The central processing unit (CPU) converts the acquisition time of the aircraft attitude data to a time reference consistent with the external image stream and head attitude data, based on the time mapping relationship between the INS time and a unified clock source. Simultaneously, the CPU obtains the aircraft's spatial position in the geographic coordinate system from the INS. This spatial position is represented by latitude, longitude, and altitude, and converted to three-dimensional Cartesian coordinates. This spatial position is acquired synchronously with the aircraft attitude data and is accompanied by a unified timestamp.

[0040] When the central processing unit receives the external image stream, head attitude data, and body attitude data, it records the arrival time of each data packet and compares the arrival time with the acquisition timestamp in the data packet. It then performs statistical calculations on multiple consecutive frames of data to obtain the average latency value of each data link under the current operating state. The link latency of each external image stream from camera exposure to central processing unit reception is approximately 32 milliseconds, the link latency of head attitude data is approximately 8 milliseconds, and the link latency of body attitude data is approximately 12 milliseconds.

[0041] The central processing unit determines the target observation viewpoint based on the body posture data and head posture data. The target observation viewpoint includes the observation spatial location and observation direction in a geographic coordinate system. The method for determining the target observation viewpoint based on the body posture data and head posture data includes: Head posture data sequences and body posture data sequences are constructed based on head posture data and body posture data, with each head posture data and each body posture data corresponding to a timestamp.

[0042] A unified target time is determined based on the timestamp; the attitude change between adjacent times is calculated based on the head attitude data sequence and the body attitude data sequence; time interpolation is performed on the head attitude data and body attitude data based on the attitude change to obtain aligned attitude data corresponding to the unified target time.

[0043] The observation spatial position and observation direction are determined based on the alignment posture data; the method for determining the observation spatial position and observation direction based on the alignment posture data includes: Calculate the head posture change rate based on the head posture data sequence; calculate the body posture change rate based on the body posture data sequence; obtain the time compensation amount from the posture data acquisition time to the image display time.

[0044] Based on the head posture change rate, body posture change rate, and time compensation, predicted posture data is obtained by extrapolating the aligned posture data along the time axis; the observation spatial position and observation direction are determined based on the predicted posture data. The target observation viewpoint is then determined based on the observation spatial position and observation direction.

[0045] The spatial location and direction of the corresponding imaging viewpoint in the geographic coordinate system are determined based on the spatial installation parameters of each imaging device.

[0046] For example, the central processing unit extracts the most recent 20 frames of head pose data and the most recent 10 frames of body pose data from the data buffer, and constructs the head pose data sequence and the body pose data sequence respectively according to the timestamp order. Each set of head pose data and each set of body pose data corresponds to a timestamp based on a unified time base. The length of the head pose data sequence is greater than the length of the body pose data sequence to match the higher sampling frequency of the head pose data.

[0047] The central processing unit (CPU) uses the vertical synchronization signal time corresponding to the current image frame as the unified target time, with a time accuracy at the microsecond level. The CPU selects two sets of head attitude data from the head attitude data sequence, with timestamps before and after the unified target time, denoted as H1 (timestamp T1) and H2 (timestamp T2), respectively. It calculates the attitude change based on the attitude difference between H1 and H2, and then, based on the attitude change and the time interval, converts the pitch, roll, and yaw angles into quaternion representations before performing spherical linear interpolation to obtain the head attitude data at the unified target time. The same method is used to select two adjacent sets of data from the body attitude data sequence and perform interpolation calculations to obtain the body attitude data at the unified target time. The head attitude data and body attitude data constitute aligned attitude data.

[0048] The central processing unit (CPU) extracts the five most recent frames of data from the head attitude data sequence, calculates the attitude difference between two adjacent frames and divides it by the corresponding time interval to obtain the rates of change of pitch, roll, and yaw angles. The arithmetic mean of these rates of change at adjacent moments is then taken to obtain the head attitude change rate. The body attitude data sequence is processed in the same way to calculate the body attitude change rate. The CPU acquires the time compensation value, which is obtained by measuring and accumulating the delays in image acquisition, data transmission, CPU processing, and graphics processor rendering. In this embodiment, the time compensation value is 58 milliseconds.

[0049] The central processing unit (CPU) calculates the rotation increment of the head posture based on the head posture change rate and time compensation amount, and represents the rotation increment as a quaternion. The CPU then multiplies the rotation increment with the quaternion corresponding to the head posture data to obtain the predicted head posture data. The CPU also calculates the rotation increment of the body posture based on the body posture change rate and time compensation amount, and multiplies the rotation increment with the quaternion corresponding to the body posture data to obtain the predicted body posture data.

[0050] The central processing unit (CPU) constructs a rotation matrix of the head coordinate system relative to the body coordinate system based on the predicted head posture data, transforming the observation direction vector [0,0,1] in the head coordinate system to the body coordinate system, thus obtaining the observation direction vector in the body coordinate system. The CPU then constructs a rotation matrix of the body coordinate system relative to the geographic coordinate system based on the predicted body posture data, transforming the observation direction vector in the body coordinate system to the geographic coordinate system, thus obtaining the observation direction vector in the geographic coordinate system. The CPU then sequentially transforms the origin of the head coordinate system to the geographic coordinate system using the aforementioned rotation matrix, and superimposes the spatial position of the body in the geographic coordinate system to obtain the observation spatial position in the geographic coordinate system.

[0051] The central processing unit (CPU) combines the observation spatial position and observation direction to obtain the target observation viewpoint. The CPU reads the spatial installation parameters of each imaging device from non-volatile memory, and transforms the installation spatial position and imaging direction of each imaging device in the body coordinate system to the geographic coordinate system based on the predicted body attitude data B_p and the body's spatial position in the geographic coordinate system, thus obtaining the corresponding imaging spatial position and imaging direction of each imaging device.

[0052] The graphics processor maps image frames from multiple extravehicular image streams to corresponding imaging frustums in a geographic coordinate system, obtaining the image field of view for each imaging viewpoint; the graphics processor extracts corresponding image sub-regions from each image field of view based on the target observation viewpoint; the method for extracting corresponding image sub-regions from each image field of view based on the target observation viewpoint includes: The target observation field of view is determined according to the observation direction of the target observation viewpoint; the target observation field of view is mapped to each image field of view to obtain the target projection area; and overlapping candidate areas are determined within the target projection area according to the spatial overlap relationship between the image fields of view.

[0053] Based on the spatial distance between the overlapping candidate regions and the boundaries of each image field of view, the regions in the overlapping candidate regions whose spatial distance from the boundaries of the image field of view is less than a preset spatial distance threshold are determined as the initial seam adjacent regions.

[0054] The target projection area is divided according to the initial seam adjacent area to obtain seam adjacent area and non-seam adjacent area; the method of dividing the target projection area according to the initial seam adjacent area to obtain seam adjacent area and non-seam adjacent area includes: The initial seam adjacent region position sequence is determined based on the initial seam adjacent region at consecutive time points; the change in region position between adjacent time points is calculated based on the initial seam adjacent region position sequence.

[0055] Based on the change in the region's position, the initial seam adjacent region at the current moment is smoothed to obtain the seam adjacent region; the region in the target projection region other than the seam adjacent region is designated as the non-seam adjacent region. The image sub-region includes the seam adjacent region and the non-seam adjacent region.

[0056] The central processing unit determines the overlapping region based on the spatial overlap relationship between the image sub-regions.

[0057] For example, the graphics processor obtains the target viewing point from the central processing unit. The target viewing point includes the three-dimensional coordinates of the viewing location in the geographic coordinate system and the viewing direction vector. The graphics processor constructs a viewing frustum based on the viewing direction vector and preset horizontal field of view of 60 degrees and vertical field of view of 40 degrees. The vertex of the viewing frustum is located at the viewing location in the viewing space, and its axis is consistent with the viewing direction vector.

[0058] The graphics processor (GPU) sequentially processes the current frame image from multiple external image streams. For each image stream, the GPU acquires the imaging spatial position, imaging direction, and intrinsic parameter matrix of the imaging device for the corresponding imaging viewpoint. The GPU transforms the observation frustum into the camera coordinate system corresponding to the imaging viewpoint, and performs spatial intersection calculations between the observation frustum and the imaging frustum corresponding to the imaging viewpoint. The resulting 3D region is then projected onto the image plane using the intrinsic parameter matrix to obtain the target projection region. During the projection process, a homogeneous coordinate transformation is performed on the boundary plane of the observation frustum, and the corresponding pixel coordinate range is calculated based on the perspective projection relationship to obtain the pixel set corresponding to the target projection region.

[0059] After obtaining the target projection region for each image, the graphics processor calculates the spatial overlap relationship between the target projection regions in the image plane. For any two images, if the target projection regions intersect in the image plane, the intersection region is determined as the candidate region for overlap.

[0060] For each overlapping candidate region, the graphics processor calculates the image plane distance from each pixel within the overlapping candidate region to the corresponding image field of view boundary based on a distance transformation algorithm, resulting in a distance distribution map. The distance transformation uses Euclidean distance calculation to calculate the shortest distance from each pixel to the polygon of the image field of view boundary. The graphics processor extracts the set of pixels in the distance distribution map whose distance values ​​are less than a preset spatial distance threshold as the initial seam adjacency region. In this embodiment, the preset spatial distance threshold is set to 25 pixels.

[0061] The graphics processor (GPU) constructs an initial seam adjacent region position sequence based on the initial seam adjacent regions at consecutive time steps. Each frame uses a set of boundary contour points to represent the position of the initial seam adjacent region. The GPU calculates the change in region position between adjacent time steps based on the initial seam adjacent region position sequence; this change is represented as the displacement vector of the corresponding boundary contour points. The GPU then performs a smooth adjustment on the boundary of the initial seam adjacent region at the current time step based on the change in region position. The smoothing adjustment uses an exponentially weighted moving average method, weighting the boundary contour points of the current frame with the corresponding boundary contour points of the previous frame to obtain the updated boundary contour, thus obtaining the seam adjacent region.

[0062] The graphics processor (GPU) defines the set of pixels in the target projection region, excluding the seam-adjacent regions, as the non-seam-adjacent regions. The seam-adjacent regions and the non-seam-adjacent regions together constitute image sub-regions. The GPU obtains corresponding image sub-regions for each image stream. The central processing unit (CPU) acquires each image sub-region and calculates the spatial overlap relationship between them within the image plane. For any two image sub-regions, if their corresponding regions intersect in the image plane, the intersection region is defined as the overlapping region.

[0063] The central processing unit calculates viewpoint deviation parameters for each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint; the method for calculating viewpoint deviation parameters for each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint includes: Based on the spatial location of the image sub-region in the geographic coordinate system, and combined with the imaging spatial location, imaging direction, observation spatial location, and observation direction, the imaging line-of-sight vector and the observation line-of-sight vector corresponding to the image sub-region are constructed respectively.

[0064] Based on the angle between the imaging line-of-sight vector and the observation line-of-sight vector, as well as the scene distance information corresponding to the image sub-region, the pixel offset of the image sub-region is calculated as the basic viewpoint deviation.

[0065] The image sub-region is divided into multiple distance layers based on the scene distance information; within each distance layer, the basic viewpoint deviation is weighted and normalized to obtain the layered deviation corresponding to each distance layer; the method for weighting and normalizing the basic viewpoint deviation within each distance layer to obtain the layered deviation corresponding to each distance layer includes: The relative positional distance between the image sub-region and the center position of the adjacent seam region is determined based on the position of the image sub-region in the overlapping region.

[0066] Within the same distance layer, the image sub-region is divided into a seam center region and a seam edge region based on the relative position distance; the local deviation adjustment coefficients corresponding to the seam center region and the seam edge region are calculated respectively.

[0067] Based on the local deviation adjustment coefficient, high-precision deviation constraint processing is applied to the basic viewpoint deviation corresponding to the center region of the seam, and low-precision deviation constraint processing is applied to the basic viewpoint deviation corresponding to the edge region of the seam, resulting in a layered deviation amount after seam sensitivity refinement. Based on the layered deviation amount, deviation mapping processing is performed on the image sub-region to obtain the viewpoint deviation parameters of the image sub-region.

[0068] The graphics processor performs spatial reprojection processing on each image sub-region according to the viewpoint deviation parameter to obtain reprojected image sub-regions; within the overlapping region, the graphics processor performs weighted fusion processing on each reprojected image sub-region according to the viewpoint deviation parameter of each reprojected image sub-region, and retains the corresponding reprojected image sub-region in the non-overlapping region to obtain a composite image. The method of performing weighted fusion processing on each reprojected image sub-region according to the viewpoint deviation parameter within the overlapping region includes: The corresponding pixel position distribution is determined based on the spatial position of the reprojected image sub-region in the overlapping region; a spatial weight distribution function is constructed based on the pixel position distribution; a deviation weight coefficient is calculated based on the viewpoint deviation parameter corresponding to each reprojected image sub-region; and a deviation weight distribution function is constructed based on the deviation weight coefficient.

[0069] Based on the spatial weight distribution function and the bias weight distribution function, the pixels in the overlapping region of the reprojected image sub-region are jointly weighted to obtain an initial fused image; an initial fused image sequence is obtained; the pixel change between adjacent time steps is calculated based on the initial fused image sequence; and the initial fused image at the current time step is temporally smoothed based on the pixel change to obtain a fused image.

[0070] For example, the central processing unit (CPU) acquires the spatial position of each image sub-region within the overlapping area in the geographic coordinate system. Using the image sub-region as a calculation unit, the CPU back-projects the scene distance information corresponding to the center pixel of the image sub-region along the corresponding line-of-sight direction to obtain the three-dimensional spatial coordinates of the corresponding point in the image sub-region. The scene distance information is synchronously acquired by the airborne LiDAR at the moment of image frame acquisition and stored in the form of a depth map, with each pixel corresponding to a distance value in meters.

[0071] The central processing unit (CPU) constructs an imaging gaze vector and an observation gaze vector for each image sub-region within the overlapping area. The imaging gaze vector is obtained by pointing from the imaging spatial position to the three-dimensional spatial point and normalized to a unit vector; the observation gaze vector is obtained by pointing from the observation spatial position to the same three-dimensional spatial point and normalized to a unit vector. Within each image sub-region, the CPU performs bilinear interpolation on the gaze vectors based on pixel coordinates to obtain the imaging gaze vector and observation gaze vector corresponding to each pixel.

[0072] The central processing unit calculates the angle between the imaging line-of-sight vector and the observation line-of-sight vector. The included angle Represented in radians, and the scene distance of the corresponding pixel is obtained. The central processing unit (CPU) determines the angle based on the angle. and scene distance Calculate pixel offset The calculation relationship is as follows ,in The focal length of the imaging device is expressed in pixels; in this embodiment, it is set to 1200 pixels. For reference distance, a value of 100 meters is used in this embodiment. The central processing unit will... As the basic viewpoint deviation.

[0073] The central processing unit (CPU) determines the distance based on the scene. The pixels within the overlapping area are processed by distance layering, dividing the pixels into multiple distance layers. In this embodiment, they are divided into three distance layers: the near-field layer corresponds to... Meters, corresponding to the mid-ground layer Meters, corresponding to the distant view layer Meters. The central processing unit performs weighted normalization on the basic viewpoint deviation within each distance layer. It divides the basic viewpoint deviation of each pixel in the same distance layer by the maximum basic viewpoint deviation in that distance layer to obtain the normalized value, which is used as the layer deviation of the corresponding distance layer.

[0074] The central processing unit (CPU) calculates the image planar distance from each pixel to the center of the seam adjacent region based on the pixel position of each image sub-region in the overlapping region. The center of the seam adjacent region is determined by the centroid coordinates of the boundary contour point set of the seam adjacent region. Within the same distance layer, the CPU divides pixels into regions based on the image planar distance. When the image planar distance is less than 15 pixels, it is divided into the seam center region, and the rest are divided into the seam edge region.

[0075] The central processing unit (CPU) sets local deviation adjustment coefficients for the center region and edge region of the seam. In this embodiment, the local deviation adjustment coefficient for the center region of the seam is 1.0, and the local deviation adjustment coefficient for the edge region of the seam is 0.4. The CPU performs weighted processing on the layer deviation amount according to the local deviation adjustment coefficients to obtain the layer deviation amount after refining the seam sensitivity. The center region of the seam uses a full correction method to maintain the basic viewpoint deviation amount, while the edge region of the seam uses an attenuation correction method to reduce the basic viewpoint deviation amount.

[0076] The central processing unit performs deviation mapping processing based on the refined layer deviation amount after seam sensitivity, mapping the layer deviation amount to a preset maximum reprojection displacement to obtain the reprojection displacement parameter. In this embodiment, the maximum reprojection displacement is set to 24 pixels, and the viewpoint deviation parameter includes the horizontal displacement amount. and vertical displacement .

[0077] The graphics processor performs spatial reprojection processing on each image sub-region based on the viewpoint deviation parameter. For each pixel, it subtracts the corresponding coordinate from the current pixel coordinate. and Sampling is performed in the sub-regions of the original image. The sampling process uses a bicubic interpolation algorithm to obtain the reprojected image sub-regions. The original pixel values ​​are directly retained in the non-overlapping regions.

[0078] The graphics processor (GPU) performs weighted fusion processing within the overlapping region. The GPU calculates the vertical distance from each pixel to the seam centerline based on the pixel's position within the overlapping region in the image plane. The seam centerline is the midline connecting the boundaries of two adjacent image sub-regions. The graphics processor constructs a spatial weight distribution function. ,in This represents the maximum width of the overlapping area in the vertical seam direction, which is set to 120 pixels in this embodiment.

[0079] The graphics processor calculates the deviation weight coefficients corresponding to each image sub-region based on the viewpoint deviation parameter. For each image sub-region, the average magnitude of the viewpoint deviation parameter of all pixels within that sub-region is calculated as the deviation index E, and then... Calculate the deviation weighting coefficient, where The preset sensitivity coefficient is set to 10 in this embodiment.

[0080] The graphics processing unit (GPU) performs joint weighted calculations on pixels within the overlapping region based on the spatial weight distribution function and the bias weight distribution function to obtain an initial fused image. The GPU acquires the initial fused image sequence and calculates the pixel changes between adjacent time steps. The GPU performs temporal smoothing on the initial fused image at the current time step based on the pixel changes. The smoothing process uses an exponential moving average method, dynamically adjusting the smoothing coefficient β according to the pixel changes: β is 0.8 when the pixel change is less than 5, β is 0.3 when the pixel change is greater than 15, and β is 0.5 in other cases. The pixel values ​​are then updated to obtain the fused image.

[0081] Example 2: Based on the same inventive concept, such as Figure 2 As shown, this embodiment also provides an image data processing system based on a heterogeneous computing platform, the system comprising: The data acquisition module is used to acquire multiple extravehicular image streams output by imaging devices at different fixed installation positions, as well as the spatial installation parameters of each imaging device; and to acquire head attitude data and body attitude data.

[0082] The viewpoint construction module is used by the central processing unit to determine the target observation viewpoint based on the body posture data and head posture data. The target observation viewpoint includes the observation spatial position and observation direction in the geographic coordinate system. The module also determines the imaging spatial position and imaging direction of the corresponding imaging viewpoint in the geographic coordinate system based on the spatial installation parameters of each imaging device.

[0083] The region extraction module is used by the graphics processor to map image frames from multiple extravehicular image streams to corresponding imaging frustums in the geographic coordinate system, thereby obtaining the image field of view for each imaging viewpoint; the graphics processor extracts corresponding image sub-regions from each image field of view based on the target observation viewpoint; and the central processing unit determines the overlapping region based on the spatial overlap relationship between the image sub-regions.

[0084] The fusion processing module is used by the central processing unit to calculate the viewpoint deviation parameters of each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint; the graphics processing unit performs spatial reprojection processing on each image sub-region based on the viewpoint deviation parameters to obtain reprojected image sub-regions; and the graphics processing unit performs weighted fusion processing on each reprojected image sub-region within the overlapping region based on the viewpoint deviation parameters of each reprojected image sub-region, retaining the corresponding reprojected image sub-regions in the non-overlapping region to obtain a composite image.

[0085] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0086] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image data processing method based on a heterogeneous computing platform, characterized in that, The method includes: Acquire multiple extravehicular image streams output by imaging devices at different fixed installation locations, as well as the spatial installation parameters of each imaging device; acquire head attitude data and body attitude data; The central processing unit determines the target observation viewpoint based on the body posture data and head posture data. The target observation viewpoint includes the observation spatial position and observation direction in the geographic coordinate system. The central processing unit determines the imaging spatial position and imaging direction of the corresponding imaging viewpoint in the geographic coordinate system based on the spatial installation parameters of each imaging device. The graphics processor maps image frames from multiple extravehicular image streams to corresponding imaging frustums in the geographic coordinate system to obtain the image field of view for each imaging viewpoint; the graphics processor extracts corresponding image sub-regions from each image field of view based on the target observation viewpoint; the central processing unit determines the overlapping region based on the spatial overlap relationship between the image sub-regions; The central processing unit (CPU) calculates the viewpoint deviation parameters of each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint. The graphics processing unit (GPU) performs spatial reprojection processing on each image sub-region based on the viewpoint deviation parameters to obtain reprojected image sub-regions. Within the overlapping region, the GPU performs weighted fusion processing on each reprojected image sub-region based on the viewpoint deviation parameters of each reprojected image sub-region, retaining the corresponding reprojected image sub-regions in the non-overlapping region to obtain a composite image.

2. The image data processing method based on a heterogeneous computing platform according to claim 1, characterized in that, The method for determining the target observation point based on body posture data and head posture data includes: Construct head posture data sequences and body posture data sequences based on head posture data and body posture data, where each head posture data and each body posture data has a corresponding timestamp. Determine a unified target time based on the timestamp; calculate the attitude change between adjacent times based on the head attitude data sequence and the body attitude data sequence; perform time interpolation calculation on the head attitude data and body attitude data based on the attitude change to obtain aligned attitude data corresponding to the unified target time. The observation spatial position and observation direction are determined based on the alignment posture data; the target observation viewpoint is determined based on the observation spatial position and observation direction.

3. The image data processing method based on a heterogeneous computing platform according to claim 2, characterized in that, The method for determining the observation spatial position and observation direction based on the alignment posture data includes: Calculate the head posture change rate based on the head posture data sequence; calculate the body posture change rate based on the body posture data sequence; obtain the time compensation amount from the posture data acquisition time to the image display time. Based on the head posture change rate, body posture change rate, and time compensation, the alignment posture data is extrapolated along the time axis to obtain the predicted posture data; the observation spatial position and observation direction are determined based on the predicted posture data.

4. The image data processing method based on a heterogeneous computing platform according to claim 3, characterized in that, The method for extracting corresponding image sub-regions from each image field of view based on the target viewing point includes: The target observation field of view is determined according to the observation direction of the target observation viewpoint; the target observation field of view is mapped to each image field of view to obtain the target projection area; and overlapping candidate areas are determined within the target projection area according to the spatial overlap relationship between the image fields of view. Based on the spatial distance between the overlapping candidate regions and the boundaries of each image field of view, the regions in the overlapping candidate regions whose spatial distance from the boundaries of the image field of view is less than a preset spatial distance threshold are determined as the initial seam adjacent regions. The target projection region is divided according to the initial seam adjacent region to obtain the seam adjacent region and the non-seam adjacent region; the image sub-region includes the seam adjacent region and the non-seam adjacent region.

5. The image data processing method based on a heterogeneous computing platform according to claim 4, characterized in that, The method for dividing the target projection region according to the initial seam adjacent region to obtain the seam adjacent region and the non-seam adjacent region includes: The initial seam adjacent region position sequence is determined based on the initial seam adjacent region at consecutive time points; the change in region position between adjacent time points is calculated based on the initial seam adjacent region position sequence; Based on the change in the region's location, the initial seam adjacent region at the current moment is smoothly adjusted to obtain the seam adjacent region; the region in the target projection region other than the seam adjacent region is taken as the non-seam adjacent region.

6. The image data processing method based on a heterogeneous computing platform according to claim 5, characterized in that, The method for calculating the viewpoint deviation parameter of each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint includes: Based on the spatial location of the image sub-region in the geographic coordinate system, and combined with the imaging spatial location, imaging direction, observation spatial location and observation direction, the imaging line-of-sight vector and the observation line-of-sight vector corresponding to the image sub-region are constructed respectively. Based on the angle between the imaging line-of-sight vector and the observation line-of-sight vector, and the scene distance information corresponding to the image sub-region, the pixel offset of the image sub-region is calculated as the basic viewpoint deviation. The image sub-region is divided into multiple distance layers based on the scene distance information; the basic viewpoint deviation is weighted and normalized within each distance layer to obtain the layered deviation corresponding to each distance layer; the image sub-region is subjected to deviation mapping processing based on the layered deviation to obtain the viewpoint deviation parameter of the image sub-region.

7. The image data processing method based on a heterogeneous computing platform according to claim 6, characterized in that, The method for weighted normalizing the basic viewpoint deviation within each distance layer to obtain the layer deviation corresponding to each distance layer includes: The relative positional distance between the image sub-region and the center position of the seam adjacent region is determined based on the position of the image sub-region in the overlapping region. Within the same distance layer, the image sub-region is divided into a seam center region and a seam edge region based on the relative positional distance; the local deviation adjustment coefficients corresponding to the seam center region and the seam edge region are calculated respectively; Based on the local deviation adjustment coefficient, the basic viewpoint deviation corresponding to the center region of the joint is subjected to high-precision deviation constraint processing, and the basic viewpoint deviation corresponding to the edge region of the joint is subjected to low-precision deviation constraint processing, so as to obtain the layered deviation amount after joint sensitivity refinement.

8. The image data processing method based on a heterogeneous computing platform according to claim 7, characterized in that, The method for weighted fusion processing of each reprojected image sub-region within the overlapping region based on the viewpoint deviation parameter of each reprojected image sub-region includes: The corresponding pixel position distribution is determined based on the spatial position of the reprojected image sub-region in the overlapping region; a spatial weight distribution function is constructed based on the pixel position distribution; a deviation weight coefficient is calculated based on the viewpoint deviation parameter corresponding to each reprojected image sub-region; and a deviation weight distribution function is constructed based on the deviation weight coefficient. Based on the spatial weight distribution function and the bias weight distribution function, the pixels in the overlapping region of the reprojected image sub-region are jointly weighted to obtain an initial fused image; an initial fused image sequence is obtained; the pixel change between adjacent time steps is calculated based on the initial fused image sequence; and the initial fused image at the current time step is temporally smoothed based on the pixel change to obtain a fused image.

9. An image data processing system based on a heterogeneous computing platform, characterized in that, The system includes: The data acquisition module is used to acquire multiple extravehicular image streams output by imaging devices at different fixed installation positions, as well as the spatial installation parameters of each imaging device; and to acquire head attitude data and body attitude data. The viewpoint construction module is used by the central processing unit to determine the target observation viewpoint based on the body posture data and head posture data. The target observation viewpoint includes the observation spatial position and observation direction in the geographic coordinate system. The module also determines the imaging spatial position and imaging direction of the corresponding imaging viewpoint in the geographic coordinate system based on the spatial installation parameters of each imaging device. The region extraction module is used by the graphics processor to map image frames from multiple extravehicular image streams to corresponding imaging frustums in the geographic coordinate system, thereby obtaining the image field of view for each imaging viewpoint; the graphics processor extracts corresponding image sub-regions from each image field of view based on the target observation viewpoint; and the central processing unit determines the overlapping region based on the spatial overlap relationship between the image sub-regions. The fusion processing module is used by the central processing unit to calculate the viewpoint deviation parameters of each image sub-region within the overlapping region based on the spatial difference between the imaging spatial position and imaging direction of the imaging viewpoint and the observation spatial position and observation direction of the target observation viewpoint; the graphics processing unit performs spatial reprojection processing on each image sub-region based on the viewpoint deviation parameters to obtain reprojected image sub-regions; and the graphics processing unit performs weighted fusion processing on each reprojected image sub-region within the overlapping region based on the viewpoint deviation parameters of each reprojected image sub-region, retaining the corresponding reprojected image sub-regions in the non-overlapping region to obtain a composite image.

Images