On-orbit video encoding method and system based on heterogeneous computing platform for remote sensing satellite

Abstract

The application provides a kind of remote sensing satellite in-orbit video encoding method and system based on heterogeneous computing platform, belongs to remote sensing satellite technical field, including: based on virtual node dynamic mapping algorithm and consistent hash dynamic allocation task algorithm, dynamically adjust the on-board computing resources of remote sensing satellite;The elastic allocation is carried out to on-board computing resources, and the parallel processing of the elastic allocation on-board computing resources is carried out using multi-unit stacking pipeline parallel algorithm, the directional model based on satellite video image and the geometric correction model between video image frames are constructed, and the optical satellite video image stabilization with geographic coding is obtained;The optical satellite video stabilization is compressed and encoded to shorten the input-output time consumption, hide the time delay of input-output time consumption, and output video stream.The application changes the traditional remote sensing satellite video encoding task needing ground processing into on-board in-orbit processing mode, significantly reduces the time consumption of remote sensing video encoding, and provides remote sensing satellite "fast, accurate and flexible" service.

Description

Technical Field

[0001] This invention relates to the field of remote sensing satellite technology, and in particular to an on-orbit video coding method and system for remote sensing satellites based on a heterogeneous computing platform. Background Technology

[0002] Remote sensing satellites can capture images of the Earth from space, but traditional remote sensing satellites only take pictures and need to transmit all the data back to the ground for further processing. With the increasing demands for timeliness in geographic information acquisition and application, developing on-orbit processing technology for remote sensing satellites, expanding from "seeing from space" to "computing from space," has become an inevitable trend in the development of remote sensing technology.

[0003] Currently, the imaging capabilities of remote sensing satellites are constantly improving, evolving from capturing images to capturing videos, with increasingly higher resolutions and larger data volumes, greatly increasing the difficulty of on-orbit processing. Compared with ground processing, the main challenges of on-orbit processing are: (1) High reliability: Due to the influence of the space radiation environment, ground computing units are subject to interference in space, resulting in decreased stability. Therefore, it is necessary to design a heterogeneous architecture with multiple computing units to ensure high reliability in the space environment. (2) High data throughput: For remote sensing satellites, the amount of data generated per second by the camera can be hundreds of MB or even GB. If the data throughput of the on-board computing units cannot keep up with the data generation rate, it will lead to the accumulation of data to be processed, causing delays in on-board data processing. (3) High computational efficiency: It is necessary to improve the utilization rate of computing resources and increase the parallelism of algorithm operation to ensure that complex algorithms can run efficiently in the on-board environment.

[0004] Video encoding is the process of converting raw video data acquired by satellites into a binary stream that is easy to transmit to the ground. For remote sensing video satellites, this is a necessary preprocessing step that must be completed onboard before data can be transmitted to the ground. Therefore, the efficiency of satellite video encoding has a significant impact on the overall response rate of the satellite. Summary of the Invention

[0005] This invention provides an on-orbit video coding method and system for remote sensing satellites based on a heterogeneous computing platform, which solves the problems of high latency, high throughput and low computational efficiency caused by the fact that video coding for remote sensing satellites usually uses ground data processing.

[0006] In a first aspect, the present invention provides an on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform, comprising:

[0007] A flexible computing heterogeneous architecture is determined to be constructed using CPU, GPU, and FPGA. In this architecture, the on-board computing resources of the remote sensing satellite are dynamically adjusted based on a virtual node dynamic mapping algorithm and a consistent hashing dynamic task allocation algorithm.

[0008] The on-board computing resources are flexibly allocated, and a multi-unit stacked pipeline parallel algorithm is used to process the flexibly allocated on-board computing resources in parallel. A directional model based on satellite video images and a geometric correction model between video image frames are constructed to obtain a geocoded optical satellite video image.

[0009] The elastic computing heterogeneous architecture is used to compress and encode the optical satellite video image stabilization to shorten the input and output time. The latency of input and output time is hidden by streaming processing and inter-frame parallel processing, and the video stream is output.

[0010] According to the present invention, an on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform dynamically adjusts the on-board computing resources of the remote sensing satellite in the elastic computing heterogeneous architecture based on a virtual node dynamic mapping algorithm and a consistent hashing dynamic task allocation algorithm, including:

[0011] In the elastic computing heterogeneous architecture, a number of virtual nodes and a number of physical nodes are determined;

[0012] If it is determined that all physical nodes are in an available state, then based on the total number of physical nodes, the virtual node is mapped to any physical node by taking the remainder of the sequence number of any virtual node.

[0013] If any physical node is determined to be unavailable, then based on the number of available physical nodes, any virtual node is mapped to any available physical node;

[0014] The consistent hashing algorithm is used to calculate the hash value of each virtual node, and the hash value of each virtual node is mapped to a virtual ring of a preset length.

[0015] The input data of the elastic computing heterogeneous architecture is obtained, the input data is divided, the hash value of each partition is calculated using a consistent hashing algorithm, and the hash value of each partition is mapped to the virtual ring of the preset length.

[0016] On the virtual ring of the preset length, determine the virtual node closest to each partition data in a clockwise direction, and assign each partition data and its corresponding computing task to the nearest virtual node.

[0017] According to the present invention, a remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform is provided, which flexibly allocates the on-board computing resources and uses a multi-unit stacked pipeline parallel algorithm to process the flexibly allocated on-board computing resources in parallel, including:

[0018] Determine the total number of input data segments l, the number of processing units k, and the input time t0, processing time t1, and output time t2 for each segment of data.

[0019] Calculate the processing time t in the parallel mode of the three-step stacked pipeline. p for:

[0020]

[0021] Among them, t max =max(t0, t1, t2), i = 0, 1, and 2;

[0022] As the total number of segments l → ∞, the speedup of the three-step stacked pipeline in parallel is:

[0023]

[0024] According to the present invention, a remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform is provided, which constructs a orientation model based on satellite video images and a geometric correction model between video image frames to obtain a geocoded stable optical satellite video image, including:

[0025] Determine any two consecutive video frames as the main frame and the auxiliary frame to form multiple pairs of main and auxiliary frame images;

[0026] Match multiple video image frames to obtain the matching points for each pair of main and auxiliary frame images;

[0027] An orientation model for optical satellite video images and an inter-frame geometric correction model for auxiliary frame images were established respectively.

[0028] Calculate the inter-frame geometric correction parameters for the supplementary frame image;

[0029] Inter-frame geometric correction is performed on the orientation parameters of the auxiliary frame image to obtain the corrected orientation parameters;

[0030] Geocoding is performed on the video images to obtain geocoded stable optical satellite video images.

[0031] According to the present invention, a remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform performs inter-frame geometric correction on the orientation parameters of auxiliary frame images to obtain corrected orientation parameters, including:

[0032] Divide the auxiliary frame image into multiple regular grids and set multiple elevation reference planes;

[0033] Based on the orientation model of the auxiliary frame image, each regular grid point on the auxiliary frame image is projected onto the elevation datum plane to generate virtual control points;

[0034] Inter-frame geometric correction is performed on the image-side coordinates of the virtual control points to obtain the corrected image-side coordinates.

[0035] Using the virtual control points after inter-frame geometric correction, the corrected orientation parameters of the auxiliary frame image are calculated.

[0036] According to the present invention, a remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform is provided, which performs geocoding on video images to obtain geocoded stable optical satellite video images, including:

[0037] Based on the orientation model of the video image, the four corner points of the first frame, middle frame and last frame of the video image are projected onto the digital elevation model to obtain the geographical range of the first frame, middle frame and last frame.

[0038] The geographical extent of the stabilized optical satellite video image is obtained by extracting the intersection of the geographical extent of the first frame, the middle frame, and the last frame.

[0039] Based on the geographic coordinates of the top left corner of the stabilized optical satellite video image, the image size, and the spatial resolution, calculate the geographic coordinates of each ground point within the geographic range of the stabilized optical satellite video image.

[0040] Optical satellite video stabilization is generated sequentially for each frame of video image.

[0041] According to the present invention, a remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform is provided, which utilizes the elastic computing heterogeneous architecture to compress and encode the stabilized video image of the optical satellite, including:

[0042] The video compression coding process includes image segmentation, motion compensation, and intra-frame prediction sub-steps.

[0043] Each image segmentation sub-step, each motion compensation sub-step, and each intra-frame prediction sub-step includes a first stage in which the CPU executes a preset low-computation algorithm and control logic to prepare for GPU-FPGA execution; a second stage in which the GPU-FPGA executes a preset large-scale parallel processing task while the CPU waits; and a third stage in which the CPU executes a preset low-computation algorithm and control logic and processes the GPU-FPGA calculation results.

[0044] The time taken in the first stage is T0, the time taken in the second stage is T1+T2, and the time taken in the third stage is T3. T0 and T3 are the CPU time, T1 is the GPU time, and T2 is the FPGA time. The CPU load in the three stages is l0, l1, and l2, respectively. The parallel processing unit is idle in the first and third stages, and the full load in the second stage is 1.

[0045] Given that the number of concurrent tasks is n, the total time for the second phase of all tasks is nT1 + nT2, and the total time t for all n tasks is... total for:

[0046] t total =nl0T0+nT1+nT2+nl2T3

[0047] Total CPU load total for:

[0048]

[0049] Calculate the total CPU time t used by the second-stage algorithm. cpu Combined with GPU-FPGA parallel computing time t par =T1+T2, where ρ is the proportion of computational workload allocated to the CPU, the total computational workload is M, and the processing performance when the second stage utilizes all CPU cores is P. cpu When using GPU-FPGA parallel computing units, the processing performance is P. par :

[0050]

[0051] Further analysis revealed:

[0052]

[0053] The second-stage time-consuming algorithm is modified using the obtained ρ. The modified time consumption t′ par for:

[0054]

[0055] According to the present invention, a remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform hides the input / output latency through streaming processing operations and inter-frame parallel processing, and outputs a video stream, including:

[0056] Multiple macroblocks are divided using pre-defined rectangular regions of different sizes;

[0057] Scan the multiple macroblocks and divide the image frames with a chromaticity difference of less than a preset percentage in multiple adjacent image frames into the same group of image frames;

[0058] Two frames of video data are extracted sequentially from the head of the buffer. Macroblock scanning is performed on the two frames of video data. If it is determined that an object exists in either frame of video data, a window search is performed in the neighborhood of the other frame of video. Once the object is found, the motion vector of the object is calculated.

[0059] For each macroblock, perform pattern prediction, find the prediction pattern that is closest to the original image, and subtract the original image from the intra-frame predicted image to obtain residual data.

[0060] The residual data is subjected to integer discrete cosine transform to remove data correlation;

[0061] By using a statistical frequency-assigned variable-length encoding table, the residual data after removing data correlation is converted into binary code to obtain the video stream.

[0062] Secondly, the present invention also provides an on-orbit video coding system for remote sensing satellites based on a heterogeneous computing platform, comprising:

[0063] The elastic computing module is used to determine the elastic computing heterogeneous architecture constructed by CPU, GPU and FPGA, and dynamically adjust the on-board computing resources of the remote sensing satellite based on the virtual node dynamic mapping algorithm and the consistent hashing dynamic task allocation algorithm in the elastic computing heterogeneous architecture.

[0064] The image stabilization module is used to flexibly allocate the on-board computing resources, and to use a multi-unit stacked pipeline parallel algorithm to process the flexibly allocated on-board computing resources in parallel. It constructs a directional model based on satellite video images and a geometric correction model between video image frames to obtain a geocoded optical satellite video image.

[0065] The compression encoding processing module is used to compress and encode the optical satellite video image stabilization using the elastic computing heterogeneous architecture to shorten the input and output time. It hides the input and output time delay through streaming processing operations and inter-frame parallel processing, and outputs a video stream.

[0066] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the on-orbit video coding method for remote sensing satellites based on heterogeneous computing platforms as described above.

[0067] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform as described above.

[0068] The remote sensing satellite on-orbit video coding method and system based on a heterogeneous computing platform provided by this invention transforms the traditional remote sensing satellite video coding task that requires ground processing into an on-orbit processing mode. It makes full use of the heterogeneous computing units of on-board CPU+GPU+FPGA, significantly reducing the time consumption of remote sensing video coding and providing remote sensing satellite with "fast, accurate and flexible" services. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating the on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform provided by the present invention.

[0071] Figure 2 This is a flowchart of the virtual node dynamic mapping algorithm provided by the present invention;

[0072] Figure 3 This is a schematic diagram of the dynamic allocation process of computing tasks provided by the present invention;

[0073] Figure 4 This is a schematic diagram of the parallel principle of a multi-unit three-step stacked pipeline provided by the present invention;

[0074] Figure 5 This is a schematic diagram of the structure of the remote sensing satellite on-orbit video coding system based on a heterogeneous computing platform provided by the present invention;

[0075] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0077] Figure 1 This is a flowchart illustrating the on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform provided in this embodiment of the invention. Figure 1 As shown, it includes:

[0078] Step 100: Determine the elastic computing heterogeneous architecture constructed by CPU, GPU and FPGA, and dynamically adjust the on-board computing resources of the remote sensing satellite based on the virtual node dynamic mapping algorithm and the consistent hashing dynamic task allocation algorithm in the elastic computing heterogeneous architecture.

[0079] Step 200: Elastically allocate the on-board computing resources, and use a multi-unit stacked pipeline parallel algorithm to process the elastically allocated on-board computing resources in parallel. Construct a directional model based on satellite video images and a geometric correction model between video image frames to obtain a geocoded optical satellite video image.

[0080] Step 300: Use the elastic computing heterogeneous architecture to compress and encode the optical satellite video image stabilization to shorten the input and output time. Hide the input and output time delay through streaming processing and inter-frame parallel processing, and output video stream.

[0081] To achieve on-board video processing and improve processing speed, this invention employs an on-orbit video coding system for remote sensing satellites based on heterogeneous computing units. This system includes a flexible computing architecture based on heterogeneous computing units, a streaming I / O video stabilization system, and a heterogeneous parallel video compression coding system. The heterogeneous computing units consist of a CPU, a GPU, and an FPGA. The CPU and GPU perform image stabilization for each frame, while the FPGA performs video compression coding.

[0082] Specifically, the elastic computing architecture based on heterogeneous computing units is mounted on the on-board intelligent computing unit. The chip system composed of CPU and GPU works with FPGA to perform computing tasks. It consists of a virtual node dynamic mapping algorithm that balances flexibility and stability and an algorithm that dynamically allocates computing tasks based on consistent hashing. The computing resources are dynamically adjusted according to the failure status of physical nodes to ensure that load changes do not affect the original computing resources.

[0083] The streaming I / O video stabilization system flexibly allocates on-board computing resources and processes data in parallel. For high bitrate videos, it uses a pipelined parallel approach for data input and output to reduce processing time and fully utilize heterogeneous computing units. During video stabilization, a chip system composed of CPU and GPU matches each pair of main and auxiliary frames in the video image, constructs a model to correct the inter-frame geometric transformations, and calculates the inter-frame geometric correction parameters for the auxiliary frame image. The corrected orientation parameters and model are obtained by generating virtual control points on the auxiliary frame image. Finally, the matching points are projected based on the orientation model on the main frame image, and the auxiliary frame image is geocoded to produce a geocoded stabilized image.

[0084] The heterogeneous parallel video compression coding system, based on stabilized image sequences, leverages the hardware parallelism of FPGAs to perform heterogeneous parallel video compression coding between a chip system (including CPU and GPU) and an FPGA. In this coding process, since the stabilization result is streamed, the compression coding is also streamed. Because the computation of each image frame is independent, it can be divided into several independent sub-loads, achieving high parallelism among heterogeneous computing units. I / O latency is hidden through FPGA streaming operations and inter-frame parallel processing techniques. After image segmentation, motion compensation, and intra-frame prediction, the video stream is output. Ultimately, this achieves two layers of heterogeneous parallelism: CPU+GPU video stabilization and GPU+FPGA compression coding, further improving the on-orbit processing efficiency of video satellites on top of a conventional architecture.

[0085] In summary, this invention employs a highly reliable heterogeneous elastic computing architecture, which possesses a certain degree of flexibility and can maintain system flexibility and stability in the harsh environment of space, making full use of onboard computing resources. The streaming I / O processing architecture ensures that the onboard video data acquisition and processing rates are matched, preventing data backlog. Finally, through heterogeneous unit collaboration and the design of a parallel computing mechanism, high-efficiency onboard video coding is achieved based on geometric registration of image stabilization in sequence images.

[0086] Based on the above embodiments, in the elastic computing heterogeneous architecture, the on-board computing resources of the remote sensing satellite are dynamically adjusted based on the virtual node dynamic mapping algorithm and the consistent hashing dynamic task allocation algorithm, including:

[0087] In the elastic computing heterogeneous architecture, a number of virtual nodes and a number of physical nodes are determined;

[0088] If it is determined that all physical nodes are in an available state, then based on the total number of physical nodes, the virtual node is mapped to any physical node by taking the remainder of the sequence number of any virtual node.

[0089] If any physical node is determined to be unavailable, then based on the number of available physical nodes, any virtual node is mapped to any available physical node;

[0090] The consistent hashing algorithm is used to calculate the hash value of each virtual node, and the hash value of each virtual node is mapped to a virtual ring of a preset length.

[0091] The input data of the elastic computing heterogeneous architecture is obtained, the input data is divided, the hash value of each partition is calculated using a consistent hashing algorithm, and the hash value of each partition is mapped to the virtual ring of the preset length.

[0092] On the virtual ring of the preset length, determine the virtual node closest to each partition data in a clockwise direction, and assign each partition data and its corresponding computing task to the nearest virtual node.

[0093] Specifically, the elastic computing architecture based on heterogeneous computing units in this embodiment of the invention extends the on-orbit processing capabilities of traditional remote sensing satellite platforms, enabling dynamic allocation of on-board computing resources with both flexibility and stability. Traditional remote sensing satellite platforms only capture images, transmit the data to the ground, and then perform geometric and radiometric corrections, information extraction, etc. In contrast, the satellite itself in this system needs to possess on-orbit data processing and intelligent analysis capabilities, enabling it to transform raw image data into the information required by users with shorter links and faster speeds, enhancing the timeliness and intelligence of remote sensing information services. The elastic computing architecture based on heterogeneous computing units is mounted on a heterogeneous computing unit consisting of onboard CPU+GPU+FPGA, fully utilizing onboard processing resources and dynamically adjusting computing resource usage based on the failure status of physical nodes, ensuring that load changes do not affect existing computing resources. This architecture consists of a virtual node dynamic mapping algorithm that balances flexibility and stability, and an algorithm for dynamically allocating computing tasks based on consistent hashing.

[0094] Firstly, a dynamic mapping of virtual nodes, combining flexibility and stability, is employed. Due to the harsh space environment, factors such as temperature, power consumption, and radiation can cause some computing resources to fail during onboard processing; therefore, the onboard processing architecture must possess a certain degree of resilience. Considering the different uses of future onboard processing systems, which require varying numbers of onboard processing units, a streaming architecture is constructed using onboard computing resource virtualization to effectively manage and utilize these resources.

[0095] For on-board computing resources, let's assume there are m virtual nodes and n physical nodes. For the u-th virtual node, we map the virtual node to a physical node using the remainder of its node index. In the ideal state where all physical nodes are available, the task on the u-th virtual node actually runs on the v-th physical node:

[0096] v = u mod n, (0 ≤ u < m)

[0097] When the v-th physical node is unavailable, for The corresponding physical node number needs to be recalculated. To avoid interfering with other running virtual node tasks, the value of n in the above formula remains unchanged; only the virtual nodes corresponding to the failed physical node are remapped. Let the number of available physical nodes be n′, then:

[0098]

[0099] At this point, the virtual node can be mapped to the v′th available node.

[0100] Figure 2 The flow of the virtual node mapping algorithm is demonstrated, and it is clear that this method has the following key characteristics, which enable it to meet the requirements for building an on-board elastic computing processing architecture:

[0101] (1) As long as there is at least one available physical node, all virtual node tasks can be assigned to these available physical nodes for execution.

[0102] (2) The failure of any physical node will not interfere with the tasks on other physical nodes;

[0103] (3) Tasks originally assigned to faulty physical nodes can be effectively redistributed to available physical nodes for execution;

[0104] (4) When the faulty physical node recovers (e.g., restarts successfully), it can continue to execute the tasks previously assigned to that node without interfering with the tasks on other physical nodes;

[0105] (5) Although the failure of physical nodes will affect the system performance, it will not affect the overall function of the system.

[0106] Then, tasks are dynamically allocated. The virtual resource mapping strategy effectively isolates the impact of physical node states on the overall system functionality. Based on this, further research is needed on the allocation strategy of computing tasks to virtual nodes. To make the overall processing tasks more uniform at a finer granularity, a certain degree of randomness needs to be introduced in the allocation process of computing tasks to virtual nodes.

[0107] This invention uses a consistent hashing algorithm to calculate the hash value of each virtual node and maps it to 0 to 2. 32 On the virtual ring of -1; secondly, apply the same hash algorithm to the partitioned input data to obtain the hash value of each data and map it to the same virtual ring; finally, for each data, find the nearest virtual node clockwise along the ring and assign the data and its computation task to that node. Figure 3 The process was demonstrated: after hashing, the six virtual nodes were randomly distributed on the ring; at the same time, after hashing, the nine data items could also be located on the ring and the nearest virtual node could be found clockwise along the ring.

[0108] Understandably, since physical computing resources have already been virtually mapped, there's no need to consider node validity and quantity variations when allocating computing tasks. The number of virtual nodes, v, can be considered a configurable constant, and the number of virtual nodes on the ring can be considered fixed. Clearly, the finer the granularity of data partitioning, the more evenly data blocks are distributed on the ring, resulting in better overall system load balancing.

[0109] Building upon the above embodiments, this invention employs a streaming I / O video stabilization system, which expands upon existing optical satellite video stabilization technologies in terms of processing speed and video quality. This system utilizes a pipelined parallel approach for data input and output, fully leveraging heterogeneous computing units to significantly reduce on-board processing time. The video stabilization method effectively reduces inter-frame geometric distortion caused by lens distortion and terrain undulations, achieving sub-pixel-level video stabilization. Furthermore, geocoding helps users obtain geometric information about targets of interest. The system consists of a streaming I / O processing architecture and an optical satellite video stabilization method with geocoding.

[0110] In a streaming I / O processing architecture, for remote sensing image processing algorithms, the object of processing is remote sensing images, which inherently involve a large amount of data. Compared to the algorithm execution time, the time cost of data input and output cannot be ignored. Furthermore, in a computer system, data input and output are accomplished through communication interfaces or external storage read / write operations, while algorithm processing is performed by the processing core and memory—these are separate components. Therefore, the plan is to use a pipelined parallel model to build processing pipelines within and between physical computing nodes, allowing the three steps of data input, algorithm processing, and data output to be stacked, thus hiding the time consumption.

[0111] Let the input, processing, and output times for each data segment be t0, t1, and t2, respectively. There are l data segments and k processing units. Then, in a typical multi-unit pipelined parallel mode, the processing time t... s for:

[0112]

[0113] In the parallel mode of the three-step stacked pipeline, the processing time is t. p for:

[0114]

[0115] Among them, t max =max(t0, t1, t2)

[0116] When the data segment is divided sufficiently finely, l→∞. At this point, with the same input data and the same number of processing units, the speedup of the three-step stacked pipeline parallelism is:

[0117]

[0118] From this formula, we can see that:

[0119] Smaller data segments can improve the efficiency of parallel pipelines;

[0120] Compared to common multi-unit pipelined parallelism, the three-step stacked pipelined parallelism is consistently more efficient when the amount of data to be processed and the number of hardware units are the same. It can improve processing performance by up to three times when the time for data input, algorithm processing, and data output is the same. Figure 4 As shown, the overall time taken to go from data 0 to data 7 is reduced to 2*t0+4*t1+t2 after the three-step stacked pipeline parallel acceleration.

[0121] Building upon the above embodiments, this invention addresses the shortcomings of existing technologies in optical satellite video stabilization, resolving the significant problems of geometric distortion caused by lens distortion and terrain undulations, and the difficulty in obtaining geometric information of targets of interest from video data obtained using traditional video stabilization models. The main approach is based on a orientation model of satellite video images. By establishing a geometric correction model between video image frames and geocoding the video images, geocoded optical satellite video stabilization can be achieved. The specific steps are as follows:

[0122] Based on the high-frequency sequence imaging characteristics of optical video satellites, two consecutive video images are defined as the main frame and the auxiliary frame, thus forming multiple pairs of main and auxiliary frame images.

[0123] Perform a matching operation between video image frames to obtain the matching points between each pair of main and auxiliary frame images;

[0124] Establish a geometric correction model between video image frames, specifically including establishing an orientation model for optical satellite video images and establishing an inter-frame geometric correction model for auxiliary frame images.

[0125] Calculate the inter-frame geometric correction parameters for the supplementary frame image;

[0126] Inter-frame geometric correction is performed on the orientation parameters of the supplementary frame image to obtain the corrected orientation parameters. This step includes:

[0127] Regular grids are divided on the auxiliary frame image, and multiple elevation reference surfaces are set within the coverage area of ​​the auxiliary frame image. Based on the orientation model of the auxiliary frame image, each regular grid point on the auxiliary frame image is projected onto the elevation reference surface to generate virtual control points.

[0128] Inter-frame geometric correction is performed on the image-side coordinates of the virtual control points to obtain the corrected image-side coordinates;

[0129] Using virtual control points after inter-frame geometric correction, the orientation parameters of the auxiliary frame image are calculated;

[0130] Geocoding the video images to obtain geocoded, stable images includes the following steps:

[0131] Based on the orientation model of the video images, the four corner points of the first, middle, and last frames are projected onto the digital elevation model to obtain the geographical extent of the first, middle, and last frames. Then, the intersection of the geographical extents of these three frames is taken as the geographical extent of the stabilized image.

[0132] Based on the geographic coordinates of the top left corner of the stabilized image, the image size, and the spatial resolution, calculate the geographic coordinates of each ground point within the geographic range of the stabilized image.

[0133] For each frame of video image, generate the corresponding stabilized image.

[0134] Based on the above embodiments, the heterogeneous parallel video compression coding system in this embodiment of the invention utilizes the hardware parallelism of FPGA to perform heterogeneous parallel video compression coding of chip system (including CPU and GPU)-FPGA, based on the stabilized image sequence. On-board processing includes two main categories of computing components: core units (CPU) suitable for flow control and serial computing, and parallel units (GPU and FPGA) suitable for parallel computing. To fully utilize hardware performance, it is necessary to combine algorithms to implement parallel collaboration of heterogeneous processing units.

[0135] During video compression encoding, since the GPU streams and outputs the stabilized image results in the previous step, the FPGA compression encoding is also streamed, meaning the parallel processing of the GPU and FPGA is performed serially. Because the computations of the image frames to be processed are independent, they can be divided into several independent sub-loads, achieving high parallelism between heterogeneous computing units. At this point, relative to the entire CPU+GPU+FPGA onboard computing system, the GPU and FPGA can be logically merged into a single parallel processing unit. The CPU is responsible for algorithm coordination and scheduling, while the parallel processing unit is responsible for performing a large number of parallel computations.

[0136] Video compression coding can be divided into several sub-steps, such as image segmentation, motion compensation, and intra-frame prediction. For chip system (including CPU and GPU)-FPGA collaborative programs, each sub-step can be divided into three stages: In the first stage, the CPU executes algorithms and control logic with low computational load and prepares for GPU-FPGA execution; in the second stage, the GPU-FPGA performs large-scale parallel processing while the CPU waits; in the third stage, the CPU executes algorithms and control logic with low computational load and processes the GPU-FPGA calculation results. Typically, stage two accounts for the majority of the algorithm's runtime. If only one algorithm is using computational resources, the CPU is almost idle at this stage.

[0137] Let the time consumption of a single task in the above three processing stages be T0, (T1+T2), and T3, respectively, where T0 and T3 are the CPU time consumption, T1 is the GPU time consumption, and T2 is the FPGA time consumption. The CPU loads are l0, l1, and l2, respectively. The parallel processing unit is idle in the first and third stages, and its full load in the second stage is 1. Let the number of concurrent tasks be n. When a subtask of compression encoding is in stage one or stage three, the GPU-FPGA parallel computing resources are always occupied by other concurrent tasks. Therefore, it can be considered that the parallel computing is always at full load, and the total time consumption of all tasks in stage two is nT1+nT2. Therefore, the total time consumption t for n tasks is... total It can be calculated using the following formula:

[0138] t total =nl0T0+nT1+nT2+nl2T3

[0139] Total CPU load total It can be calculated using the following formula:

[0140]

[0141] Thus, 1-l total This refers to the proportion of idle computing power that the CPU can use to assist in parallel processing of time-consuming tasks in Phase Two. By controlling the CPU resources used for assisting parallel processing to not exceed this proportion, the execution time of each task in Phase Two can be shortened without affecting the processing of other concurrent tasks, thus reducing the overall processing time.

[0142] Based on this, the runtime t of the Phase 2 time-consuming algorithm when it occupies all CPU cores was further measured. cpu Combined with GPU-FPGA parallel computing time t par (t par=T1+T2), which can be used to determine the optimal computational allocation ratio for CPU / GPU collaborative modification of time-consuming algorithms under high-concurrency processing conditions on the satellite. Let ρ be the proportion of computational workload allocated to the CPU, the total computational workload be M, and the processing performance of the Phase 2 algorithm when it occupies all CPU cores be P. cpu When using GPU-FPGA parallel computing units, the processing performance is P. par Then we have:

[0143]

[0144] After sorting, we can obtain:

[0145]

[0146] Analysis of the above formula shows that, among the values ​​needed to calculate ρ, t par , t cpu Both are easily obtained through actual measurement; while l total Since many factors are involved, variables l0, l1, and l2 are difficult to measure accurately. In practice, the proportion of the time spent in stage two to the total time spent in the three stages can be used to estimate the time indirectly.

[0147] The time consumption algorithm for stage two is modified using ρ obtained from the above formula. The modified time consumption t′ par for:

[0148]

[0149] This enables parallel processing of heterogeneous processing units under high concurrency, further reducing processing time.

[0150] By employing CPU+PGU+FPGA streaming operations and inter-frame parallel processing techniques to hide I / O latency, and performing operations such as image segmentation, motion compensation, and intra-frame prediction, the output video stream is completed. Specifically, this includes:

[0151] 1) Divide into macroblocks. By default, a 16x16 region is used as a macroblock.

[0152] 2) Frame grouping. Macroblock scanning and macroblock search can reveal the correlation between frames, thus grouping multiple frames into a group. In several adjacent image frames, those with differences typically accounting for only 10% or less, with brightness differences not exceeding 2% and chromaticity differences only 1%, can be grouped together.

[0153] 3) Motion estimation and compensation. First, two frames of video data are retrieved sequentially from the head of the buffer, and then a macroblock scan is performed. When an object is found in one of the images, a window search is performed in the neighborhood of the other image. If the object is found in the other image, its motion vector can be calculated.

[0154] 4) Intra-frame prediction. Perform pattern prediction for each macroblock and find the prediction pattern that is closest to the original image. Then, subtract the original image from the intra-frame predicted image to obtain the residual value and record it.

[0155] 5) Discrete Cosine Transform. Perform an integer discrete cosine transform on the residual data to remove data correlation.

[0156] 6) Entropy coding. A variable-length coding table is allocated using statistical frequency to losslessly convert data into binary code.

[0157] After the above processing, the present invention ultimately achieves two-layer heterogeneous parallel processing of CPU+GPU video stabilization and GPU+FPGA compression encoding, further improving the on-orbit processing efficiency of video satellites on the basis of conventional architecture.

[0158] The following describes the on-orbit video coding system for remote sensing satellites based on a heterogeneous computing platform provided by this invention. The on-orbit video coding system for remote sensing satellites based on a heterogeneous computing platform described below can be referred to in correspondence with the on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform described above.

[0159] Figure 5 This is a schematic diagram of the structure of the on-orbit video coding system for remote sensing satellites based on a heterogeneous computing platform provided in an embodiment of the present invention, as shown below. Figure 5 As shown, it includes: an elastic calculation module 51, an image stabilization processing module 52, and a compression encoding processing module 53, wherein:

[0160] The elastic computing module 51 is used to determine the elastic computing heterogeneous architecture constructed by CPU, GPU and FPGA. In the elastic computing heterogeneous architecture, the on-board computing resources of the remote sensing satellite are dynamically adjusted based on the virtual node dynamic mapping algorithm and the consistent hashing dynamic task allocation algorithm. The image stabilization processing module 52 is used to elastically allocate the on-board computing resources and use a multi-unit stacked pipeline parallel algorithm to process the elastically allocated on-board computing resources in parallel. It constructs an orientation model based on satellite video images and a geometric correction model between video image frames to obtain an optical satellite video image with geocoding. The compression encoding processing module 53 is used to compress and encode the optical satellite video image with the elastic computing heterogeneous architecture to shorten the input and output time. It hides the input and output time delay through streaming processing operations and inter-frame parallel processing and outputs a video stream.

[0161] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logic instructions in the memory 630 to execute an on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform. This method includes: determining a flexible computing heterogeneous architecture constructed from CPU, GPU, and FPGA; dynamically adjusting the on-board computing resources of the remote sensing satellite within the flexible computing heterogeneous architecture based on a virtual node dynamic mapping algorithm and a consistent hashing dynamic task allocation algorithm; flexibly allocating the on-board computing resources; using a multi-unit stacked pipeline parallel algorithm to process the flexible-allocated on-board computing resources in parallel; constructing a orientation model based on satellite video images and a geometric correction model between video image frames to obtain a geocoded stable optical satellite video image; using the flexible computing heterogeneous architecture to compress and encode the stable optical satellite video image to shorten input / output time; hiding the input / output time delay through streaming processing operations and inter-frame parallel processing; and outputting a video stream.

[0162] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform provided by the above methods. This method includes: determining a flexible computing heterogeneous architecture constructed from CPU, GPU, and FPGA; dynamically adjusting the on-board computing resources of the remote sensing satellite within the flexible computing heterogeneous architecture based on a virtual node dynamic mapping algorithm and a consistent hashing dynamic task allocation algorithm; flexibly allocating the on-board computing resources; using a multi-unit stacked pipeline parallel algorithm to process the flexible-allocated on-board computing resources in parallel; constructing a directional model based on satellite video images and a geometric correction model between video image frames to obtain a geocoded stable optical satellite video image; compressing and encoding the stable optical satellite video image using the flexible computing heterogeneous architecture to shorten input / output time; hiding the input / output time delay through streaming processing operations and inter-frame parallel processing; and outputting a video stream.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform, characterized in that, include: A flexible computing heterogeneous architecture is constructed using a central processing unit (CPU), a graphics processing unit (GPU), and a field-programmable gate array (FPGA). In this flexible computing heterogeneous architecture, the on-board computing resources of the remote sensing satellite are dynamically adjusted based on a virtual node dynamic mapping algorithm and a consistent hashing dynamic task allocation algorithm. The on-board computing resources are flexibly allocated, and a multi-unit stacked pipeline parallel algorithm is used to process the flexibly allocated on-board computing resources in parallel. A directional model based on satellite video images and a geometric correction model between video image frames are constructed to obtain a geocoded optical satellite video image. The elastic computing heterogeneous architecture is used to compress and encode the optical satellite video image stabilization to shorten the input and output time. The latency of input and output time is hidden by streaming processing and inter-frame parallel processing, and the video stream is output. In the aforementioned elastic computing heterogeneous architecture, based on the virtual node dynamic mapping algorithm and the consistent hashing dynamic task allocation algorithm, the on-board computing resources of the remote sensing satellite are dynamically adjusted, including: In the elastic computing heterogeneous architecture, a number of virtual nodes and a number of physical nodes are determined; If it is determined that all physical nodes are in an available state, then based on the total number of physical nodes, the virtual node is mapped to any physical node by taking the remainder of the index of any virtual node. If any physical node is determined to be unavailable, then based on the number of available physical nodes, any virtual node is mapped to any available physical node; The consistent hashing algorithm is used to calculate the hash value of each virtual node, and the hash value of each virtual node is mapped to a virtual ring of a preset length. The input data of the elastic computing heterogeneous architecture is obtained, the input data is divided, the hash value of each partition is calculated using a consistent hashing algorithm, and the hash value of each partition is mapped to the virtual ring of the preset length. On the virtual ring of the preset length, determine the virtual node closest to each partition data in a clockwise direction, and assign each partition data and its corresponding computing task to the nearest virtual node.

2. The on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform according to claim 1, characterized in that, The on-board computing resources are elastically allocated, and a multi-cell stacked pipeline parallel algorithm is used to process the elastically allocated on-board computing resources in parallel, including: Determine the total number of segments in the input data Number of processing units and the input time for each segment of data. Processing time and output time ; Calculate the processing time in a three-step stacked pipeline in parallel mode. for: in, , ; Total number of segments The speedup ratio of the three-step stacked pipeline in parallel is: 。 3. The on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform according to claim 1, characterized in that, Construct a orientation model based on satellite video images and a geometric correction model between video image frames to obtain geocoded stable optical satellite video images, including: Determine any two consecutive video frames as the main frame and the auxiliary frame to form multiple pairs of main and auxiliary frame images; Match multiple video image frames to obtain the matching points for each pair of main and auxiliary frame images; An orientation model for optical satellite video images and an inter-frame geometric correction model for auxiliary frame images were established respectively. Calculate the inter-frame geometric correction parameters for the supplementary frame image; Inter-frame geometric correction is performed on the orientation parameters of the auxiliary frame image to obtain the corrected orientation parameters; Geocoding is performed on the video images to obtain geocoded stable optical satellite video images.

4. The on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform according to claim 3, characterized in that, Inter-frame geometric correction is performed on the orientation parameters of the supplementary frame image to obtain the corrected orientation parameters, including: Divide the auxiliary frame image into multiple regular grids and set multiple elevation reference planes; Based on the orientation model of the auxiliary frame image, each regular grid point on the auxiliary frame image is projected onto the elevation datum plane to generate virtual control points; Inter-frame geometric correction is performed on the image-side coordinates of the virtual control points to obtain the corrected image-side coordinates. Using the virtual control points after inter-frame geometric correction, the corrected orientation parameters of the auxiliary frame image are calculated.

5. The on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform according to claim 3, characterized in that, Geocoding is performed on video images to obtain geocoded, stabilized optical satellite video images, including: Based on the orientation model of the video image, the four corner points of the first frame, middle frame and last frame of the video image are projected onto the digital elevation model to obtain the geographical range of the first frame, middle frame and last frame. The geographical extent of the stabilized optical satellite video image is obtained by extracting the intersection of the geographical extent of the first frame, the middle frame, and the last frame. Based on the geographic coordinates of the top left corner of the stabilized optical satellite video image, the image size, and the spatial resolution, calculate the geographic coordinates of each ground point within the geographic range of the stabilized optical satellite video image. Optical satellite video stabilization is generated sequentially for each frame of video image.

6. The on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform according to claim 1, characterized in that, The optical satellite video image stabilization is compressed and encoded using the aforementioned elastic computing heterogeneous architecture, including: The video compression coding process includes image segmentation, motion compensation, and intra-frame prediction sub-steps. Each image segmentation sub-step, each motion compensation sub-step, and each intra-frame prediction sub-step includes a first stage in which the CPU executes a preset low-computation algorithm and control logic to prepare for GPU-FPGA execution; a second stage in which the GPU-FPGA executes a preset large-scale parallel processing task while the CPU waits; and a third stage in which the CPU executes a preset low-computation algorithm and control logic and processes the GPU-FPGA calculation results. The time required for the first phase is determined to be The second phase took a time of The third stage took a time of ,in CPU time consumption GPU time consumption For FPGA time consumption, the CPU load is in three stages as follows: The parallel processing unit is idle in the first and third stages, and is fully loaded in the second stage (1). Determine the number of concurrent tasks as The total time for the second phase of all tasks is , Total time spent on each task for: CPU total load for: The computational time of the second-stage algorithm is calculated based on the total CPU runtime. Combined with GPU-FPGA parallel computing time , The proportion of computational workload allocated to the CPU; the total computational workload is... The processing performance during the second stage, when all CPU cores are used, is... When using GPU-FPGA parallel computing units, the processing performance is : Further analysis revealed: Obtained by sorting The second-stage time-consuming algorithm was modified, and the modified time consumption... for: 。 7. The on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform according to claim 1, characterized in that, By hiding the latency of input and output through streaming operations and inter-frame parallel processing, the output video stream includes: Multiple macroblocks are divided using pre-defined rectangular regions of different sizes; Scan the multiple macroblocks and divide the image frames with a chromaticity difference of less than a preset percentage in multiple adjacent image frames into the same group of image frames; Two frames of video data are extracted sequentially from the head of the buffer. Macroblock scanning is performed on the two frames of video data. If it is determined that an object exists in either frame of video data, a window search is performed in the neighborhood of the other frame of video. Once the object is found, the motion vector of the object is calculated. For each macroblock, perform pattern prediction, find the prediction pattern that is closest to the original image, subtract the original image from the intra-frame predicted image, and obtain the residual data. The residual data is subjected to integer discrete cosine transform to remove data correlation; By using a statistical frequency-assigned variable-length encoding table, the residual data after removing data correlation is converted into binary code to obtain the video stream.

8. A remote sensing satellite on-orbit video coding system based on a heterogeneous computing platform, comprising the remote sensing satellite on-orbit video coding method based on a heterogeneous computing platform as described in any one of claims 1 to 7, characterized in that, include: The elastic computing module is used to determine the elastic computing heterogeneous architecture constructed by CPU, GPU and FPGA, and dynamically adjust the on-board computing resources of the remote sensing satellite based on the virtual node dynamic mapping algorithm and the consistent hashing dynamic task allocation algorithm in the elastic computing heterogeneous architecture. The image stabilization module is used to flexibly allocate the on-board computing resources, and to use a multi-unit stacked pipeline parallel algorithm to process the flexibly allocated on-board computing resources in parallel. It constructs a directional model based on satellite video images and a geometric correction model between video image frames to obtain a geocoded optical satellite video image. The compression encoding processing module is used to compress and encode the optical satellite video image stabilization using the elastic computing heterogeneous architecture to shorten the input and output time. It hides the input and output time delay through streaming processing operations and inter-frame parallel processing, and outputs a video stream.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the on-orbit video coding method for remote sensing satellites based on a heterogeneous computing platform as described in any one of claims 1 to 7.

Images