A lightweight hybrid heterogeneous image processing system for Mars rover constrained environments
By adopting a hybrid heterogeneous architecture of main processor, coprocessor and FPGA in Mars rover, image preprocessing and parallel processing are performed, the problem that traditional architecture cannot meet high real-time and high image processing performance is achieved, and efficient Image processing and autonomous management capabilities reduce design difficulty and resource consumption.
Patent Information
- Application Number
- CN202111164641.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-09-30
AI Technical Summary
The traditional CPU+FPGA architecture cannot meet the requirements of the Mars rover's high real-time, high reliability and high image processing performance, and is difficult to design and cannot achieve independent management capabilities.
Design a lightweight hybrid heterogeneous computing architecture, including the main processor, coprocessor and FPGA, image preprocessing and parallel processing are performed through the FPGA, the coprocessor performs DEM coordinate conversion and filtering, and the main processor is responsible for health monitoring. and collaborative control to achieve resource optimization and high reliability.
It realizes that the image processing time is shortened by an order of magnitude, improves image processing performance and autonomous management capabilities, reduces the difficulty of selecting CPU and FPGA, and enhances the reliability and collaborative operation capabilities of the entire machine.
Smart Images

Figure CN114119339B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a lightweight hybrid heterogeneous computing architecture in a restricted environment, which is suitable for computers with high real-time, high reliability, high image processing performance requirements and high autonomous management requirements. Background Art
[0002] Mars exploration is affected by the movement of celestial bodies. Compared with traditional satellites, the measurement and control time delay is large, and the ground cannot perform measurement and control of the spacecraft in real time. This requires the spacecraft to have a high degree of autonomy in orbit to ensure the safety requirements of the spacecraft in the event of a fault.
[0003] The rover navigation control unit uses image processing technology to achieve path planning, autonomous obstacle avoidance and other control functions for the vehicle. Image processing performance directly determines the vehicle's motion performance and autonomous operation capabilities. In view of the large delay in Mars exploration and measurement and control and the short daily operation time of the vehicle, the image processing task of the rover navigation control unit is required to take less than 92.3 seconds, and the image processing performance is required to be high.
[0004] The traditional independent implementation architecture of the central computer and the image processing unit has a large weight overhead, and the data interaction capability between the central computer and the image processing unit is weak. The traditional CPU+FPGA architecture has two implementation methods. One is that the FPGA is used to implement the interface logic control management function and the image data acquisition function, and the CPU is used to implement the guidance navigation control and image processing functions. In this solution, the FPGA function is simple, and various complex image processing algorithms are implemented through the processor software. The whole machine is simple to implement and the development difficulty is small, but the image processing performance is poor and cannot meet the time performance requirements. The other is that the FPGA is used to implement the interface logic control management function, the image data acquisition function and some image processing algorithm functions, and the CPU is used to implement the guidance navigation control and some image processing functions. This solution requires high CPU performance, no optional processor, and poor engineering feasibility; the FPGA function is complex and the design is difficult. The traditional architecture cannot meet the requirements of autonomous management capabilities and image processing performance of Mars exploration.
[0005] In the traditional CPU+FPGA architecture, whether the image processing function is performed by the processor or the FPGA, the software design is difficult, the CPU and FPGA are difficult to select, and the image processing performance is relatively poor. Summary of the invention
[0006] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art and provide a satellite-borne hybrid heterogeneous computing architecture with simple design, high reliability and high computing performance, so as to achieve the goal of reducing circuit scale but increasing computing power, image processing performance and autonomous management performance.
[0007] The solution to the technical problem of the present invention is: a lightweight hybrid heterogeneous image processing system in a constrained environment of a Mars rover, the system comprising a main processor, a coprocessor and an FPGA; wherein:
[0008] The main processor sends an acquisition command to the image acquisition device, and then sends an image processing task start command to the coprocessor and FPGA at the same time;
[0009] FPGA, receiving and storing the two-dimensional image data frames sent by the image acquisition device in real time, and after receiving the image processing task start command, preprocessing the two-dimensional image data frames, wherein the preprocessing includes extraction correction, removing distortion in the data, and realizing epipolar alignment and DOG filtering, performing three-dimensional restoration on multiple two-dimensional image data frames in the same scene after preprocessing to obtain three-dimensional image data, storing the three-dimensional image data in a buffer, and setting an image processing flag to indicate that the three-dimensional image data has been formed;
[0010] After receiving the image processing start command from the main processor, the coprocessor periodically queries the image processing flag of the FPGA. When it is found that the image processing flag is set, it reads the three-dimensional image data in the FPGA cache area, and performs DEM coordinate conversion and DEM filtering on the three-dimensional image data in turn to generate a topographic map of the Martian surface.
[0011] Preferably, when the image acquisition device is a binocular camera, the two-dimensional image data frame includes
[0012] For the left and right images of the same scene, FPGA extracts and corrects the left and right images respectively to remove the distortion in the data and realize epipolar alignment, and then performs DOG filtering. For the filtered left and right images, the matching same-name points are obtained through moving window registration and left-right consistency check. Sub-pixel interpolation is performed on the positions corresponding to the matching same-name points to improve the registration accuracy. The parallax obtained by the registration is used for three-dimensional restoration to obtain three-dimensional image data.
[0013] Preferably, the main processor is implemented by a CPU.
[0014] Preferably, the coprocessor is implemented using DSP.
[0015] Preferably, the main processor obtains the health status of the coprocessor and performs status determination processing;
[0016] If the coprocessor is unhealthy, the coprocessor is reconstructed, including reloading and resetting the coprocessor, and sending an image processing task start command to the coprocessor.
[0017] Preferably, the main processor obtains the FPGA health status and performs status determination processing;
[0018] If the FPGA is unhealthy, the FPGA is reconstructed, including FPGA reconfiguration and reset processing measures, and an image processing task start command is sent to the FPGA.
[0019] Preferably, the coprocessor implements the DEM coordinate conversion processing program and the DEM filtering processing program of the three-dimensional image data by using a parallel computing optimization method based on a single-core DSP. The parallel computing optimization method based on a single-core DSP is specifically as follows:
[0020] The DEM filtering processing program is regarded as a key complex program, and the DEM coordinate conversion processing program and the program for generating the Martian surface terrain map are regarded as non-key complex programs; the key complex program is fixedly run in the internal cache of the DSP, and the non-key complex program is run in the externally configured RAM memory;
[0021] The large loop bodies of key complex programs and non-key complex programs are split into simple small loop bodies. The hardware resources used by each loop body do not exceed the number of computing units and registers in the coprocessor, so that the statements in each loop body can be executed in parallel in multiple computing units at the same time; the loop counter in each loop body is decremented, and loop nesting, function calls and break jumps are not performed in the loop body;
[0022] Use conditional operation statements to implement branch judgment and reduce program jumps;
[0023] Single instruction multiple data instructions are used to assemble data for the same calculation processing to improve the parallelism of data processing.
[0024] Preferably, fixed-address compilation is used for both critical complex programs and non-critical complex programs. Fixed-address compilation is implemented by defining code segments and specifying code segment address spaces in compiler link files.
[0025] The beneficial effects of the present invention compared with the prior art are:
[0026] (1) The present invention adopts a hybrid heterogeneous architecture of main processor + coprocessor to achieve high-performance computing under resource constraints. Compared with the single-processor computing method in the prior art, the image processing time is shortened by one order of magnitude while ensuring the reliability of the whole product.
[0027] (2) The present invention uses FPGA to complete parallel processing and integer processing operations in the image algorithm, and uses DSP to complete floating-point operations in the image algorithm, thereby achieving optimal application of hardware resources and ensuring high performance and real-time performance of image processing;
[0028] (3) The present invention improves the parallelism of storage access: segmented addressing compilation is used to fix the key complex algorithms to run in the DSP internal cache with a larger data access bit width;
[0029] (4) The present invention uses a high-reliability processor to complete the coordinated control of the computing process of the hybrid computing architecture as well as health monitoring and fault-tolerant management, thereby ensuring the high coordinated operation of the hybrid computing architecture and improving the reliability of the entire machine. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a topological diagram of the internal interconnection of the hybrid heterogeneous computing architecture according to an embodiment of the present invention;
[0031] Figure 2 This is a schematic diagram of the image processing task flow in an embodiment of the present invention;
[0032] Figure 3 This is a flowchart of the health monitoring and reconstruction tasks according to an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described below in conjunction with the embodiments.
[0034] The present invention provides a lightweight hybrid heterogeneous image processing system in a constrained environment of a Mars rover, the system comprising a main processor, a coprocessor and an FPGA; wherein:
[0035] The main processor sends an acquisition command to the image acquisition device, and then sends an image processing task start command to the coprocessor and FPGA at the same time;
[0036] FPGA, receiving and storing the two-dimensional image data frames sent by the image acquisition device in real time, and after receiving the image processing task start command, preprocessing the two-dimensional image data frames, wherein the preprocessing includes extraction correction, removing distortion in the data, and realizing epipolar alignment and DOG filtering, performing three-dimensional restoration on multiple two-dimensional image data frames in the same scene after preprocessing to obtain three-dimensional image data, storing the three-dimensional image data in a buffer, and setting an image processing flag to indicate that the three-dimensional image data has been formed;
[0037] After receiving the image processing start command from the main processor, the coprocessor periodically queries the image processing flag of the FPGA. When it is found that the image processing flag is set, it reads the three-dimensional image data in the FPGA cache area, and performs DEM coordinate conversion and DEM filtering on the three-dimensional image data in turn to generate a topographic map of the Martian surface.
[0038] When the image acquisition device is a binocular camera, the two-dimensional image data frame includes a left image and a right image of the same scene at the same time. The FPGA extracts and corrects the left image and the right image in turn to remove the distortion in the data and realize the epipolar alignment, and then performs DOG filtering. For the filtered left and right images, the matching same-name points are obtained through moving window registration and left-right consistency check. Sub-pixel interpolation is performed on the positions corresponding to the matching same-name points to improve the registration accuracy. The parallax obtained by the registration is used to perform three-dimensional restoration to obtain three-dimensional image data.
[0039] The coprocessor implements the DEM coordinate conversion processing program and the DEM filtering processing program of the three-dimensional image data by using a parallel computing optimization method based on a single-core DSP. The parallel computing optimization method based on a single-core DSP is specifically as follows:
[0040] The DEM filtering processing program is regarded as a key complex program, and the DEM coordinate conversion processing program and the program for generating the Martian surface terrain map are regarded as non-key complex programs; the key complex program is fixedly run in the internal cache of the DSP, and the non-key complex program is run in the externally configured RAM memory;
[0041] Split the large loop bodies in key complex programs and non-key complex programs into simple small loop bodies. The hardware resources used by each loop body do not exceed the number of computing units and registers in the coprocessor, so that the statements in each loop body can be executed in parallel in multiple computing units at the same time. The loop counter in each loop body is decremented, and loop nesting, function call and break are not performed in the loop body.
[0042] Use conditional operation statements to implement branch judgment and reduce program jumps;
[0043] Single instruction multiple data instructions are used to assemble data for the same calculation processing to improve the parallelism of data processing.
[0044] Both critical complex programs and non-critical complex programs use fixed-address compilation, which is implemented by defining code segments and specifying the code segment address space in the compiler link file.
[0045] Define code snippet mode:
[0046] #pragma CODE_SECTION(key algorithm function name, "KeyAlgorithmText")
[0047] Specify the code segment address space in the compiler link file:
[0048] MEMORY
[0049] {
[0050] InternalRAM: origin = internal cache start address, length = internal cache byte length
[0051] ExternalRAM: origin = external memory starting address, length = external memory byte length
[0052] }
[0053] SECTIONS
[0054] {
[0055] .KeyAlgorithmText:>InternalRAM
[0056] .NonKeyAlgorithmText:>ExternalRAM
[0057] }
[0058] The system works on Mars. Mars exploration is affected by the movement of celestial bodies. Compared with traditional satellites, the measurement and control time delay is large, and the ground cannot measure and control the aircraft in real time. The aircraft is required to have a high degree of autonomy in orbit to ensure that the safety requirements of the aircraft are met in the event of a fault. Therefore, the main processor of this system is responsible for real-time monitoring and management of the working status of the coprocessor and FPGA to ensure the reliability of the system.
[0059] The main processor obtains the health status of the coprocessor and performs status determination processing;
[0060] The main processor determines the health status of the coprocessor by accessing the content of the designated memory area of the coprocessor. When the content of the designated memory area is a preset health mark, the coprocessor is considered healthy, otherwise, the coprocessor is considered unhealthy.
[0061] If the coprocessor is unhealthy, the coprocessor is reconstructed, including reloading and resetting the coprocessor, and sending an image processing task start command to the coprocessor.
[0062] The main processor obtains the health status of the FPGA and performs status determination processing;
[0063] The FPGA health status is detected by the following method: when the FPGA image processing flag is not set for more than a preset time, the FPGA is considered to be unhealthy, otherwise, the FPGA is considered to be healthy.
[0064] If the FPGA is unhealthy, the FPGA is reconstructed, including FPGA reconfiguration and reset processing measures, and an image processing task start command is sent to the FPGA.
[0065] Example 1
[0066] like Figure 1 As shown, in a specific embodiment of the present invention, the "main processor CPU + floating-point processing device DSP + parallel processing and integer processing device FPGA" method is adopted to realize the functions of the central computer and the image processing unit, the parallel processing and integer processing device FPGA is used to realize the interface logic control management function, the image data collection and acquisition function, the image processing parallel processing and integer processing algorithm function, the floating-point processing device DSP is used to complete the image processing floating-point operation function, the main processor CPU is used for the image processing control management function, the DSP and FPGA health monitoring and system reconstruction function.
[0067] The main processor selection has few constraints, but the floating-point processing device, parallel processing device and integer processing device still need high performance. This architecture completes the high-performance computing tasks of the whole machine through the division of labor and cooperation between the main processor and the floating-point processing device, parallel processing device and integer processing device, reducing the performance requirements for a single processor.
[0068] like Figure 2 As shown, in this embodiment, the image acquisition device selects a binocular camera, the main processing is selected as a CPU, and the coprocessor is selected as a DSP. The division of labor and cooperation between the CPU, DSP and FPGA is divided into two parts: 1) division of labor and cooperation in image processing; 2) division of labor and cooperation in state monitoring and reconstruction.
[0069] 1) The division of labor and coordination of image processing is as follows:
[0070] S1.1, the CPU sends an image acquisition command to the image acquisition device;
[0071] S1.2, CPU sends image processing task start command to DSP and FPGA;
[0072] S2.1, FPGA receives and stores a two-dimensional image data frame sent by a binocular camera, wherein the two-dimensional image data frame includes a left image and a right image;
[0073] S2.2, FPGA extracts and corrects the left image and the right image in turn, removes the distortion in the data and realizes the epipolar alignment, and performs DOG filtering;
[0074] S2.3, after FPGA completes data filtering, binocular matching is performed through matching of same-name points and left-right consistency check;
[0075] S2.4, after completing binocular matching, FPGA performs sub-pixel interpolation, 3D point cloud generation and visual history feature point calculation to achieve 3D restoration.
[0076] S2.5, FPGA completes the 3D restoration, that is, the FPGA image processing task is completed, the processed data is cached in its internal cache area, and a flag is set to notify the DSP that the data processing is completed.
[0077] S3.1, after receiving the image processing start command from the CPU, the DSP periodically queries the image processing flag of the FPGA, and when it is found that the FPGA has completed the image processing task, it reads the data in the FPGA buffer area;
[0078] S3.2, DSP takes over FPGA to continue image processing, perform DEM coordinate conversion, DEM filtering, generate suitability map for the Mars rover to select obstacle avoidance path, and return the final obstacle avoidance path selection to the main processor CPU.
[0079] 2) Division of labor and coordination between status monitoring and reconstruction
[0080] like Figure 3 As shown, the processor obtains the DSP health status and performs status judgment processing;
[0081] If the DSP is unhealthy, the DSP is reconstructed, including DSP reloading, resetting and other processing measures, and an image processing task start command is sent to the coprocessor.
[0082] The processor obtains the FPGA health status and performs status judgment processing;
[0083] If the FPGA is unhealthy, the FPGA is reconstructed, including FPGA reconfiguration, reset and other processing measures, and an image processing task start command is sent to the FPGA.
[0084] The health monitoring and reconstruction tasks are completed.
[0085] The above technical solutions reduce the difficulty of selecting CPU, DSP and FPGA and the difficulty of software design, and have strong engineering feasibility. According to this hybrid heterogeneous computing architecture, through the division of labor and cooperation between the main processor and the floating-point processing device and the parallel processing and integer processing devices, the difficulty of selecting a single processor is reduced while improving the processing performance of the whole machine.
[0086] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.
Claims
1. A lightweight hybrid heterogeneous image processing system for Mars rover in a restricted environment, characterized by It includes a main processor, a coprocessor and an FPGA; among which: The main processor sends an acquisition command to the image acquisition device, and then sends an image processing task start command to the coprocessor and FPGA at the same time; FPGA, receiving and storing the two-dimensional image data frames sent by the image acquisition device in real time, and after receiving the image processing task start command, preprocessing the two-dimensional image data frames, wherein the preprocessing includes extraction correction, removing distortion in the data, and realizing epipolar alignment and DOG filtering, performing three-dimensional restoration on multiple two-dimensional image data frames in the same scene after preprocessing to obtain three-dimensional image data, storing the three-dimensional image data in a buffer, and setting an image processing flag to indicate that the three-dimensional image data has been formed; The coprocessor, after receiving the image processing start command from the main processor, periodically queries the image processing flag of the FPGA. When the image processing flag is set, it reads the three-dimensional image data in the FPGA buffer area, performs DEM coordinate conversion and DEM filtering on the three-dimensional image data in sequence, and generates a topographic map of the Martian surface. The coprocessor implements the DEM coordinate conversion processing program and the DEM filtering processing program of the three-dimensional image data by using a parallel computing optimization method based on a single-core DSP. The parallel computing optimization method based on a single-core DSP is specifically as follows: The DEM filtering processing program is regarded as a key complex program, and the DEM coordinate conversion processing program and the program for generating the Martian surface terrain map are regarded as non-key complex programs; the key complex program is fixedly run in the internal cache of the DSP, and the non-key complex program is run in the externally configured RAM memory; The large loop bodies of key complex programs and non-key complex programs are split into simple small loop bodies. The hardware resources used by each loop body do not exceed the number of computing units and registers in the coprocessor, so that the statements in each loop body can be executed in parallel in multiple computing units at the same time; the loop counter in each loop body is decremented, and loop nesting, function calls and break jumps are not performed in the loop body; Use conditional operation statements to implement branch judgment and reduce program jumps; Single instruction multiple data instructions are used to assemble data for the same calculation processing to improve the parallelism of data processing.
2. A lightweight hybrid heterogeneous image processing system for Mars rover in a constrained environment according to claim 1, characterized in that When the image acquisition device is a binocular camera, the two-dimensional image data frame includes a left image and a right image of the same scene at the same time. The FPGA extracts and corrects the left image and the right image in turn to remove the distortion in the data and realize the epipolar alignment, and then performs DOG filtering. For the filtered left and right images, the matching same-name points are obtained through moving window registration and left-right consistency check. Sub-pixel interpolation is performed on the positions corresponding to the matching same-name points to improve the registration accuracy. The parallax obtained by the registration is used to perform three-dimensional restoration to obtain three-dimensional image data.
3. A lightweight hybrid heterogeneous image processing system for a Mars rover in a restricted environment as claimed in claim 1, characterized in that The main processor is implemented by a CPU.
4. A lightweight hybrid heterogeneous image processing system for a Mars rover in a restricted environment as claimed in claim 1, characterized in that The coprocessor is implemented by using DSP.
5. A lightweight hybrid heterogeneous image processing system for Mars rover in a constrained environment according to claim 1, characterized in that The main processor obtains the health status of the coprocessor and performs status determination processing; If the coprocessor is unhealthy, the coprocessor is reconstructed, including reloading and resetting the coprocessor, and sending an image processing task start command to the coprocessor.
6. A lightweight hybrid heterogeneous image processing system for Mars rover in a restricted environment according to claim 1, characterized in that The main processor obtains the health status of the FPGA and performs status determination processing; If the FPGA is unhealthy, the FPGA is reconstructed, including FPGA reconfiguration and reset processing measures, and an image processing task start command is sent to the FPGA.
7. A lightweight hybrid heterogeneous image processing system for Mars rover in a restricted environment according to claim 1, characterized in that Fixed-address compilation is used for both critical complex programs and non-critical complex programs. Fixed-address compilation is implemented by defining code segments and specifying the code segment address space in the compiler link file.
