Software and hardware integrated parallel reconfigurable image real-time processing system

Abstract

The invention discloses a software and hardware integrated parallel reconfigurable image real-time processing system. A system input module is used for receiving a to-be-processed image and a preset process configuration file and sending the to-be-processed image and the preset process configuration file to a resource scheduling module; the main control unit of the resource scheduling module reconstructs the architecture of each DSP node unit according to the flow configuration file to form a processing flow of each DSP node unit, and configures an image processing algorithm in each DSP node unit based on the flow configuration file; the SRIO high-speed interconnection unit is a routing type data exchange channel; a main core processing unit of the multi-core task distribution module is responsible for main flow processing of an image processing algorithm in the DSP node unit and forms a task queue; and the slave core processing unit executes each image processing task in the task queue by using a slave core of the DSP node unit, summarizes and outputs the image processing tasks to the master core processing unit after all the image processing tasks are executed, and the master core processing unit outputs an image processing result through the SRIO high-speed interconnection unit.

Description

Technical Field

[0001] The invention belongs to the field of embedded real-time image processing, and in particular relates to a software and hardware integrated parallel reconfigurable image real-time processing system. Background Art

[0002] The real-time image processing system mainly uses parallel processing resources such as DSP to complete image processing tasks such as enhancement, detection and recognition. The processing efficiency is equivalent to the frame rate of the acquired image. Existing real-time image processing systems developed based on DSP often rely on software development tools provided by hardware developers, use static compilation and linking, have high multi-core task coupling, and cannot dynamically load or replace algorithm components at runtime. When multiple tasks are parallel, resource competition lacks hardware-level isolation, such as cache pollution and memory bandwidth preemption, resulting in performance fluctuations.

[0003] Patent application with publication number CN109710399A: A DSP communication task scheduling system and method, which classifies the communication tasks of each submodule of the DSP communication task scheduling system according to the communication task function, sets the priority of each type of communication task according to the system function requirements, sets the task scheduling system running state machine, and schedules tasks according to the state of the state machine and the message authority and classification. This solution focuses on the collaborative work between submodules, and does not consider the comprehensive scheduling of multiple DSPs and multiple cores and the integrated optimization of software algorithms on this hardware platform, making it difficult to collaboratively play the advantages of the software and hardware computing resources of the real-time image processing system. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a parallel reconfigurable image real-time processing system with integrated hardware and software, so as to overcome the problems of high difficulty in process reconstruction, difficulty in efficient allocation of computing resources, and difficulty in effective utilization of DSP multi-core resources under traditional multi-core DSP hardware platforms.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions:

[0006] A software and hardware integrated parallel reconfigurable image real-time processing system, including a system input module, a resource scheduling module, and a multi-core task allocation module;

[0007] The system input module is used to receive the image to be processed and the preset process configuration file and send them to the resource scheduling module;

[0008] The resource scheduling module includes the main control unit, DSP node unit and SRIO high-speed interconnection unit:

[0009] The main control unit reconstructs the architecture of each DSP node unit according to the process configuration file to form the processing flow of each DSP node unit, and configures the image processing algorithm in each DSP node unit based on the process configuration file; the SRIO high-speed interconnection unit is a routing data exchange channel to realize the exchange of input and output data when each DSP node unit uses the image processing algorithm to process the image to be processed;

[0010] The multi-core task allocation module is deployed in each DSP node unit, including the main core processing unit and the slave core processing unit:

[0011] The main core processing unit is responsible for the main process processing of the image processing algorithm in the DSP node unit. It uses the main core of the DSP node unit to allocate image processing tasks and form a task queue and then output it to the slave core processing unit; the slave core processing unit uses the slave core of the DSP node unit to execute each image processing task in the task queue, and after all the image processing tasks are completed, they are summarized and output to the main core processing unit. The main core processing unit outputs the image processing results through the SRIO high-speed interconnection unit.

[0012] Furthermore, the configuration information in the process configuration file stores the serial-parallel structure of each DSP node unit and the parameters of the image processing algorithm deployed by each DSP node unit in a depth-first traversal manner; wherein the serial-parallel structure includes a serial architecture, a parallel architecture, or a serial-parallel combination architecture.

[0013] Furthermore, the image to be processed is sent to the SRIO high-speed interconnect unit, and then the image is forwarded to the DSP node unit again based on routing according to the serial-parallel structure of the DSP node unit set in the process configuration file.

[0014] Furthermore, the DSP node unit is a computing node, which acts as a different process node according to its processing flow and executes a corresponding image processing algorithm.

[0015] Furthermore, the main control unit extracts the image processing algorithm in each DSP node unit, obtains the parameters of the image processing algorithm and the serial-parallel structure of each DSP node unit through the process configuration file, and reconstructs the DSP node unit architecture based on the serial-parallel structure;

[0016] After receiving the data packet of the image processing algorithm configured with parameters, the DSP node unit performs verification. After the verification passes, the original data in the FLASH is first erased, and then the image processing algorithm is written into the FLASH. After that, the DSP node unit runs according to the written image processing algorithm and the serial-parallel sequence of each DSP node unit in the serial-parallel structure of the DSP node unit.

[0017] Furthermore, a routing switching circuit is adopted in the SRIO high-speed interconnection unit, and each DSP node unit is modularly designed as an end node unit. Each DSP node unit is interconnected through a high-speed interface; the data and results of each DSP node unit in the process of executing the image processing task are transmitted between each DSP node unit through the routing switching circuit.

[0018] Furthermore, the processing result of the last DSP node unit in the serial-parallel structure of the DSP node units is output through the SRIO high-speed interconnect unit as the final image processing result.

[0019] Furthermore, the calling relationship and process of the master core processing unit and the slave core processing unit are as follows:

[0020] When the master core processing unit receives the image task execution instruction from the user, it enters the system state and starts to execute the main process of the image processing algorithm. At this time, the slave core processing unit is in a waiting state;

[0021] When the main process runs to the stage where parallel processing is required, the main core processing unit divides the image to be processed into blocks, forms different image processing tasks based on the blocks, adds them to the task queue, and stores the resources required for the image processing tasks in the handle; the slave core processing unit wakes up the slave core of the corresponding DSP node unit according to the handle to perform parallel processing of the image processing tasks; whenever there is an idle slave core, a new image processing task is taken out of the task queue for processing; when the slave core has completed processing all image processing tasks, the main core processing unit switches from the system state to the user state to obtain control, and then summarizes the data obtained from the image processing tasks, and sends the processing results to the SRIO high-speed interconnection unit.

[0022] A large-field-of-view target segmentation and counting device adopts the hardware-software integrated parallel reconfigurable image real-time processing system to perform spot counting on a large-field-of-view starry sky image.

[0023] Compared with the prior art, the present invention has the following technical features:

[0024] 1. The present invention provides a routing-based efficient resource scheduling method, which adopts a routing switching circuit method to design each DSP node unit as an independently decoupled computing module. Image data, parameters, and processing results can all be sent to the DSP node unit at a specific address through the routing switching circuit, effectively improving the communication efficiency between various resources.

[0025] 2. The present invention can use configuration files and main control units to organize DSP node units into any serial architecture, parallel architecture or serial-parallel combination architecture according to actual application requirements; if a DSP node unit fails, it is only necessary to modify the process configuration file to reconstruct the processing relationship of the DSP node unit, avoid the faulty DSP node unit, and restore the system function.

[0026] 3. The present invention can adapt to a variety of image processing applications and different forms of image input by updating the image processing algorithm application data and parameters in the main control unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a system block diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention provides a software and hardware integrated parallel reconfigurable image real-time processing system, comprising a system input module, a resource scheduling module, and a multi-core task allocation module; the resource scheduling module comprises a main control unit, a DSP node unit, and an SRIO high-speed interconnection unit; the multi-core task allocation module is deployed in each DSP node unit, comprising a main core processing unit and a slave core processing unit, wherein:

[0029] 1. System input module.

[0030] The system input module is used to receive the image to be processed and the preset process configuration file and send them to the resource scheduling module for allocation and processing of image processing tasks; the process configuration file is preset with the serial and parallel structure of the DSP node unit and the parameters of the image processing algorithm.

[0031] Among them, the configuration information in the process configuration file stores the serial-parallel structure of each DSP node unit and the parameters of the image processing algorithm deployed by each DSP node unit in a depth-first traversal manner; wherein, the serial-parallel structure is set according to the actual application requirements, including a serial architecture, a parallel architecture or a serial-parallel combination architecture; under the parallel architecture, the image can be processed in blocks, and the computing power of each DSP node unit is M. Under the framework of independent operation of each DSP node unit, the computing power of N DSP node units running simultaneously can reach M*N, and the computing power is superimposed. Under the serial architecture, multiple DSP node units can jointly complete a set of image processing algorithm data flows. Through pipeline functional division, each DSP node unit can be modularized, convenient and specialized, improve the computing efficiency of each DSP, simplify the operation of a single DSP, and improve the reliability of the design.

[0032] Among them, the image to be processed is sent to the SRIO high-speed interconnection unit of the resource scheduling module, and then the image is forwarded to the DSP node unit based on routing according to the serial-parallel structure of the DSP node unit set in the process configuration file.

[0033] 2. Resource scheduling module.

[0034] (2.1) Main control unit and DSP node unit.

[0035] After receiving the process configuration file, the main control unit in the resource scheduling module reconstructs the serial architecture, parallel architecture or serial-parallel combination architecture of each DSP node unit according to the configuration information in the process configuration file to form the processing flow of each DSP node unit; at the same time, the image processing algorithm in each DSP node unit is configured according to the parameters in the configuration information; the DSP node unit is a computing node, which acts as a different process node according to its processing flow and executes the corresponding image processing algorithm.

[0036] The main control unit extracts the image processing algorithm in each DSP node unit, and uses the configuration information in the process configuration file to obtain the parameters corresponding to the image processing algorithm and the serial-parallel structure of the DSP node unit, and updates the DSP node unit through the SPI / UART low-speed bus; the DSP node unit verifies the data packet of the image processing algorithm configured with the parameters, and erases the original data in the FLASH after the verification, and then writes the image processing algorithm into the FLASH, and then the DSP node unit runs according to the written image processing algorithm and the serial-parallel sequence of each DSP node unit in the serial-parallel structure of the DSP node unit.

[0037] (2.2)SRIO high-speed interconnect unit.

[0038] The SRIO high-speed interconnect unit in the resource scheduling module is a routing data exchange channel, which realizes the high-speed exchange of input and output data when each DSP node unit uses the image processing algorithm to process the image to be processed.

[0039] The specific settings of the SRIO high-speed interconnect unit are as follows:

[0040] A routing switching circuit is used in the SRIO high-speed interconnection unit, and each DSP node unit is modularly designed as an end node unit. Each DSP node unit is interconnected through a 4x SRIO Gen2 high-speed interface with a single link bandwidth of 5 Gbps / channel. The data and results of each DSP node unit in the process of executing image processing tasks are transmitted between each DSP node unit through a routing switching circuit.

[0041] 3. Multi-core task allocation module.

[0042] The multi-core task allocation module is deployed in each DSP node unit, including the main core processing unit and the slave core processing unit, where:

[0043] The main core processing unit is responsible for the main process processing of the image processing algorithm in the DSP node unit. It uses the main core of the DSP node unit to distribute and recycle image processing tasks and form a task queue, which is output to the slave core processing unit using the SRIO bus.

[0044] The slave core processing unit receives the task queue output by the main core processing unit, uses the slave core of the DSP node unit to execute each image processing task in the task queue, and summarizes and outputs the image processing tasks to the main core processing unit of the DSP node unit after the image processing tasks are completed. The main core processing unit outputs the image processing results through the SRIO high-speed interconnection unit; wherein, the processing result of the last DSP node unit in the serial-parallel structure of the DSP node unit output through the SRIO high-speed interconnection unit is used as the final image processing result.

[0045] The calling relationship and process of the master core processing unit and the slave core processing unit are as follows:

[0046] When the main core processing unit receives the image task execution instruction issued by the user, it enters the system state and starts to execute the main process of the image processing algorithm. At this time, the slave core processing unit is in a waiting state; when the main process runs to the stage where parallel processing is required, the main core processing unit divides the image to be processed into blocks, forms different image processing tasks based on the blocks, adds them to the task queue, and stores the resources required for the image processing tasks in the handle; the slave core processing unit wakes up the slave core of the corresponding DSP node unit according to the handle to perform parallel processing of the image processing tasks; whenever there is an idle slave core, a new image processing task is taken out from the task queue for processing; when the slave core has completed processing all image processing tasks, the main core processing unit switches from the system state to the user state to obtain control, and then performs summary processing on the data obtained from the image processing tasks, and sends the processing results to the SRIO high-speed interconnection unit.

[0047] In one embodiment of the present invention, the image processing function of counting the light spots of the starry sky image is realized by using the processing system, as follows:

[0048] The main control unit is pre-written with various image processing algorithms, including image segmentation threshold calculation application, spot binary segmentation application (including master core and slave core), spot connected area calculation and counting application; the parameters of each image processing algorithm and the set serial-parallel structure of the DSP node unit are pre-written in the configuration information of the process configuration file; after receiving the process configuration file, the resource scheduling module deploys the corresponding image processing algorithm to the DSP node units DSP1 to DSP4; the image processing algorithm of the corresponding DSP node unit is:

[0049] DSP1: Image segmentation threshold calculation application;

[0050] DSP2, DSP3: Spot binary segmentation application, including master core control code and slave core calculation code;

[0051] DSP4: Light spot connected area calculation and counting application.

[0052] The serial-parallel structure is set as follows: DSP2 and DSP3 are in parallel relationship, and they are in serial relationship with DSP1 and DSP4.

[0053] The image to be processed is a 16-bit grayscale image of size 4608×4608, which is sent to DSP1 through the SRIO high-speed interconnection unit; after receiving the image, DSP1 traverses the image pixels, statistically calculates the image histogram, and calculates the image segmentation threshold through the OTSU algorithm. The image segmentation threshold calculation result is sent to DSP2 through the SRIO high-speed interconnection unit to send the upper half of the image (size 2304×4608) and the lower half of the image data (size 2304×4608) to DSP3.

[0054] The main core processing unit of DSP2 divides the image of size 2304×4608 into data of size 144×144, totaling 512 blocks; all image blocks are formed into image processing tasks and added to the task queue, traversing the slave cores, and if there is an idle slave core, the image processing task is taken out from the task queue for allocation and processing.

[0055] After receiving the image processing task, the slave core processing unit of DSP2 segments the 144×144 image block according to the image segmentation threshold, sets the points greater than the threshold to 1, and sets the points less than the threshold to 0. After completion, the processed image block result is returned to the master core processing unit.

[0056] After receiving the results of all image processing tasks, the main core processing unit in DSP2 aggregates them into a binary image of size 2304×4608. The non-zero points in the image represent the target and the rest represent the background. The binary image result is sent to DSP4 through the SRIO high-speed interconnect channel.

[0057] The processing process of DSP3 is the same as that of DSP2.

[0058] After receiving the processing results from DSP2 and DSP3, DSP4 first stitches the results into a whole image, and marks the connected regions through the region growing method. The number of regions is the total number of light spots. The final light spot counting result can also be output through the SRIO high-speed interconnection unit.

[0059] In summary, the routing-based efficient resource scheduling method of the present invention adopts a routing switching circuit method to design each DSP node unit as an independently decoupled computing module. Image data, parameters, and processing results can be sent to the DSP node unit at a specific address through the routing switching circuit, effectively improving the communication efficiency between resources; through the process configuration file and the main control unit, the DSP node units are organized into any serial architecture, parallel architecture, or serial-parallel combination architecture according to actual application requirements; by updating the image processing algorithm and parameters in the main control unit, it can adapt to a variety of image processing applications and different forms of image input.

[0060] This system design scheme improves the flexibility of software operation on the hardware platform and the overall optimization of software and hardware performance, and improves the operating efficiency of the software and hardware integration. It can also achieve performance improvement in real-time image processing scenarios such as satellites and drones; the method of the present invention is applied to the 4608×4608 large field of view target segmentation and counting task, and the average time consumption is increased from 4400ms to 620ms, and the system operation efficiency is improved.

[0061] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A software and hardware integrated parallel reconfigurable image real-time processing system, characterized in that: Including system input module, resource scheduling module, and multi-core task allocation module; The system input module is used to receive the image to be processed and the preset process configuration file and send them to the resource scheduling module; The resource scheduling module includes the main control unit, DSP node unit and SRIO high-speed interconnection unit: The main control unit reconstructs the architecture of each DSP node unit according to the process configuration file to form the processing flow of each DSP node unit, and configures the image processing algorithm in each DSP node unit based on the process configuration file; the SRIO high-speed interconnection unit is a routing data exchange channel to realize the exchange of input and output data when each DSP node unit uses the image processing algorithm to process the image to be processed; The multi-core task allocation module is deployed in each DSP node unit, including the main core processing unit and the slave core processing unit: The main core processing unit is responsible for the main process processing of the image processing algorithm in the DSP node unit. It uses the main core of the DSP node unit to allocate image processing tasks and form a task queue and then output it to the slave core processing unit; the slave core processing unit uses the slave core of the DSP node unit to execute each image processing task in the task queue, and after all the image processing tasks are completed, they are summarized and output to the main core processing unit. The main core processing unit outputs the image processing results through the SRIO high-speed interconnection unit.

2. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 1, characterized in that: The configuration information in the process configuration file stores the serial-parallel structure of each DSP node unit and the parameters of the image processing algorithm deployed by each DSP node unit in a depth-first traversal manner; wherein the serial-parallel structure includes a serial architecture, a parallel architecture, or a serial-parallel combination architecture.

3. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 2, characterized in that: The image to be processed is sent to the SRIO high-speed interconnection unit, and then the image is forwarded to the DSP node unit based on routing according to the serial-parallel structure of the DSP node unit set in the process configuration file.

4. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 1, characterized in that: The DSP node unit is a computing node, which acts as a different process node according to its processing flow and executes a corresponding image processing algorithm.

5. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 2, characterized in that: The main control unit extracts the image processing algorithm in each DSP node unit, obtains the parameters of the image processing algorithm and the serial-parallel structure of each DSP node unit through the process configuration file, and reconstructs the DSP node unit architecture based on the serial-parallel structure; After receiving the data packet of the image processing algorithm configured with parameters, the DSP node unit performs verification. After the verification passes, the original data in the FLASH is first erased, and then the image processing algorithm is written into the FLASH. After that, the DSP node unit runs according to the written image processing algorithm and the serial-parallel sequence of each DSP node unit in the serial-parallel structure of the DSP node unit.

6. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 1, characterized in that: A routing switching circuit is used in the SRIO high-speed interconnection unit, and each DSP node unit is modularly designed as an end node unit. Each DSP node unit is interconnected through a high-speed interface; the data and results of each DSP node unit in the process of executing image processing tasks are transmitted between each DSP node unit through a routing switching circuit.

7. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 2, characterized in that: The processing result of the last DSP node unit in the serial-parallel structure of the DSP node unit is output through the SRIO high-speed interconnect unit as the final image processing result.

8. The software-hardware integrated parallel reconfigurable image real-time processing system according to claim 1, characterized in that: The calling relationship and process of the master core processing unit and the slave core processing unit are as follows: When the master core processing unit receives the image task execution instruction from the user, it enters the system state and starts to execute the main process of the image processing algorithm. At this time, the slave core processing unit is in a waiting state; When the main process runs to the stage where parallel processing is required, the main core processing unit divides the image to be processed into blocks, forms different image processing tasks based on the blocks and adds them to the task queue, and stores the resources required for the image processing tasks in the handle; The slave core processing unit wakes up the slave core of the corresponding DSP node unit according to the handle to perform parallel processing of image processing tasks; whenever there is an idle slave core, it takes out a new image processing task from the task queue for processing; When the slave core completes all image processing tasks, the master core processing unit switches from system state to user state to obtain control, and then summarizes the data obtained from the image processing tasks and sends the processing results to the SRIO high-speed interconnection unit.

9. A large-field-of-view target segmentation and counting device, characterized in that: The device uses the hardware-software integrated parallel reconfigurable image real-time processing system according to any one of claims 1 to 8 to count the number of spots on a large-field starry sky image.

Images