Neural network processing method for multi-source image or video processing

Abstract

The invention relates to a neural network processing method for multi-source image or video processing, and the method comprises the following steps: 1) designing a hardware architecture based on NPU, the hardware architecture comprises a source data collection module, an intelligent computing hardware platform and a display control module, the source data collection module is connected with the intelligent computing hardware platform, and the display control module is connected with the intelligent computing hardware platform; the intelligent computing hardware platform is connected with the display control module; 2) performing special format conversion on a neural network algorithm according to multi-source image or video data features and intelligent processing requirements; 3) designing an edge end low-power-consumption intelligent computing hardware platform based on the multi-core NPU, the multi-core CPU and the FPGA; 4) realizing a neural network processing system for multi-source image or video processing based on the intelligent computing hardware platform designed in the step 3), wherein source data acquisition, data preprocessing, multi-source data parallel processing software design and result output are included; and 5) multi-source image or video processing performance is optimized, and real-time performance and accuracy are improved. Through a special NPU hardware architecture and a memory optimization technology, the multi-source data processing efficiency can be remarkably improved, the platform energy consumption is reduced, and powerful technical support is provided for the application fields of security monitoring and the like.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning hardware acceleration and multi-source data processing, and in particular to a neural network algorithm processing method for multi-source image or video processing based on NPU acceleration. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, image and video processing technology based on deep learning has been widely used in target detection, image classification, semantic segmentation, scene understanding and other fields. However, in practical applications, more and more scenarios involve the processing of multi-source data, which comes from image and video sensors of different devices or different bands (such as RGB cameras, infrared cameras, lidar, etc.). Since these data have different resolutions, formats and transmission rates, traditional computing platform acceleration processing methods are difficult to efficiently and reasonably complete task allocation, resulting in low processing efficiency and even affecting the accuracy of the task.

[0003] The training and reasoning process of neural network models usually requires a large amount of computing resources. When processing multi-channel data, the amount of computing increases dramatically, and computing resources need to be reasonably allocated for task processing. Especially in edge computing devices, the balance between power consumption and processing efficiency becomes more important. Summary of the invention

[0004] In order to solve the technical problems existing in the background technology, the present invention proposes a neural network algorithm processing method for multi-source image or video processing based on NPU acceleration. Through the dedicated NPU hardware architecture and memory optimization technology, it can significantly improve the multi-source data processing efficiency and reduce platform energy consumption, providing strong technical support for application fields such as security monitoring.

[0005] The technical solution of the present invention is: the present invention is a neural network processing method for multi-source image or video processing, and its special feature is that the method comprises the following steps:

[0006] 1) Design a hardware architecture based on NPU, which includes a source data acquisition module, an intelligent computing hardware platform and a display and control module, wherein the source data acquisition module is connected to the intelligent computing hardware platform, and the intelligent computing hardware platform is connected to the display and control module;

[0007] 2) According to the characteristics of multi-source image or video data and the requirements of intelligent processing, the neural network algorithm is converted into a dedicated format;

[0008] 3) Design a low-power intelligent computing hardware platform at the edge based on multi-core NPU, multi-core CPU and FPGA;

[0009] 4) Implementing a neural network processing system for multi-source image or video processing based on the intelligent computing hardware platform designed in step 3), including source data acquisition, data preprocessing, multi-source data parallel processing software design and result output;

[0010] 5) Optimize multi-source image or video processing performance to improve real-time performance and accuracy.

[0011] Furthermore, the specific steps of step 1) include:

[0012] The source data acquisition module is responsible for the collection and forwarding of source data, the intelligent processing and calculation module is responsible for the conversion of data formats, the deployment of intelligent image processing algorithms and algorithm post-processing, and the display and control module is responsible for the display of video intelligent processing results;

[0013] Furthermore, the specific steps of step 2) include:

[0014] 2.1) Design a neural network model compilation method. The model file suitable for NPU deployment needs to be compiled and generated by the NPU compilation tool. The converted model file is a binary data file. The file mainly consists of general control data, NPU configuration data, weight data parameters, weight data, instruction data parameters, and instruction data;

[0015] 2.2) Use the model compilation tool in step 2.1) to perform dedicated format conversion. According to the algorithm's computational workload and processing performance requirements, allocate bound core resources for different algorithms. When the number of tasks is greater than the number of NPU cores, the NPU cores need to be time-division multiplexed.

[0016] Furthermore, the specific steps of step 3) include:

[0017] 3.1) According to the power consumption and computing performance requirements of the scenario, a computing architecture based on multi-core CPU + multi-core NPU + FPGA is constructed;

[0018] 3.2) NPU focuses on accelerating the matrix operation computation-intensive tasks of forward reasoning of neural networks, CPU completes the non-computation-intensive tasks of data processing, multi-task scheduling and system management, and FPGA completes the interface implementation and video data preprocessing tasks.

[0019] Furthermore, the specific steps of step 4) include:

[0020] 4.1) Obtain video stream source data from multiple cameras or sensors and input it into the intelligent computing hardware platform;

[0021] 4.2) Preprocess multi-source data through the FPGA module of the intelligent computing hardware platform, including data format conversion, data scaling, and cropping;

[0022] 4.3) Input the preprocessed data into the CPU+NPU module;

[0023] 4.4) Based on the multi-core multi-algorithm software programming model, develop multi-source data parallel processing software: create an inference engine according to the task volume requirements, associate an algorithm with each inference engine and bind the required core resources; create multiple threads, and each thread completes the input loading, accelerated calculation and inference result output of the neural network algorithm by calling the inference engine;

[0024] 4.5) The CPU completes post-processing of multi-source output data, including picture frames, marks, etc., and outputs the processing results.

[0025] Furthermore, the specific steps of step 5) are: according to the multi-channel neural network task processing performance requirements, improve data transmission efficiency, and adjust the NPU core and CPU resource allocation method to meet higher requirements. The present invention provides a neural network processing system and method for multi-source image or video processing, which uses NPU dedicated hardware to design a multi-algorithm acceleration method to accelerate the deep learning model in parallel, significantly improving the computing efficiency of the neural network reasoning stage, especially in the feature extraction and feature fusion links, through parallel computing and memory access optimization, greatly shortening the reasoning time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The overall design flow chart of the present invention;

[0027] Figure 2 A schematic diagram of the model compilation file structure in step 2.1) of the present invention;

[0028] Figure 3 Schematic diagram of the NPU core allocation method in step 2.2) of the present invention;

[0029] Figure 4 Schematic diagram of the intelligent computing hardware platform in step 3) of the present invention;

[0030] Figure 5 It is a schematic diagram of the overall architecture and data flow of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] See also Figure 1 , the method steps of the specific embodiment of the present invention are as follows:

[0033] 1) Design a hardware architecture based on NPU, which includes a source data acquisition module, an intelligent computing hardware platform, and a display and control module. The source data acquisition module is connected to the intelligent computing hardware platform, and the intelligent computing hardware platform is connected to the display and control module. The process is as follows: Figure 1 shown.

[0034] 2) According to the characteristics of multi-source image or video data and the requirements of intelligent processing, the neural network algorithm is converted into a dedicated format;

[0035] 2.1) Design a neural network model compilation method. Figure 2 As shown, the model file suitable for NPU deployment needs to be compiled and generated by the NPU compilation tool. The converted model file is a binary data file, and the file composition mainly includes general control data, NPU configuration data, weight data parameters, weight data, instruction data parameters, and instruction data.

[0036] 2.2) Taking 3-way data as an example, they can be different types of data sources such as visible light, infrared, and multi-spectral, and corresponding intelligent image processing algorithms can be designed according to different task requirements. Use the dedicated format conversion of the neural network model compilation method in step 2.1) to allocate the bound core resources of different algorithms according to the algorithm's computational load and processing performance requirements. The NPU core allocation method is as follows: Figure 3 As shown, for example, Algorithm 1 is bound to 1 NPUCORE, Algorithm 2 is bound to 2 NPUCOREs, and Algorithm 3 is bound to 1 NPUCORE. When the number of tasks is greater than the number of NPU cores, the NPU cores need to be time-division multiplexed.

[0037] 3) Design a low-power intelligent computing hardware platform at the edge based on multi-core NPU, multi-core CPU and FPGA;

[0038] 3.1) According to the requirements of scene power consumption, computing performance, etc., a computing architecture based on multi-core CPU + multi-core NPU + FPGA is constructed. The computing architecture based on multi-core CPU + multi-core NPU + FPGA is as follows Figure 4 As shown,

[0039] 3.2) Among them, NPU focuses on accelerating computationally intensive tasks such as matrix operations for forward reasoning of neural networks, CPU completes non-computationally intensive tasks such as data processing, multi-task scheduling and system management, and FPGA completes tasks such as interface implementation and video data preprocessing.

[0040] 4) Based on the intelligent computing hardware platform designed in step 3), a neural network processing system for multi-source image or video processing is implemented, such as Figure 5 As shown, it includes source data acquisition, data preprocessing, multi-source data parallel processing software design and result output;

[0041] 4.1) Obtain video stream source data from multiple cameras or sensors and input it into the intelligent computing hardware platform;

[0042] 4.2) Preprocess multi-source data through the FPGA module of the intelligent computing hardware platform, including data format conversion, data scaling, and cropping;

[0043] 4.3) Input the preprocessed data into the CPU+NPU module;

[0044] 4.4) Based on the multi-core multi-algorithm software programming model, develop multi-source data parallel processing software: create an inference engine according to the task volume requirements, associate an algorithm with each inference engine and bind the required core resources; create multiple threads, and each thread completes the input loading, accelerated calculation and inference result output of the neural network algorithm by calling the inference engine;

[0045] 4.5) The CPU completes post-processing of multi-source output data, including picture frames, marks, etc., and outputs the processing results.

[0046] 5) Test and optimize multi-source image or video processing performance to improve real-time performance and accuracy.

[0047] According to the performance requirements of multi-channel neural network task processing, improve data transmission efficiency, adjust the NPU core and CPU resource allocation method, and optimize the task scheduling method to meet higher requirements.

[0048] The above are only specific embodiments disclosed in the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

[0049] The content of the present invention and the technical content not specifically described in the above embodiments are the same as the prior art.

[0050] The present invention is not limited to the above embodiments, and all of the contents of the present invention can be implemented and have the above good effects.

Claims

1. A neural network processing method for multi-source image or video processing, characterized in that: The method comprises the following steps: 1) Design a hardware architecture based on NPU, which includes a source data acquisition module, an intelligent computing hardware platform and a display and control module, wherein the source data acquisition module is connected to the intelligent computing hardware platform, and the intelligent computing hardware platform is connected to the display and control module; 2) According to the characteristics of multi-source image or video data and the requirements of intelligent processing, the neural network algorithm is converted into a dedicated format; 3) Design a low-power intelligent computing hardware platform at the edge based on multi-core NPU, multi-core CPU and FPGA; 4) Implementing a neural network processing system for multi-source image or video processing based on the intelligent computing hardware platform designed in step 3), including source data acquisition, data preprocessing, multi-source data parallel processing software design and result output; 5) Optimize multi-source image or video processing performance to improve real-time performance and accuracy.

2. The neural network processing method for multi-source image or video processing according to claim 1, characterized in that: The specific steps of step 1) include: The source data acquisition module is responsible for the collection and forwarding of source data, the intelligent processing and calculation module is responsible for the conversion of data formats, the deployment of intelligent image processing algorithms and algorithm post-processing, and the display and control module is responsible for the display of video intelligent processing results; 3. The neural network processing method for multi-source image or video processing according to claim 2, characterized in that: The specific steps of step 2) include: 2.1) Design a neural network model compilation method. The model file suitable for NPU deployment needs to be compiled and generated by the NPU compilation tool. The converted model file is a binary data file. The file mainly consists of general control data, NPU configuration data, weight data parameters, weight data, instruction data parameters, and instruction data; 2.2) Use the model compilation tool in step 2.1) to perform dedicated format conversion. According to the algorithm's computational workload and processing performance requirements, allocate bound core resources for different algorithms. When the number of tasks is greater than the number of NPU cores, the NPU cores need to be time-division multiplexed.

4. The neural network processing method for multi-source image or video processing according to claim 3, characterized in that: The specific steps of step 3) include: 3.1) According to the power consumption and computing performance requirements of the scenario, a computing architecture based on multi-core CPU + multi-core NPU + FPGA is constructed; 3.2) NPU focuses on accelerating the matrix operation computation-intensive tasks of forward reasoning of neural networks, CPU completes the non-computation-intensive tasks of data processing, multi-task scheduling and system management, and FPGA completes the interface implementation and video data preprocessing tasks.

5. The neural network processing system for multi-source image or video processing according to claim 4, characterized in that: The specific steps of step 4) include: 4.1) Obtain video stream source data from multiple cameras or sensors and input it into the intelligent computing hardware platform; 4.2) Preprocess multi-source data through the FPGA module of the intelligent computing hardware platform, including data format conversion, data scaling, and cropping; 4.3) Input the preprocessed data into the CPU+NPU module; 4.4) Based on the multi-core multi-algorithm software programming model, develop multi-source data parallel processing software: create an inference engine according to the task volume requirements, associate an algorithm with each inference engine and bind the required core resources; create multiple threads, and each thread completes the input loading, accelerated calculation and inference result output of the neural network algorithm by calling the inference engine; 4.5) The CPU completes post-processing of multi-source output data, including picture frames, marks, etc., and outputs the processing results.

6. The neural network processing method for multi-source image or video processing according to claim 5, characterized in that: The specific steps of step 5) are: according to the multi-channel neural network task processing performance requirements, improve data transmission efficiency, and adjust the NPU core and CPU resource allocation method to meet higher requirements.

Images