Video processing system based on domestic deep learning processor

Abstract

The invention discloses a video processing system based on a domestic deep learning processor, which belongs to the field of video processing systems and comprises a multi-dimensional Bayesian optimization module, an unweighted neural network module and a space-time denoising and filtering module. The multi-dimensional Bayesian optimization module uses multi-dimensional Bayesian optimization to explore and optimize the design space of the deep learning processor; the unweighted neural network module is used for introducing an unweighted neural network in neural network design, and replacing traditional weighted connection with an LUT structure, so that the calculation overhead and power consumption of a processor are reduced; and the space-time denoising filtering module provides a group of space-time denoising filters connected with the cache, and filters noise events in the preprocessing of the video stream to improve the signal-to-noise ratio of the video data. According to the invention, by combining the power consumption advantage of hardware neurons and the lightweight design of the unweighted neural network, the calculation and power consumption overhead of the processor is greatly reduced, so that the processor can run on low-power-consumption edge equipment.

Description

Technical Field

[0001] The present invention relates to the field of video processing systems, and in particular to a video processing system based on a domestic deep learning processor. Background Art

[0002] Video images are a sequence of continuous static images, which provide a more vivid and lively description of objective things. The video processing system is a system for collecting, transmitting, storing, processing (such as encoding, decoding, enhancing, denoising, etc.) and displaying these video images. It uses specialized hardware and software to process video images in real time or non-real time according to preset algorithms.

[0003] In modern video processing applications, especially edge computing scenarios (such as intelligent monitoring, autonomous driving, drone navigation, etc.), large amounts of video data need to be processed in real time and efficiently. However, existing deep learning models (such as convolutional neural networks and RNNs), although they perform well in image recognition and video analysis, are not efficient enough in terms of power consumption and hardware resource usage. This makes it challenging to run these models on resource-constrained devices.

[0004] The existing technology has the following problems: traditional deep learning models require a lot of computing resources and high power consumption when processing high-resolution videos, and are not suitable for implementation on low-power, resource-constrained devices. Therefore, those skilled in the art provide a video processing system based on a domestic deep learning processor to solve the problems raised in the above background technology. Summary of the invention

[0005] The purpose of the present invention is to provide a video processing system based on a domestic deep learning processor, which greatly reduces the computing and power consumption overhead of the processor by combining the power consumption advantages of hardware neurons and the lightweight design of unweighted neural networks, so that it can run on low-power edge devices to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] Video processing system based on domestic deep learning processor, including multi-dimensional Bayesian optimization module, unweighted neural network module and spatiotemporal denoising filter module;

[0008] The multi-dimensional Bayesian optimization module uses multi-dimensional Bayesian optimization to explore and optimize the design space of deep learning processors, covering the architecture, parameters and hardware implementation details of neural networks, combining the energy efficiency advantages of hardware neurons, and through a flexible modular architecture, enabling the processor to achieve an optimal balance in accuracy, power consumption, area and latency; the unweighted neural network module is used to introduce unweighted neural networks in neural network design, using LUT structures to replace traditional weighted connections, thereby reducing the computational overhead and power consumption of the processor, and reducing the complexity of the network and improving the efficiency of data processing through learnable mapping and reduction techniques; the spatiotemporal denoising filter module provides a spatiotemporal denoising filter with a group-connected cache, which filters noise events in the preprocessing of video streams, improves the signal-to-noise ratio of video data, and uses pipeline processing and read-out optimization techniques to achieve real-time denoising while reducing memory usage and power consumption.

[0009] As a further solution of the present invention: the multidimensional Bayesian optimization module includes a hardware neuron library, a multidimensional evaluator, and a network evaluator, wherein the hardware neuron library is used to provide efficient computing resources and optimized neural network performance, and support large-scale neural network training and reasoning; the multidimensional evaluator is used to efficiently evaluate the performance of multiple candidate solutions, and support multi-objective optimization, combine with the hardware neuron library to accelerate the evaluation process, provide rich evaluation results, and support adaptive evaluation strategies; the network evaluator is used to evaluate the performance of the neural network model.

[0010] As a further solution of the present invention: the network estimator includes a network generator, a power estimator, a delay estimator, an area estimator and an accuracy estimator, wherein the network generator is used to generate a corresponding neural network model according to the parameter configuration provided by the Bayesian optimization algorithm; the power estimator is used to evaluate the power consumption of the neural network model when running on hardware; the delay estimator is used to evaluate the delay of the neural network model when running on hardware; the area estimator is used to evaluate the physical area required for the neural network model when implemented in hardware; and the accuracy estimator is used to evaluate the prediction accuracy of the neural network model.

[0011] As a further solution of the present invention: the unweighted neural network module includes an interconnected network, two hidden layers of neural networks, a summation unit and a decision unit, wherein configurable logic blocks are provided between the interconnected network and the hidden layers of the neural networks, between the hidden layers of the two neural networks, and between the hidden layers of the neural networks and the summation unit, and the configurable logic blocks include a six-input lookup table and a trigger.

[0012] As a further solution of the present invention: the six-input lookup table includes two five-input lookup tables and a 2:1 multiplexer.

[0013] As a further solution of the present invention: the spatiotemporal denoising filter module includes a row storage unit, a column storage unit and a spatiotemporal denoising filter, wherein the row storage unit is used to store image data in the row direction; the column storage unit is used to store image data in the column direction; the spatiotemporal denoising filter is used to perform multi-directional filtering operations on the image, including filtering in the row and column directions, and filtering across time frames.

[0014] As a further solution of the present invention: the spatiotemporal denoising filter uses four row memory groups, and each memory block can store four events.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. This application can significantly improve the energy efficiency of video analysis. By combining the power consumption advantages of hardware neurons and the lightweight design of unweighted neural networks, the computing and power consumption overhead of the processor can be greatly reduced, enabling it to run on low-power edge devices.

[0017] 2. This application has the advantages of real-time processing and high accuracy. Through the optimized denoising preprocessing module and the automated design optimization engine, efficient real-time video processing can be achieved to ensure the accuracy and robustness of the analysis.

[0018] 3. This application has a flexible modular design, in which the integrated design framework has high scalability and flexibility. Users can adjust the network architecture and hardware implementation according to different application requirements, thereby quickly developing video analysis processors that adapt to specific scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of the structure of a video processing system based on a domestic deep learning processor;

[0020] Figure 2 This is a schematic diagram of the structure of the multi-dimensional Bayesian optimization module in the video processing system based on the domestic deep learning processor;

[0021] Figure 3 This is a schematic diagram of the structure of the unweighted neural network module in the video processing system based on the domestic deep learning processor;

[0022] Figure 4 This is a schematic diagram of the structure of the spatiotemporal denoising filter module in the video processing system based on the domestic deep learning processor. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] As mentioned in the background technology of this application, research has found that traditional deep learning models require a lot of computing resources and high power consumption when processing high-resolution videos. They are not suitable for implementation on low-power, resource-constrained devices and have certain defects.

[0025] In order to solve the above-mentioned defects, the present application discloses a video processing system based on a domestic deep learning processor. By combining the power consumption advantages of hardware neurons and the lightweight design of unweighted neural networks, the computing and power consumption overhead of the processor is greatly reduced, so that it can run on low-power edge devices.

[0026] The following will describe in detail how the solution of the present application solves the above technical problems in conjunction with the accompanying drawings.

[0027] See also Figures 1 to 4 In an embodiment of the present invention, a video processing system based on a domestic deep learning processor includes a multi-dimensional Bayesian optimization module, an unweighted neural network module, and a spatiotemporal denoising filter module; the multi-dimensional Bayesian optimization module uses multi-dimensional Bayesian optimization to explore and optimize the design space of the deep learning processor, covering the architecture, parameters and hardware implementation details of the neural network, and combines the energy efficiency advantages of hardware neurons. Through a flexible modular architecture, the processor can achieve an optimal balance in terms of accuracy, power consumption, area and delay; the unweighted neural network module is used to introduce an unweighted neural network into the neural network design, and use the LUT structure to replace the traditional weighted connection, thereby reducing the computational overhead and power consumption of the processor, and through a learnable mapping and reduction technology, reduce the complexity of the network and improve the efficiency of data processing; the spatiotemporal denoising filter module provides a spatiotemporal denoising filter with a group-connected cache, filters noise events in the preprocessing of the video stream, improves the signal-to-noise ratio of the video data, and uses pipeline processing and read cancellation optimization technology to achieve real-time denoising while reducing memory usage and power consumption. The present application can significantly improve the energy efficiency of video analysis. By combining the power consumption advantages of hardware neurons and the lightweight design of unweighted neural networks, the computing and power consumption overhead of the processor can be greatly reduced, allowing it to run on low-power edge devices.

[0028] In this embodiment, if Figure 1As shown in the figure, the multidimensional Bayesian optimization module includes a hardware neuron library, a multidimensional evaluator, and a network evaluator. The hardware neuron library is used to provide efficient computing resources and optimized neural network performance, and supports the training and reasoning of large-scale neural networks; the multidimensional evaluator is used to efficiently evaluate the performance of multiple candidate solutions, support multi-objective optimization, combine with the hardware neuron library to accelerate the evaluation process, provide rich evaluation results, and support adaptive evaluation strategies; the network evaluator is used to evaluate the performance of the neural network model. The core goal of Bayesian optimization is to find the global optimal solution through a limited number of evaluations. The multidimensional evaluator is designed for this purpose, and it can efficiently evaluate the performance of multiple candidate solutions (or parameter configurations). This means that in each iteration, the multidimensional evaluator can quickly test multiple possible solutions, thereby providing the Bayesian optimization algorithm with enough information to update its internal posterior distribution. In many practical applications, optimization problems often involve multiple objective functions or performance indicators. The multidimensional evaluator can evaluate these objectives simultaneously, thereby helping the optimization algorithm to make trade-offs and choices in multiple dimensions. This capability makes the Bayesian optimization framework more flexible and effective in dealing with complex multi-objective optimization problems. Multidimensional evaluators usually need to be combined with specific hardware neuron libraries to make full use of available computing resources and accelerate the evaluation process. By leveraging accelerators (such as GPUs or TPUs) in hardware neuron libraries and optimized hardware implementations, multidimensional evaluators can significantly reduce evaluation time, thereby improving the overall efficiency of Bayesian optimization. Multidimensional evaluators not only provide performance indicators of candidate solutions, but may also provide other useful information, such as model stability, generalization ability, etc. This information is crucial for in-depth understanding of the characteristics of candidate solutions and selecting the optimal solution. During the Bayesian optimization process, as the number of iterations increases, the evaluation requirements for candidate solutions may change. Multidimensional evaluators can support adaptive evaluation strategies, dynamically adjust the accuracy and scope of evaluation according to the current optimization status and needs, to further improve the efficiency and effectiveness of optimization.

[0029] In this embodiment, the network evaluator includes a network generator, a power estimator, a delay estimator, an area estimator and an accuracy estimator, wherein the network generator is used to generate a corresponding neural network model according to the parameter configuration provided by the Bayesian optimization algorithm (these parameter configurations may include the number of network layers, the number of neurons in each layer, the type of activation function, the connection weights, etc. The network generator constructs a specific neural network structure based on these configuration information to provide a basis for subsequent training and evaluation processes); the power estimator is used to evaluate the power consumption of the neural network model when it runs on hardware (this is crucial for designing energy-saving and efficient neural network models. By simulating or actually measuring the power consumption of the neural network running on hardware, the power estimator can help the optimization algorithm take power consumption factors into account when looking for the optimal solution, thereby generating a neural network model that is both efficient and energy-saving); the delay estimator is used to evaluate the delay of the neural network model when it runs on hardware (delay refers to the time required for the entire processing process from input data to output results. For real-time applications or those that require fast response In the scenario, latency is a very important performance indicator. The latency estimator provides feedback on latency to the optimization algorithm by simulating or actually measuring the running time of the neural network on the hardware, helping to generate a neural network model with lower latency); the area estimator is used to evaluate the physical area required for the neural network model to be implemented in hardware (this is crucial for designing compact and efficient hardware accelerators. By estimating the chip area required for the neural network to be implemented on hardware, the area estimator can help the optimization algorithm consider the feasibility of hardware implementation when looking for the optimal solution, thereby generating a neural network model that is both efficient and saves hardware resources); the accuracy estimator is used to evaluate the prediction accuracy of the neural network model (this is one of the key indicators for measuring the performance of the neural network model. By comparing the difference between the prediction results of the neural network on the training set and the validation set and the true label, the accuracy estimator can calculate the model's accuracy, recall rate, F1 score and other performance indicators, provide feedback on the model performance to the optimization algorithm, and help generate a neural network model with higher accuracy). The main responsibility of the network evaluator is to evaluate the performance of the neural network model. During the Bayesian optimization process, the algorithm generates a series of candidate parameter configurations that determine the structure and training method of the neural network. The network evaluator is responsible for training the neural network according to these parameter configurations and evaluating its performance on specific tasks, such as classification accuracy, regression error, etc. The network evaluator needs to interact closely with the hardware neuron library. The hardware neuron library provides the hardware resources required to implement neural networks, such as processors, memory, and accelerators. The network evaluator uses these resources to train the neural network and shortens the training time and improves the evaluation efficiency through hardware acceleration. In addition to evaluating the performance of the neural network under a given parameter configuration, the network evaluator also supports hyperparameter tuning and architecture search.This means that the network evaluator can automatically adjust the hyperparameters of the neural network (such as learning rate, batch size, etc.) or try different network architectures based on the recommendations of the Bayesian optimization algorithm to find a model with better performance. The network evaluator feeds the evaluation results back to the Bayesian optimization algorithm. These results include performance indicators of the neural network on the training set and validation set, as well as possible training time and resource consumption. The Bayesian optimization algorithm updates its internal posterior distribution based on this feedback and generates new candidate parameter configurations. By efficiently utilizing hardware resources and optimizing the evaluation process, the network evaluator helps speed up the overall process of Bayesian optimization. It enables the algorithm to explore more candidate solutions in a limited time, making it more likely to find a global optimal solution or a solution close to the global optimal solution.

[0030] In this embodiment, if Figure 2 As shown in FIG. 1 , the unweighted neural network module includes an interconnected network, two hidden layers of neural networks, a summing unit and a decision unit, wherein configurable logic blocks are provided between the interconnected network and the hidden layers of the neural network, between the hidden layers of the two neural networks, and between the hidden layers of the neural network and the summing unit, and the configurable logic blocks include a six-input lookup table (LUT-6) and a trigger. The concept of unweighted neural network can be understood from its literal meaning. In a neural network, we usually abstract the network into a structure consisting of many nodes (neurons) and edges connecting these nodes. If a corresponding weight is assigned to each edge in the network, this type of network is called a weighted network. In contrast, a network without edge weights, that is, a network in which the weight of each edge is regarded as equal (or can be regarded as a network with a weight of 1 for each edge), is called an unweighted network. Furthermore, an unweighted neural network is a neural network model that does not particularly rely on weight adjustment during design and optimization, but focuses more on the optimization of the network structure itself.

[0031] In this embodiment, the six-input lookup table (LUT-6) includes two five-input lookup tables (LUT-5) and a 2:1 multiplexer. The basic principle of the LUT-6 structure is to realize the calculation of the logic function through the lookup table. The lookup table is a data structure that stores the truth table of the logic function, and the corresponding output signal is found by the combination of input signals. The six input signals in the LUT-6 structure can be the output signals of the logic variables or logic gates, and the output signal is calculated according to the truth table in the lookup table. Specifically, when the input state of LUT-6 changes, it will find the corresponding output value in the lookup table according to the current input state. The lookup table stores all possible input states and corresponding output values, which enables the LUT-6 structure to realize any logic function. Once the lookup table returns the correct output value, the LUT-6 structure will output the value to the output end. LUT-5 has 5 inputs and 1 output, and realizes the logic function by means of a lookup table. The lookup table is a data structure that stores the truth table of the logic function. When the state of the input signal is determined, the lookup table will return the corresponding output signal. The five input signals in the LUT-5 structure can be logic variables or output signals of logic gates, and the output signal is calculated based on the truth table in the lookup table.

[0032] In this embodiment, if Figure 3 As shown, the spatiotemporal denoising filter module includes a row storage unit, a column storage unit and a spatiotemporal denoising filter, wherein the row storage unit is used to store the data of the image in the row direction; the column storage unit is used to store the data of the image in the column direction; and the spatiotemporal denoising filter is used to perform multi-directional filtering operations on the image, including filtering in the row direction and the column direction, and filtering across time frames. In the spatiotemporal filtering process, the row storage unit can temporarily store the data of the current row or the previous rows for subsequent filtering operations in the row direction. This is particularly important for processing signals with temporal or spatial continuity (such as video signals) because it allows the filter to use the correlation of adjacent pixels or signal values ​​in the row direction to perform denoising or enhancement processing. In the spatiotemporal filtering process, the column storage unit can also temporarily store the data of the current column or the previous columns for subsequent filtering operations in the column direction. In this way, the filter can use the correlation of adjacent pixels or signal values ​​in the column direction to perform denoising or enhancement processing, thereby further improving the quality of the image or signal. The spatiotemporal denoising filter is a filter that combines spatial and temporal information, and it can perform denoising processing in the spatiotemporal domain of an image or signal. Compared with pure spatial or temporal filters, spatiotemporal denoising filters can more comprehensively utilize the information in images or signals, thereby more effectively removing noise and retaining useful signal components.

[0033] In this embodiment, the spatiotemporal denoising filter uses four row memory groups, and each memory block can store four events. In the spatiotemporal denoising filter, the row memory group is used to store intermediate data during the processing. These data may include image information of the previous frame or previous frames, as well as pixel values ​​of the current frame, etc. By storing this information, the filter can use them to remove noise in subsequent processing. Each memory block can store four events, where "event" refers to some data or information related to image processing. Specifically, these events may represent a certain pixel value in the image, a grayscale change in a certain area, a signal strength at a certain time point, etc. By storing these events, the filter can more effectively analyze and process the noise in the image.

[0034] The present invention can significantly improve the energy efficiency of video analysis. Specifically, by combining the power consumption advantages of hardware neurons and the lightweight design of unweighted neural networks, the computational and power consumption overhead of the processor is greatly reduced, so that it can run on low-power edge devices. The present application has the advantages of real-time processing and high accuracy. Specifically, through the optimized denoising preprocessing module and the automated design optimization engine, efficient real-time video processing can be achieved to ensure the accuracy and robustness of the analysis. The present application has a flexible modular design. Specifically, the integrated design framework has high scalability and flexibility. Users can adjust the network architecture and hardware implementation according to different application requirements, thereby quickly developing video analysis processors adapted to specific scenarios.

[0035] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0036] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. Video processing system based on domestic deep learning processor, characterized by: Including multi-dimensional Bayesian optimization module, unweighted neural network module and spatiotemporal denoising filter module; The multi-dimensional Bayesian optimization module uses multi-dimensional Bayesian optimization to explore and optimize the design space of deep learning processors, covering the architecture, parameters and hardware implementation details of neural networks, combining the energy efficiency advantages of hardware neurons, and through a flexible modular architecture, enabling the processor to achieve an optimal balance in accuracy, power consumption, area and latency; the unweighted neural network module is used to introduce unweighted neural networks in neural network design, using LUT structures to replace traditional weighted connections, thereby reducing the computational overhead and power consumption of the processor, and reducing the complexity of the network and improving the efficiency of data processing through learnable mapping and reduction techniques; the spatiotemporal denoising filter module provides a spatiotemporal denoising filter with a group-connected cache, which filters noise events in the preprocessing of video streams, improves the signal-to-noise ratio of video data, and uses pipeline processing and read-out optimization techniques to achieve real-time denoising while reducing memory usage and power consumption.

2. The video processing system based on a domestic deep learning processor according to claim 1, characterized in that: The multidimensional Bayesian optimization module includes a hardware neuron library, a multidimensional evaluator, and a network evaluator, wherein the hardware neuron library is used to provide efficient computing resources and optimized neural network performance, and support the training and reasoning of large-scale neural networks; the multidimensional evaluator is used to efficiently evaluate the performance of multiple candidate solutions, and support multi-objective optimization, combine with the hardware neuron library to accelerate the evaluation process, provide rich evaluation results, and support adaptive evaluation strategies; the network evaluator is used to evaluate the performance of the neural network model.

3. The video processing system based on the domestic deep learning processor according to claim 2 is characterized in that: The network evaluator includes a network generator, a power estimator, a delay estimator, an area estimator and an accuracy estimator, wherein the network generator is used to generate a corresponding neural network model according to the parameter configuration provided by the Bayesian optimization algorithm; the power estimator is used to evaluate the power consumption of the neural network model when it runs on hardware; the delay estimator is used to evaluate the delay of the neural network model when it runs on hardware; the area estimator is used to evaluate the physical area required for the neural network model when it is implemented in hardware; and the accuracy estimator is used to evaluate the prediction accuracy of the neural network model.

4. The video processing system based on a domestic deep learning processor according to claim 1, characterized in that: The unweighted neural network module includes an interconnected network, two hidden layers of neural networks, a summing unit and a decision unit, wherein a configurable logic block is provided between the interconnected network and the hidden layer of the neural network, between the hidden layers of the two neural networks, and between the hidden layer of the neural network and the summing unit, and the configurable logic block includes a six-input lookup table and a trigger.

5. The video processing system based on the domestic deep learning processor according to claim 4 is characterized in that: The six-input lookup table includes two five-input lookup tables and a 2:1 multiplexer.

6. The video processing system based on a domestic deep learning processor according to claim 1, characterized in that: The spatiotemporal denoising filter module includes a row storage unit, a column storage unit and a spatiotemporal denoising filter, wherein the row storage unit is used to store image data in the row direction; the column storage unit is used to store image data in the column direction; the spatiotemporal denoising filter is used to perform multi-directional filtering operations on the image, including filtering in the row and column directions, and filtering across time frames.

7. The video processing system based on a domestic deep learning processor according to claim 6, characterized in that: The spatiotemporal denoising filter uses four row memory groups, and each memory block can store four events.

Images