Image processing system, method and device based on flow architecture convolutional neural network accelerator, medium and product
By introducing a hierarchical pipeline module into the stream architecture convolutional neural network accelerator and adjusting the parallelism of the input and output channels, the problem of low image processing efficiency caused by the delay of the stream architecture convolutional neural network accelerator is solved, and more efficient image processing is achieved.
Patent Information
- Application Number
- CN202510551859.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-29
- Publication Date
- 2025-08-01
AI Technical Summary
When the stream architecture convolutional neural network accelerator processes streaming data, the calculation amount of each layer varies greatly, resulting in increased delay and low image processing efficiency.
An image processing system based on a stream architecture convolutional neural network accelerator is designed. By introducing multiple connected hierarchical pipeline modules, each layer of computing tasks is processed in parallel, and the parallelism of the input channel and output channel is adjusted to control the running time by using read in, convolution, pooling and write back to the kernel.
The running time of different layers of the model is achieved consistent, improving the balance and computing efficiency of the overall pipeline, and reducing resource consumption.
Smart Images
Figure CN120409580A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular, to an image processing system, method, device, medium, and product based on a stream architecture convolutional neural network accelerator. Background Art
[0002] A Convolutional Neural Network (CNN) is an emerging deep learning architecture that is widely used in the field of image processing. Traditional computing architectures usually rely on high-performance cloud servers for computing and transmit the processing results to terminal devices. However, this mode is limited by network bandwidth, has a high latency, and there are privacy and security issues. Especially for streaming data, its characteristic is to continuously input in the form of independent frames and requires real-time processing within strict time limits. Therefore, the acquisition and processing of streaming data usually occur on the edge side to avoid network latency affecting the real-time computing efficiency.
[0003] The traditional computing mode requires the entire CNN model to be loaded into the computing platform at one time for inference and simultaneously process the entire image, which has a large latency problem in real-time computing scenarios. To meet the real-time requirements of streaming data processing, a stream architecture CNN accelerator is often used. In this architecture, streaming data passes through each layer of the CNN in a pipeline manner at each time step for calculation. Each layer of the CNN can be regarded as a stage in the pipeline, where the output of each layer is directly used as the input of the next layer, so as to parallelly process the input frames received at different times and achieve efficient data processing with limited hardware resources.
[0004] Currently, stream architecture CNN accelerators mainly use different hardware modules to perform operations on different layers, and perform convolution operations on the rows or columns of the entire convolution kernel. However, as the depth of the CNN model increases, the computational amounts of different layers of the model vary greatly, and it is difficult to balance each stage of the pipeline, resulting in an increase in the overall latency and low image processing efficiency. Summary of the Invention
[0005] The purpose of this application is to provide an image processing system, method, device, medium, and product based on a stream architecture convolutional neural network accelerator, which solves the problem of low image processing efficiency caused by the latency of the stream architecture convolutional neural network accelerator.
[0006] To achieve the above objective, this application provides the following solutions:
[0007] In a first aspect, this application provides an image processing system based on a stream architecture convolutional neural network accelerator, including: a stream architecture convolutional neural network accelerator;
[0008] The flow architecture convolutional neural network accelerator includes multiple cascaded hierarchical pipeline modules for parallelly processing the computing tasks of each layer of the convolutional neural network;
[0009] The hierarchical pipeline module includes an input kernel, a convolutional kernel, a pooling kernel, and a write-back kernel; the input kernel, the convolutional kernel, the pooling kernel, and the write-back kernel are connected in sequence; the convolutional kernel includes multiple computing units; the computing unit includes an accumulator, an activation function, and multiple multiplier-accumulators, and the computing unit is used to perform operations on the data within the hierarchical pipeline module to determine the output feature values; the data within the hierarchical pipeline module includes input image feature values, weights, and output feature values; the hierarchical pipeline module introduces input channel parallelism and output channel parallelism to control the running time of the current layer.
[0010] In one embodiment, the hierarchical pipeline module specifically includes:
[0011] The input kernel is used to read data from the shared memory;
[0012] The convolutional kernel is used to perform a convolutional operation on the read data to determine the output feature values after the convolutional operation of the current layer;
[0013] The pooling kernel is used to perform a pooling operation on the output feature values of the current layer to determine the output feature values after the pooling operation;
[0014] The write-back kernel is used to write the output feature values after the pooling operation back to the shared memory.
[0015] In one embodiment, the convolutional kernel specifically includes:
[0016] A running time judgment unit for judging whether the current running time is greater than the target running time to obtain a judgment result; the running time includes the current running time and the target running time;
[0017] A parallelism increasing unit for preferentially increasing the parallelism of the input channels and reducing the data processing time of the current flow architecture convolutional neural network accelerator when the judgment result is yes;
[0018] A parallelism decreasing unit for reducing the parallelism of the output channels and increasing the data processing time of the current flow architecture convolutional neural network accelerator when the judgment result is no.
[0019] In one embodiment, the shared memory is used to store the output feature values of the current layer calculated by each hierarchical pipeline module and transmit the output feature values calculated by the current layer hierarchical pipeline module to the next layer hierarchical pipeline module.
[0020] In a second aspect, the present application provides an image processing method based on a stream architecture convolutional neural network accelerator, including:
[0021] Based on a hierarchical pipeline module, adjust the parallelism of input channels and the parallelism of output channels to control the running time;
[0022] Based on the running time, process the images in the data stream according to the parallelism of input channels and the parallelism of output channels to determine output feature values.
[0023] In one embodiment, use to determine the intermediate result of the current layer of the hierarchical pipeline module; where f in (i) is the input feature value of the convolutional kernel; w(i,j) is the weight of the convolutional kernel; i ∈ [1, m], m is the parallelism of input channels; j ∈ [1, n], n is the parallelism of output channels; f out (j,k) is the intermediate result;
[0024] Use to sum up the intermediate results of the current layer of the hierarchical pipeline module to determine the summation result; where k is the number of times of the input feature value, ch in is the number of input channels, m is the parallelism of input channels; f out (j) is the output feature value;
[0025] Based on the activation function and the running time, determine the output feature value according to the summation result.
[0026] In one embodiment, based on the hierarchical pipeline module, use to determine the parallelism of the output channels of the current layer to adjust the parallelism of the output channels of the current layer; where W temp is the parallelism of the output channels of the current layer; T i is the running time of the current layer; T target is the target running time of the current layer; W i is the initial value of the parallelism of the output channels of the current layer;
[0027] Use to determine the parallelism of the input channels of the current layer to adjust the parallelism of the input channels of the current layer; where D temp is the parallelism of the input channels; D i is the initial value of the parallelism of the input channels of the current layer;
[0028] Based on the adjusted parallelism of the output channels of the current layer and the parallelism of the input channels of the current layer, use to determine the adjusted running time of the current layer to control the running time; where T tempThe adjusted running time for the current layer.
[0029] In a third aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the steps of the image processing method of the flow architecture convolutional neural network accelerator described in any one of the above.
[0030] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image processing method of the flow architecture convolutional neural network accelerator described in any one of the above.
[0031] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the image processing method of the flow architecture convolutional neural network accelerator described in any one of the above.
[0032] According to the specific embodiments provided by the present application, the following technical effects are disclosed in the present application:
[0033] The present application provides an image processing system, method, device, medium, and product based on a flow architecture convolutional neural network accelerator. The flow architecture convolutional neural network accelerator includes a plurality of cascaded hierarchical pipeline modules for parallel processing of the calculation tasks of each layer of the convolutional neural network; the convolutional kernel is composed of a plurality of computing units for performing operations on the data within the hierarchical pipeline module to determine the output feature values; the hierarchical pipeline module introduces input channel parallelism and output channel parallelism to control the running time of this layer. Existing flow architecture convolutional neural network accelerators need to use different modules to perform operations on the hierarchical pipeline modules of each layer, and there is a large difference in the amount of calculation between layers, making it difficult to balance each pipeline. The present application can make the running times of different layers of the model tend to be consistent by preferentially adjusting the input channel parallelism, thereby improving the balance of the overall pipeline for image processing and reducing resource consumption. Description of the Drawings
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.
[0035] Figure 1 It is a schematic structural diagram of the model hierarchical pipeline module in the flow architecture CNN accelerator provided by an embodiment of the present application;
[0036] Figure 2Spatial-temporal graph schematic of data segment calculation in the model-level pipeline module in the flow architecture CNN accelerator provided by an embodiment of the present application;
[0037] Figure 3 Schematic structural diagram of the model inter-layer pipeline in the flow architecture CNN accelerator provided by an embodiment of the present application;
[0038] Figure 4 Schematic flowchart of an image processing method based on a flow architecture convolutional neural network accelerator provided by an embodiment of the present application;
[0039] Figure 5 Schematic flowchart of adjusting the model channel parallelism in the flow architecture CNN accelerator provided by an embodiment of the present application;
[0040] Figure 6 Schematic structural diagram of the model calculation unit in the flow architecture CNN accelerator provided by an embodiment of the present application;
[0041] Figure 7 Schematic structural diagram of the convolutional kernel of the model-level pipeline module in the flow architecture CNN accelerator provided by an embodiment of the present application;
[0042] Figure 8 Spatial-temporal graph of batch data calculation in the model inter-layer pipeline in the flow architecture CNN accelerator provided by an embodiment of the application;
[0043] Figure 9 Schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners
[0044] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
[0045] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.
[0046] The present application provides a data processing system for a stream architecture convolutional neural network accelerator, which includes designing independent hierarchical pipeline modules for each layer of the CNN model. The hierarchical pipeline module is composed of a read kernel, a convolution kernel, a pooling kernel, and a write-back kernel. Among them, the convolution kernel is composed of multiple computing units; the computing unit is composed of an accumulator, an activation function, and multiple multiplier-accumulators; a layer-level pipeline design is introduced for the overall CNN model, and the overall architecture is composed of hierarchical pipeline modules of each layer and a shared memory; based on the two dimensions of the input channels and output channels of each layer of the CNN model, two variables, namely the input channel parallelism and the output channel parallelism, are introduced. By preferentially adjusting the input channel parallelism, the operation time of different levels of the model is made more uniform, and the resource occupation of the model is smaller.
[0047] An embodiment of the present application provides an image processing system based on a stream architecture convolutional neural network accelerator, and the specific content is described as follows.
[0048] The stream architecture convolutional neural network accelerator includes multiple cascaded hierarchical pipeline modules, and the hierarchical pipeline module is used to parallelly process the calculation tasks of each layer of the convolutional neural network.
[0049] The hierarchical pipeline module includes a read kernel, a convolution kernel, a pooling kernel, and a write-back kernel; the read kernel, the convolution kernel, the pooling kernel, and the write-back kernel are connected in sequence; the convolution kernel includes multiple computing units; the computing unit includes an accumulator, an activation function, and multiple multiplier-accumulators, and the computing unit is used to perform operations on the data in the hierarchical pipeline module to determine the output feature value; the data in the hierarchical pipeline module includes input image feature values, weights, and output feature values; the hierarchical pipeline module introduces input channel parallelism and output channel parallelism to control the running time of the current layer.
[0050] As Figure 1 shown, each hierarchical pipeline module 100 includes four kernels, namely a read kernel 101, a convolution kernel 102, a pooling kernel 103, and a write-back kernel 104.
[0051] Among them, the read kernel 101 is responsible for reading data from the shared memory and allocating data for subsequent convolution operations; the convolution kernel 102 receives the data transmitted by the read kernel 101 through the data flow pipeline and is responsible for performing convolution operations on the data; the pooling kernel 103 receives the data transmitted by the convolution kernel 102 through the data flow pipeline and is responsible for performing pooling operations on the data; the write-back kernel 104 receives the data transmitted by the pooling kernel 103 through the data flow pipeline and is responsible for writing the data back to the shared memory. The data completes the calculation of batch continuous tasks in the hierarchical pipeline module 100 in a pipeline manner.
[0052] Further, in an exemplary embodiment, the hierarchical pipeline module specifically includes:
[0053] The read-in kernel is used to read data from the shared memory.
[0054] The convolution kernel is used to perform a convolution operation on the read data to determine the output feature values of the current layer.
[0055] The pooling kernel is used to perform a pooling operation on the output feature values of the current layer to determine the output feature values after the pooling operation.
[0056] The write-back kernel is used to write the output feature values after the pooling operation back to the shared memory.
[0057] As Figure 2 shown, the spatio-temporal graph of the data segment calculated in a pipeline manner within the model layer in the stream architecture CNN accelerator provided by the present application. In this embodiment, Figure 2 the four kernels in respectively correspond to the read-in kernel 101, the convolution kernel 102, the pooling kernel 103, and the write-back kernel 104. The cycle time of the spatio-temporal graph is divided into the first cycle, the second cycle,.... Among them, the data segment refers to the segment into which each picture is divided according to the data volume collected or transmitted by the system per unit time. Several segments enter the hierarchical pipeline module in sequence for processing.
[0058] In the first cycle, the read-in kernel 101 reads data segment one from the shared memory and performs data processing and distribution. In the second cycle, when the convolution kernel 102 performs a convolution operation on data segment one, the read-in kernel 101 reads data segment two from the shared memory and performs data processing and distribution, and the two are parallel. In the third cycle, when the pooling kernel 103 performs a pooling operation on data segment one, the convolution kernel 102 performs a convolution operation on data segment two, and the two are parallel. In the fourth cycle, when the write-out kernel 104 writes data segment one back to the shared memory, the pooling kernel 103 performs a data pooling operation on data segment two, and the two are parallel. In the fifth cycle, the write-back kernel 104 writes data segment two back to the shared memory.
[0059] As Figure 3 shown, in this embodiment, the inter-layer pipeline architecture includes l hierarchical pipeline modules 100 and a shared memory 105.
[0060] Among them, the input kernel 101 and the write-back kernel 104 of each hierarchical pipeline module 100 are connected to the shared memory 105, and the processing of the data stream is completed in the form of an inter-layer pipeline through the data transmission pipeline. The first hierarchical pipeline module 100 is responsible for receiving input data, loading corresponding weights and bias parameters, performing calculations, and transmitting the calculated data to the shared memory 105 through the data transmission channel. The calculation result is then transmitted to the second hierarchical pipeline module 100 through the shared memory 105, and this process is repeated until the l-th hierarchical pipeline module 100 finishes calculating the data and writes the final calculation result back to the shared memory 105.
[0061] Further, in an exemplary embodiment, the convolutional kernel specifically includes:
[0062] A running time judgment unit for judging whether the current running time is greater than the target running time to obtain a judgment result; the running time includes the current running time and the target running time.
[0063] A parallelism increasing unit for preferentially increasing the parallelism of the input channels when the judgment result is yes, so as to reduce the data processing time of the current stream architecture convolutional neural network accelerator.
[0064] A parallelism decreasing unit for decreasing the parallelism of the output channels when the judgment result is no, so as to increase the data processing time of the current stream architecture convolutional neural network accelerator.
[0065] Further, in an exemplary embodiment, the shared memory is used to store the output feature values of the current layer calculated by each hierarchical pipeline module, and transmit the output feature values calculated by the current layer hierarchical pipeline module to the next layer hierarchical pipeline module.
[0066] As Figure 4 shown, the embodiments of the present application provide an image processing method based on a stream architecture convolutional neural network accelerator, and the specific content is as follows.
[0067] The image processing method of the stream architecture convolutional neural network accelerator includes:
[0068] Step 101: Based on the hierarchical pipeline module, adjust the parallelism of the input channels and the parallelism of the output channels to control the running time.
[0069] Step 102: Based on the running time, process the images in the data stream according to the parallelism of the input channels and the parallelism of the output channels to determine the output feature values.
[0070] Further, in an exemplary embodiment, step 101 can be replaced by the following steps.
[0071] Step 1011: Based on the hierarchical pipeline module, use to determine the parallelism of the output channels of the current layer, so as to adjust the parallelism of the output channels of the current layer; where W temp is the parallelism of the output channels of the current layer; T i is the running time of the current layer; T target is the target running time of the current layer; W i is the initial value of the parallelism of the output channels of the current layer.
[0072] Step 1012: Use to determine the parallelism of the input channels of the current layer, so as to adjust the parallelism of the input channels of the current layer; where D temp is the parallelism of the input channels; D i is the initial value of the parallelism of the input channels of the current layer.
[0073] Step 1013: Based on the adjusted parallelism of the output channels of the current layer and the parallelism of the input channels of the current layer, use to determine the adjusted running time of the current layer, so as to control the running time; where T temp is the adjusted running time of the current layer.
[0074] Further, in an exemplary embodiment, step 102 can be replaced by the following steps.
[0075] Step 1021: Use to determine the intermediate result of the current layer of the hierarchical pipeline module; where f in (i) is the input eigenvalue of the convolution kernel; w(i,j) is the weight of the convolution kernel; i ∈ [1, m], m is the parallelism of the input channels; j ∈ [1, n], n is the parallelism of the output channels; f out (j,k) is the intermediate result.
[0076] Step 1022: Use to sum the intermediate results of the current layer of the hierarchical pipeline module to determine the sum result; where k is the number of times of the input eigenvalue, ch in is the number of input channels, m is the parallelism of the input channels; f out (j) is the output eigenvalue.
[0077] In this embodiment, the convolution kernel 102 is composed of multiple computing units 106, and the input eigenvalue and weight of the convolution kernel 102 are respectively: f in (i) and w(i,j). Where i ∈ [1, m], m is the parallelism of the input channels; j ∈ [1, n], n is the parallelism of the output channels, which determines the number of computing units 106 in the convolution kernel 102.
[0078] Step 1023: Determine the output eigenvalue based on the activation function and the running time according to the summation result.
[0079] As Figure 5 shown, the flowchart of adjusting the model channel parallelism in the stream architecture CNN accelerator provided by this application. This embodiment includes the following steps:
[0080] S11: Obtain the number of input channels and the number of output channels of the current layer of the model.
[0081] Specifically, obtain the number of input channels C in and the number of output channels C out of the current layer of the CNN model.
[0082] S12: When the model is first run on the hardware platform, obtain the initial values of the running time, the input channel parallelism, and the output channel parallelism of the current layer.
[0083] Specifically, when the CNN model is first run on the hardware platform, obtain the running time of the current layer as T i , the target running time of the current layer as T target , the initial value of the input channel parallelism of the current layer as D i , and the initial value of the output channel parallelism of the current layer as W i . Among them, the hardware platform refers to the physical device or component that can run this architecture.
[0084] S13: Determine whether the obtained running time of the current layer is greater than the target running time of the current layer.
[0085] Specifically, if the obtained running time of the current layer is less than or equal to the target running time of the current layer, continue with step S14; otherwise, continue with step S15.
[0086] S14: Prioritize changing the output channel parallelism to control the inter-layer running time.
[0087] As Figure 6 and Figure 7 shown, specifically, if T i ≤T target , the size of the output channel parallelism in the convolutional kernel 102 has a greater impact on the model's computing resources. Therefore, in order to reduce the computing resources, it is necessary to first reduce the output channel parallelism to increase the running time of the current layer to reach the target running time of the current layer. The following function expression is established:
[0088]
[0089] Among them, W tempFor the output channel parallelism of the obtained current layer, if C out is divisible by W temp , then W target = W temp . Otherwise, find the value of W out that is divisible by C temp and closest to the value of W target , and the target input channel parallelism is equal to the initial value of the input channel parallelism, i.e., D target = D i .
[0090] S15: Based on the input channel number of the current layer of the model, preferentially change the input channel parallelism to control the inter-layer running time.
[0091] Specifically, when T i > T target , the influence of the input channel parallelism size of the computing unit 106 in the convolution kernel 102 on the model computing resources is smaller. Therefore, in order to reduce the computing resources, preferentially increase the input channel parallelism to reduce the running time of the current layer and reach the target running time of the current layer. The following function expression is established:
[0092]
[0093] Among them, D temp in the function expression is the input channel parallelism of the current layer. If D temp ≤ C in and C in is divisible by D temp , then D target = D temp , and the target output channel parallelism is equal to the initial value of the output channel parallelism, i.e., W target = W i . If D temp ≤ C in but C in is not divisible by D temp , then find the value of D in that is divisible by C temp and closest to the value of D target , and the target output channel parallelism is equal to the initial value of the output channel parallelism, i.e., W target = W i .
[0094] S16: Based on the output channel number and input channel parallelism of the current layer of the model, change the output channel parallelism to further control the inter-layer running time.
[0095] Specifically, if D temp > C in , then let Dtemp = C in , and establish the following function expression to calculate the target output channel parallelism required to reach the target running time of the current layer.
[0096]
[0097] Among them, T temp is the running time currently reached by the current layer. Therefore, T i = T temp . According to the function expression in step S14, the output channel parallelism of the current layer is obtained as W temp . If C out can be divided evenly by W temp , then W target = W temp . Otherwise, find the value that can be divided evenly by C out and is closest in value to W temp as W target . The target input channel parallelism D target = D temp .
[0098] S17: Obtain the input channel parallelism and output channel parallelism suitable for each layer of the model to reach the target running time of the current layer.
[0099] As Figure 8 shown, it is the space-time diagram of a large amount of data calculated in a pipelined manner between model layers in the stream architecture CNN accelerator provided by this application. In this embodiment, there are a total of l hierarchical pipeline modules 100. The cycle time of the space-time diagram is divided into the first cycle, the second cycle,.... Among them, the cycle of this space-time diagram is the time required for one hierarchical pipeline module to process one picture (instead of a segment).
[0100] In the first cycle, Picture 1 is calculated in the first hierarchical pipeline module; in the second cycle, when Picture 1 is being calculated in the second hierarchical pipeline module, since the first hierarchical pipeline module is idle, Picture 2 can immediately enter for calculation. Therefore, the first hierarchical pipeline module and the second hierarchical pipeline module can perform data calculations in parallel at the same time; this is repeated until all pictures are calculated in the l hierarchical pipeline modules.
[0101] The computing unit is composed of multiple multiplier modules 107, accumulator modules 108, and activation unit modules 109.
[0102] Among them, the number of multipliers is equal to the input channel parallelism. Combining Figure 8 , the input of each computing unit 106 includes eigenvalue f in (1), f in (2)…fin (m) and the weights w(1,j), w(2,j) … w(m,j) of input channel j. Each computing unit 106 completes a specific eigenvalue f in (i) and the multiplication operation of the weights w(i,j). After all the product results are processed by the accumulator 108 and the activation function 109, the output feature map result of each computing unit 106 is f out (j).
[0103] Compared with the prior art, the above method has the advantages of high computing efficiency and small resource occupancy.
[0104] The above method has high computing efficiency in implementation. The methods designed in some prior arts do not segment the CNN model in a fine-grained manner and only perform subdivision within the model layer. Therefore, the models established by their methods can only be parallel within the layer and cannot achieve an inter-layer pipeline. The method provided in this application, based on segmenting within the model layer, segments between the model layers into hierarchical modules equivalent to the number of model layers, improving the parallelism of the model and enabling the model to achieve a true pipeline.
[0105] The above method has a smaller resource occupancy in implementation. The methods designed in some prior arts have too high a resource occupancy, with a large difference in the operation complexity of each layer of the model, resulting in mismatched speeds of each layer and thus affecting the latency of the overall pipeline. The method provided in this application adjusts the parallel parameters of the data in two dimensions of the model, and preferentially adjusts the parallel parameters in the input channel dimension of the model, so that the model has a smaller resource occupancy and the operation time of all layers is uniform, optimizing the pipeline cycle of the model and making the computing efficiency of the model higher.
[0106] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 9As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for image processing of the stream architecture convolutional neural network accelerator. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes image processing of the stream architecture convolutional neural network accelerator.
[0107] Those skilled in the art can understand that Figure 9 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.
[0108] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.
[0109] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.
[0110] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.
[0111] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.
[0112] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0113] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.
[0114] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.
[0115] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; at the same time, for those of ordinary skill in the art, according to the ideas of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.
Claims
1. An image processing system based on a stream architecture convolutional neural network accelerator, characterized in that, The image processing system of the stream architecture convolutional neural network accelerator includes: a stream architecture convolutional neural network accelerator; The stream architecture convolutional neural network accelerator includes a plurality of serially connected hierarchical pipeline modules, and the hierarchical pipeline modules are used to parallelly process the calculation tasks of each layer of the convolutional neural network; The hierarchical pipeline module includes an input kernel, a convolutional kernel, a pooling kernel, and a write-back kernel; the input kernel, the convolutional kernel, the pooling kernel, and the write-back kernel are connected in sequence; the convolutional kernel includes a plurality of computing units; the computing unit includes an accumulator, an activation function, and a plurality of multiplier-accumulators, and the computing unit is used to perform operations on the data in the hierarchical pipeline module to determine the output feature value; the data in the hierarchical pipeline module includes input image feature values, weights, and output feature values; the hierarchical pipeline module introduces input channel parallelism and output channel parallelism to control the running time of the current layer.
2. The image processing system of the stream architecture convolutional neural network accelerator according to claim 1, characterized in that, The hierarchical pipeline module specifically includes: The input kernel is used to read data from the shared memory; The convolutional kernel is used to perform a convolution operation on the read data to determine the output feature value after the convolution operation of the current layer; The pooling kernel is used to perform a pooling operation on the output feature value of the current layer to determine the output feature value after the pooling operation; The write-back kernel is used to write the output feature value after the pooling operation back to the shared memory.
3. The image processing system of the stream architecture convolutional neural network accelerator according to claim 1, wherein The convolutional kernel specifically includes: A running time judgment unit, which is used to judge whether the current running time is greater than the target running time to obtain a judgment result; the running time includes the current running time and the target running time; A parallelism increasing unit, which is used to preferentially increase the parallelism of the input channels and reduce the data processing time of the current stream architecture convolutional neural network accelerator when the judgment result is yes; A parallelism decreasing unit, which is used to reduce the parallelism of the output channels and increase the data processing time of the current stream architecture convolutional neural network accelerator when the judgment result is no.
4. The image processing system of the stream architecture convolutional neural network accelerator according to claim 2, characterized in that, The shared memory is used to store the output feature value of the current layer calculated by each hierarchical pipeline module and transmit the output feature value calculated by the hierarchical pipeline module of the current layer to the next hierarchical pipeline module.
5. An image processing method based on a stream architecture convolutional neural network accelerator, characterized in that, The image processing method of the stream architecture convolutional neural network accelerator applies the image processing system of the stream architecture convolutional neural network accelerator according to any one of claims 1-4, and the image processing method of the stream architecture convolutional neural network accelerator includes: Based on the hierarchical pipeline module, adjust the parallelism of the input channels and the parallelism of the output channels to control the running time; Based on the running time, process the images in the data stream according to the parallelism of the input channels and the parallelism of the output channels to determine the output feature value.
6. The image processing method of the stream architecture convolutional neural network accelerator according to claim 5, wherein Based on the running time, process the images in the data stream according to the parallelism of the input channels and the parallelism of the output channels to determine the output feature value, specifically including: Utilize to determine the intermediate result of the current layer of the hierarchical pipeline module; where f in (i) is the input eigenvalue of the convolution kernel; w(i,j) is the weight of the convolution kernel; i ∈ [1, m], m is the parallelism of the input channels; j ∈ [1, n], n is the parallelism of the output channels; f out (j,k) is the intermediate result; Utilize Sum the intermediate results of the current layer of the hierarchical pipeline module to determine the summation result; where k is the number of times of the input feature value, ch in Number of input channels; f out (j) is the output feature value; Based on the activation function and the running time, determine the output feature value according to the summation result.
7. The image processing method of the stream architecture convolutional neural network accelerator according to claim 5, characterized in that Based on the hierarchical pipeline module, adjust the parallelism of the input channels and the parallelism of the output channels to control the running time, specifically including: Based on the hierarchical pipeline module, using to determine the parallelism of the output channels of the current layer, so as to adjust the parallelism of the output channels of the current layer; where W temp is the parallelism of the output channels of the current layer; T i is the running time of the current layer; T target is the target running time of the current layer; W i is the initial value of the parallelism of the output channels of the current layer. Utilize Determine the parallelism of the input channels of the current layer to adjust the parallelism of the input channels of the current layer; where D temp is the parallelism of the input channels; D i is the initial value of the parallelism of the input channels of the current layer; Based on the parallelism of the output channels of the current layer after adjustment and the parallelism of the input channels of the current layer, use to determine the adjusted running time of the current layer to control the running time; where T temp is the adjusted running time of the current layer.
8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method of the stream architecture convolutional neural network accelerator according to any one of claims 5-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method of the stream architecture convolutional neural network accelerator according to any one of claims 5-7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image processing method of the stream architecture convolutional neural network accelerator according to any one of claims 5-7.