Joint computing system and method, storage medium and electronic device

Abstract

The present disclosure provides a joint computing system and method, storage medium and device. The system includes: a first processing module, configured to perform a first processing on input data in a block manner to generate a plurality of first block data respectively, and output a first data reading notification corresponding to each first block data; a neural network accelerator module, configured to read a plurality of first block data from the first processing module in response to the first data reading notification, perform a second processing on each first block data using a neural network model to generate a plurality of second block data, and output a second data reading notification corresponding to each second block data; a second processing module, configured to read a plurality of second block data from the neural network accelerator module in response to the second data reading notification, and perform a third processing on each second block data to generate a plurality of third block data. The joint computing system can reduce the latency in the system and the system's running memory.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a joint computing system and method, a storage medium, and an electronic device. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of intelligent computing systems has become more and more extensive. When processing the image data frames collected by the camera, the image data frames usually pass through the image processing module in a serial manner, which will cause a large delay in the entire intelligent computing system.

[0003] Public Content

[0004] One of the purposes of the present disclosure is to provide a joint computing system and method, a storage medium and an electronic device, which are used to solve the problem of large delay in intelligent computing systems.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present disclosure provides a joint computing system. The joint computing system includes: a first processing module, configured to perform a first processing on input data in a block manner to generate a plurality of first block data respectively, and output a first data reading notification corresponding to each of the first block data; a neural network accelerator module, coupled to the first processing module, configured to read a plurality of the first block data from the first processing module in response to the first data reading notification, perform a second processing on each of the first block data using a neural network model to generate a plurality of second block data respectively, and output a second data reading notification corresponding to each of the second block data; and a second processing module, coupled to the neural network accelerator module, configured to read a plurality of the second block data from the neural network accelerator module in response to the second data reading notification, and perform a third processing on each of the second block data to generate a plurality of third block data respectively.

[0006] In an embodiment of the first aspect, the first processing module is configured to perform the first processing on the input data in a horizontal blocking manner and / or a vertical blocking manner.

[0007] In an embodiment of the first aspect, the neural network model is divided into multiple sub-network models that respectively match multiple first block data, and the neural network accelerator module is configured to use each of the sub-network models to process the corresponding first block data.

[0008] In an embodiment of the first aspect, the neural network model includes a convolution operation layer and an activation operation layer, and the neural network accelerator module is configured to perform convolution operations on multiple first block data based on multiple convolution operation layers to generate multiple intermediate block data, and to perform activation operations on multiple intermediate block data based on multiple activation operation layers to generate multiple second block data.

[0009] In an embodiment of the first aspect, the neural network accelerator module is configured to start the second processing of the predetermined amount of data in response to reading the predetermined amount of data in the first block data.

[0010] In an embodiment of the first aspect, the first processing module includes a pre-processing accelerator, and the second processing module includes a post-processing accelerator.

[0011] In an embodiment of the first aspect, the first processing module includes an image processing accelerator, and the second processing module includes a graphics rendering processor.

[0012] In an embodiment of the first aspect, the first processing module, the neural network accelerator module and the second processing module are configured to perform the first processing, the second processing and the third processing respectively in parallel.

[0013] In an embodiment of the first aspect, the first processing module is configured to generate a first group of the first block data in a first operating cycle, the neural network accelerator module is configured to read the first group of the first block data and generate a first group of the second block data in a second operating cycle, and the first processing module is configured to generate a second group of the first block data in the second operating cycle, the second processing module is configured to read the first group of the second block data and generate a first group of the third block data in a third operating cycle, the neural network accelerator module is configured to read the second group of the first block data and generate a second group of the second block data in the third operating cycle, and the first processing module is configured to generate a third group of the first block data in the third operating cycle.

[0014] In an embodiment of the first aspect, the second processing module is configured to read a second group of the second block data and generate a second group of the third block data in a fourth operating cycle, the neural network accelerator module is configured to read a third group of the first block data and generate a third group of the second block data in the fourth operating cycle, and the first processing module is configured to generate a fourth group of the first block data in the fourth operating cycle.

[0015] In an embodiment of the first aspect, the first processing module is further configured to obtain the size, number and / or interval of the blocks according to the received instructions.

[0016] In an embodiment of the first aspect, the first processing module is configured to output the first data reading notification in the form of an interrupt when the first processing is completed, and / or the neural network accelerator module is configured to output the second data reading notification in the form of an interrupt when the second processing is completed.

[0017] The second aspect of the present disclosure provides a joint computing method. The joint computing method includes: performing a first processing on input data in a block manner to generate a plurality of first block data respectively, and outputting a first data reading notification corresponding to each of the first block data; reading a plurality of the first block data respectively in response to the first data reading notification, performing a second processing on each of the first block data using a neural network model to generate a plurality of second block data respectively, and outputting a second data reading notification corresponding to each of the second block data; and reading a plurality of the second block data respectively in response to the second data reading notification, and performing a third processing on each of the second block data to generate a plurality of third block data respectively.

[0018] A third aspect of the present disclosure provides a computer-readable storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the joint computing method according to the second aspect of the present disclosure is implemented.

[0019] A fourth aspect of the present disclosure provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to call the computer program to execute the joint computing method according to the second aspect of the present disclosure.

[0020] As described above, the joint computing system described in one or more embodiments of the present disclosure can process input data in blocks and in parallel, which is beneficial to reducing the latency in the system and the operating memory of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Shown is a schematic diagram of the structure of the joint computing system described in the present disclosure in a specific embodiment.

[0022] Figure 2 Shown is a schematic diagram of a neural network sub-model in a specific embodiment of the joint computing system described in the present disclosure.

[0023] Figure 3 Shown is an example diagram of a neural network model in a specific embodiment of the joint computing system described in the present disclosure.

[0024] Figure 4 Shown is a schematic diagram of interrupt triggering of the joint computing system described in the present disclosure in a specific embodiment.

[0025] Figure 5 Shown is a schematic diagram of the structure of the joint computing system described in the present disclosure in a specific embodiment.

[0026] Figure 6 Shown is a flow chart of the joint calculation method described in the present disclosure in a specific embodiment.

[0027] Figure 7 Shown is a schematic diagram of the structure of the electronic device described in the present disclosure in a specific embodiment. DETAILED DESCRIPTION

[0028] The following is an explanation of the embodiments of the present disclosure by specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0029] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure, and the illustrations only show the components related to the present disclosure rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout type may also be more complex. In addition, in this article, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0030] Existing intelligent computing systems include a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), a camera, an image processing accelerator (IPU), and a neural network accelerator (NPU). The CPU loads the initialization and intelligent model data stored in the ROM to initialize various system devices. The CPU controls the camera to collect external image data frames, and the data frames are stored in the RAM for sharing. The data frames are passed to the IPU and undergo image processing, format conversion and other operations. The data frames passing through the IPU are passed to the NPU to perform intelligent application operations and output relevant intelligent results. The GPU draws the relevant intelligent results and renders them to the display. In this intelligent computing system, from camera input to display, the data frames are usually executed serially frame by frame when passing through the IPU and NPU. Therefore, the time consumed by the IPU and NPU to process the image frames will be superimposed, resulting in a large delay in the entire intelligent system.

[0031] In order to solve at least the above technical problems, the present disclosure proposes an improved solution for joint computing, in which the NPU is jointly calculated with other modules. The NPU mainly performs accelerated calculations based on the neural network model, while the other modules are one or more hardware processing modules, which are coupled with the NPU in front or behind, including hardware modules such as image processing accelerator (IPU), graphics rendering processor (GPU), and video output unit (VOP). Taking the IPU as an example of other modules in the system, the IPU processes the image in blocks. When the IPU processes a block, a special interrupt or signal will be generated to notify the NPU to prefetch data. At the same time, the IPU executes the next block. When the data prefetch reaches the computable condition, the NPU starts the calculation, and repeats this process, so that the original serial pipeline is transformed into a parallel pipeline.

[0032] According to the scheme disclosed in the present invention, improvements are made through the design of hardware and software to support the execution mode of block division, and the block division includes horizontal or vertical block division, and its size, number, interval, etc. can be configured through software. After the IPU executes each data block, an interrupt or signal is immediately triggered to quickly notify the NPU, and the NPU can get the image frame data block that has been partially processed by the IPU in advance for processing. At the same time, the IPU then executes the data task of the next block. In this way, by improving the processing methods of the hardware and software of the IPU and NPU, the processing of each data block in each image frame is carried out in parallel, reducing the system delay, thereby improving the response of the intelligent system.

[0033] Hereinafter, specific embodiments of the present disclosure will be described through exemplary embodiments with reference to the accompanying drawings.

[0034] Figure 1 is a block diagram showing a joint computing system according to an embodiment of the present disclosure. Figure 1As shown, the joint computing system 1 includes a first processing module 11, a neural network accelerator module 12 and a second processing module 13.

[0035] The first processing module 11 is configured to perform first processing on input data in a block manner to generate a plurality of first block data respectively, and output a first data reading notification corresponding to each of the first block data.

[0036] The neural network accelerator module 12 is coupled to the first processing module 11, and is configured to read multiple first block data respectively from the first processing module 11 in response to the first data reading notification, perform second processing on each of the first block data using a neural network model to generate multiple second block data respectively, and output second data reading notifications corresponding to each of the second block data.

[0037] The third processing module 13 is coupled to the neural network accelerator module 12, and is configured to read multiple second block data from the neural network accelerator module 12 respectively in response to the second data reading notification, and perform third processing on each of the second block data to generate multiple third block data respectively.

[0038] Optionally, the first processing module 11 is configured to perform the first processing on the input data in a horizontal and / or vertical manner, wherein horizontal blocking refers to blocking in the width direction of the image data frame, and vertical blocking refers to blocking in the height direction of the image data frame.

[0039] Optionally, the neural network model is divided into a plurality of sub-network models that match the plurality of first block data respectively, and the neural network accelerator module 12 is configured to use each of the sub-network models to process the corresponding first block data. Specifically, the neural network model is a directed acyclic graph structure composed of a plurality of processing layers, and each layer thereof needs to be segmented when the neural network is segmented. For example, a single convolutional layer can be recursively or reversely deduced according to the input or output blocks to obtain a plurality of sub-convolutional layers, and the plurality of sub-convolutional layers are independent of each other, and each sub-convolutional layer will obtain a block output, and the plurality of block outputs together constitute the complete output of the convolutional layer. The plurality of block outputs will be applied to the next processing layer, and by recursion, the entire neural network model will be divided into a plurality of sub-network models.

[0040] Figure 2 is an example diagram showing the segmentation result of the neural network model according to an embodiment of the present disclosure. Figure 2As shown, each segmentation operation of the neural network model will form a complete sub-network model, and each sub-network model is independent of each other. The input of each sub-network model can complete the parallel operation of each first block data by connecting to each first block data output by the first processing module 11. Repeating this process can transform the original serial pipeline into a parallel pipeline, thereby effectively increasing the degree of parallelism and reducing system latency.

[0041] Optionally, the neural network model includes a convolution operation layer and an activation operation layer, and the neural network accelerator module is configured to perform convolution operations on the plurality of first block data based on the plurality of convolution operation layers to generate a plurality of intermediate block data, and to perform activation operations on the plurality of intermediate block data based on the plurality of activation operation layers to generate a plurality of second block data.

[0042] Figure 3 is an example diagram showing a neural network model according to an embodiment of the present disclosure. Figure 3 As shown, the neural network model may include a Conv layer that mainly performs convolution operations and a ReLU layer that mainly performs activation operations, and there are three edges between the two layers, namely, the input of the Conv layer, the output of the Conv layer, and the output of the ReLU layer. At this time, the output data of the first processing module 11 will be used as the input of the Conv layer, and the output data will be output from the ReLU layer after being processed by the neural network model.

[0043] Optionally, the neural network accelerator module 12 is configured to start the second processing of a predetermined amount of data in response to reading a predetermined amount of data in the first block data.

[0044] Optionally, the first processing module 11 includes a pre-processing accelerator, and the second processing module 13 includes a post-processing accelerator.

[0045] Optionally, the first processing module 11 includes an image processing accelerator, and the second processing module 13 includes a graphics rendering processor.

[0046] Optionally, the first processing module 11, the neural network accelerator module 12 and the second processing module 13 are configured to perform the first processing, the second processing and the third processing respectively in parallel.

[0047] Optionally, the first processing module 11 is configured to generate a first group of the first block data in a first operation cycle. The neural network accelerator module 12 is configured to read the first group of the first block data and generate the first group of the second block data in a second operation cycle, and the first processing module 11 is configured to generate a second group of the first block data in the second operation cycle. The second processing module 13 is configured to read the first group of the second block data and generate the first group of the third block data in a third operation cycle, the neural network accelerator module 12 is configured to read the second group of the first block data and generate the second group of the second block data in the third operation cycle, and the first processing module 11 is configured to generate a third group of the first block data in the third operation cycle.

[0048] Optionally, the second processing module 13 is configured to read a second group of the second block data and generate a second group of the third block data in a fourth operating cycle, the neural network accelerator module 12 is configured to read a third group of the first block data and generate a third group of the second block data in the fourth operating cycle, and the first processing module 11 is configured to generate a fourth group of the first block data in the fourth operating cycle.

[0049] Optionally, the first processing module 11 is further configured to obtain the size, number and / or interval of the blocks according to the received instruction. At this time, the user can configure the block parameters of the first processing module 11 through software.

[0050] Optionally, the first processing module 11 is configured to output the first data reading notification in the form of an interrupt when the first processing is completed, and / or the neural network accelerator module 12 is configured to output the second data reading notification in the form of an interrupt when the second processing is completed.

[0051] Figure 4 2 is an example diagram showing sending a first data read notification according to an embodiment of the present disclosure. Figure 4 As shown, the first processing module 11 can output the first data reading notification corresponding to each of the first block data by triggering an interrupt. In addition, the first processing module 11 can also output the first data reading notification corresponding to each of the first block data by sending a signal or the like. It should be noted that the above two notification methods are only two feasible solutions of the present disclosure, but the present disclosure is not limited thereto.

[0052] Optionally, the neural network accelerator module 12 stores the plurality of first block data read from the first processing module 11 in a cache. When the start condition is met, the neural network accelerator module 12 reads the first block data from the cache and performs a second processing to generate a plurality of second block data respectively.

[0053] Optionally, the joint computing system 1 may further include a data acquisition module, which is used to continuously acquire data and send the data to the first processing module 11. The data acquisition module is, for example, a camera.

[0054] According to an embodiment of the present disclosure, the joint computing system 1 also includes a control module and a storage module, wherein the control module is used to load pre-stored data in the storage module to initialize the first processing module 11, the neural network accelerator module 12 and the second processing module 13.

[0055] Optionally, the control module is also used to control a data acquisition device to collect data, wherein the data collected by the data acquisition device can be stored in a RAM so as to be shared by various modules.

[0056] Figure 5 is an example diagram showing a joint computing system according to an embodiment of the present disclosure. Figure 5 As shown, the first processing module includes an image processing accelerator IPU, the neural network accelerator module includes a neural network accelerator NPU, the second processing module includes a graphics processing unit GPU, the control module includes a central processing unit CPU, and the storage module includes a random access memory RAM and a read-only memory ROM. In addition, in this embodiment, the joint computing system may also include a camera and a display.

[0057] The operation scenario of the joint computing system described in this embodiment is: the CPU loads the initialization and intelligent model data stored in the ROM to initialize each system device; the CPU controls the camera to collect external image data frames, and the image data frames collected by the camera are stored in the RAM for sharing; the image data frames are passed to the image processing accelerator for image processing and format conversion and other operations; the image data frames processed by the image processing accelerator are passed to the neural network accelerator to perform intelligent application operations and output the operation results; the GPU draws the relevant operation results and renders them to the display screen.

[0058] This embodiment improves the hardware and software processing methods to support the block execution method. The blocks include horizontal or vertical blocks, and their size, number, interval, etc. can be configured through software. After the IPU executes each data block, an interrupt or signal is immediately triggered to quickly notify the NPU, so that the NPU can get the image frame data block that the IPU has partially processed in advance for processing. At the same time, the IPU then executes the data task of the next block. In this way, by improving the hardware and software processing methods of the IPU and NPU, the processing of each data block in each image frame is carried out in parallel, reducing system latency, thereby improving the response of the intelligent system.

[0059] The present disclosure also provides a joint computing method. Figure 6 FIG. 1 is a flow chart showing a joint calculation method according to an embodiment of the present disclosure. Figure 6 As shown, the joint calculation method includes the following steps S61 to S63.

[0060] In step S61 , the input data is first processed in a block manner to generate a plurality of first block data respectively, and a first data read notification corresponding to each of the first block data is output.

[0061] In step S62, in response to the first data reading notification, multiple first block data are read respectively, and a neural network model is used to perform a second processing on each of the first block data to generate multiple second block data respectively, and a second data reading notification corresponding to each of the second block data is output.

[0062] In step S63 , in response to the second data reading notification, a plurality of second block data are read respectively, and a third process is performed on each of the second block data to generate a plurality of third block data respectively.

[0063] In this embodiment, the above steps S61 to S63 are similar to the steps Figure 1 The corresponding modules in the joint computing system 1 shown correspond to each other, and will not be described in detail here. In addition, the joint computing method may include other operations or actions performed by the corresponding modules in the joint computing system 1.

[0064] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the aforementioned joint computing method when executed by a processor.

[0065] The present disclosure also provides an electronic device. Figure 7 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure. Figure 7As shown, the electronic device 700 includes a memory 710 and a processor 720. The memory 710 is configured to store a computer program. The processor 720 is configured to call the computer program to execute the aforementioned joint computing method.

[0066] Optionally, the electronic device 700 may further include a display 730, and the display 730 is configured to display a GUI interaction interface related to the joint computing method.

[0067] The protection scope of the joint computing method described in the present disclosure is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.

[0068] In summary, the joint computing system described in one or more embodiments of the present disclosure can process input data in blocks and in parallel, which is beneficial to reducing the system latency and system operating memory. Therefore, the present disclosure effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0069] The above embodiments are merely illustrative of the principles and effects of the present disclosure, and are not intended to limit the present disclosure. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present disclosure. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present disclosure shall still be covered by the claims of the present disclosure.

Claims

1. A joint computing system, characterized in that: The joint computing system comprises: A first processing module is configured to perform a first processing on the input data in a block manner to generate a plurality of first block data respectively, and output a first data reading notification corresponding to each of the first block data; a neural network accelerator module, coupled to the first processing module, configured to read a plurality of the first block data from the first processing module in response to the first data read notification, perform a second process on each of the first block data using a neural network model to generate a plurality of second block data, and output a second data read notification corresponding to each of the second block data; and a second processing module, coupled to the neural network accelerator module, configured to read a plurality of the second block data from the neural network accelerator module in response to the second data reading notification, and perform a third processing on each of the second block data to generate a plurality of third block data, respectively; The first processing module, the neural network accelerator module and the second processing module are configured to perform the first processing, the second processing and the third processing respectively in parallel.

2. The joint computing system according to claim 1, characterized in that: The first processing module is configured to perform the first processing on the input data in a horizontal blocking manner and / or a vertical blocking manner.

3. The joint computing system according to claim 1, characterized in that: The neural network model is divided into a plurality of sub-network models respectively matching a plurality of the first block data, and the neural network accelerator module is configured to use each of the sub-network models to process the corresponding first block data.

4. The joint computing system according to claim 1, characterized in that: The neural network model includes a convolution operation layer and an activation operation layer. The neural network accelerator module is configured to perform convolution operations on the plurality of first block data based on the plurality of convolution operation layers to generate a plurality of intermediate block data, and to perform activation operations on the plurality of intermediate block data based on the plurality of activation operation layers to generate a plurality of second block data.

5. The joint computing system according to claim 1, characterized in that: The neural network accelerator module is configured to start the second processing of a predetermined amount of data in response to reading a predetermined amount of data in the first block data.

6. The joint computing system according to claim 1, characterized in that: The first processing module includes a pre-processing accelerator, and the second processing module includes a post-processing accelerator.

7. The joint computing system according to claim 6, characterized in that: The first processing module includes an image processing accelerator, and the second processing module includes a graphics rendering processor.

8. The joint computing system according to claim 1, characterized in that: The first processing module is configured to generate a first set of the first block data in a first operation cycle, The neural network accelerator module is configured to read a first set of the first block data and generate a first set of the second block data in a second operation cycle, and the first processing module is configured to generate a second set of the first block data in the second operation cycle, The second processing module is configured to read the first group of the second-block data and generate a first group of the third-block data in a third operating cycle, the neural network accelerator module is configured to read the second group of the first-block data and generate a second group of the second-block data in the third operating cycle, and the first processing module is configured to generate a third group of the first-block data in the third operating cycle.

9. The joint computing system according to claim 8, characterized in that: The second processing module is configured to read the second group of the second-block data and generate a second group of the third-block data in a fourth operating cycle, the neural network accelerator module is configured to read the third group of the first-block data and generate a third group of the second-block data in the fourth operating cycle, and the first processing module is configured to generate a fourth group of the first-block data in the fourth operating cycle.

10. The joint computing system according to claim 1, characterized in that: The first processing module is further configured to obtain the size, number and / or interval of the blocks according to the received instructions.

11. The joint computing system according to claim 1, characterized in that: The first processing module is configured to output the first data read notification in the form of an interrupt when the first processing is completed, and / or the neural network accelerator module is configured to output the second data read notification in the form of an interrupt when the second processing is completed.

12. A joint calculation method, characterized in that: The joint calculation method comprises: Performing a first process on the input data in a block manner to generate a plurality of first block data respectively, and outputting a first data reading notification corresponding to each of the first block data; In response to the first data reading notification, the plurality of first block data are read respectively, a neural network model is used to perform a second processing on each of the first block data to generate a plurality of second block data respectively, and a second data reading notification corresponding to each of the second block data is output; and In response to the second data reading notification, a plurality of second block data are read respectively, and a third process is performed on each of the second block data to generate a plurality of third block data respectively. The first processing, the second processing and the third processing are performed in parallel.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the joint computing method according to claim 12 is implemented.

14. An electronic device, characterized in that: The electronic device comprises: a memory configured to store a computer program; and A processor is configured to call the computer program to execute the joint computing method according to claim 12.

Images