Image data processing method and convolutional neural network accelerator

Abstract

The invention belongs to the field of convolution operation acceleration, and provides an image data processing method and a convolutional neural network accelerator, and the method comprises the steps: selecting original image information according to a reading instruction, adjusting a specified preset segmentation model according to the specification of an overlapping region, and obtaining an image segmentation model, reading original sub-image information from the original image information according to the image cutting model, and sending the original sub-image information to the convolution operation unit of each network layer to obtain target sub-image information, and sending intermediate sub-image information generated by the convolution operation unit of each network layer to an on-chip memory for other convolution operation units to read and update until target sub-image information is obtained. According to the scheme, the access speed limitation of an external memory is broken through for image processing, and the overall accuracy is prevented from being influenced by edge distortion of image processing by flexibly setting the specification of the overlapping region.

Description

Technical Field

[0001] The present invention relates to the field of convolution operation acceleration, and in particular to an image data processing method and a convolutional neural network accelerator. Background Art

[0002] With the development of the Internet of Things and artificial intelligence, Convolutional Neural Network (CNN), as a common algorithm for image recognition, is widely used in image segmentation, image recognition, image denoising and other fields, and plays a vital role in scenarios such as target detection in the field of the Internet of Things.

[0003] Convolutional neural network accelerators mainly include multiple operator units such as on-chip storage modules, convolution operation modules, activation functions, etc. Usually, the amount of image data and network parameter data is relatively large, and convolutional neural network accelerators often need to set up additional external memory to meet the requirements of high data volume storage.

[0004] The existing convolutional neural network accelerator stores the feature map data of the multi-layer network in an external memory, and the on-chip memory module is used for data caching. During operation, the data access speed of the external memory and the on-chip memory needs to match the operation speed of the convolution operation module, and the access speed of the external memory itself is limited, which makes the operation speed and processable resolution of the overall convolutional neural network accelerator subject to the access speed of the external memory. In order to solve the problem that the access speed of the external memory limits the image processing resolution, when setting the convolutional neural network accelerator, the original image is often uniformly cut to generate multiple sub-images, and each sub-image is processed by convolution operation respectively, and finally the image is fused to obtain the overall processed image of the original image after processing. Since the edge will be distorted during image processing, each sub-image obtained by directly performing convolution operation on the original image will have edge distortion, which makes the overall processed image after the final image fusion distorted.

[0005] Therefore, it is necessary to develop image data processing methods and convolutional neural network accelerators, which can not only avoid the impact of external memory access speed when processing high-resolution images, but also avoid edge distortion effects to ensure the accuracy of the image processing process. Summary of the invention

[0006] The purpose of the present invention is to provide an image data processing method and a convolutional neural network accelerator to solve the problems in the prior art that the access speed of the external memory limits the image processing resolution, and the edge distortion caused by processing the sub-images obtained by cutting the original image affects the accuracy of the overall image processing results.

[0007] The basic scheme of the present invention is: an image data processing method for a convolutional neural network accelerator, the method comprising:

[0008] Selecting original image information according to a reading instruction;

[0009] Adjust the specified preset segmentation model according to the overlap area specifications to obtain the image segmentation model;

[0010] Reading original sub-image information from the original image information according to the image segmentation model;

[0011] The original sub-image information is sent to a convolution operation network to obtain target sub-image information; the convolution operation network includes at least two convolution operation units, and the information output by the convolution operation unit is stored in an on-chip memory for reading by other convolution operation units.

[0012] Beneficial effects of the basic program:

[0013] The original image information is selected according to the reading instruction from the user. The source of the original image information can be sent to the convolutional neural network accelerator of this case by external equipment / software through communication methods such as data transmission lines and data transmission networks, can be stored in an external memory connected to the convolutional neural network accelerator of this case, or can be generated by other modules in the convolutional neural network accelerator of this case. The original image information is selected, and the preset segmentation model is adjusted according to the overlapping area specifications to generate an image segmentation model. The image segmentation model is used to read the original image information in blocks to obtain multiple original sub-image information. The convolution operation network operates on the original sub-image to generate the target sub-image information, and the output of the convolution operation unit of each network layer in the convolution operation network is stored in the on-chip memory for reading by other convolution operation units. The target sub-image information is then stored. The storage location of the target sub-image information can be an external memory or an internal memory, and can also be sent to other devices connected to the convolutional neural network accelerator of this case for storage.

[0014] Compared with the prior art, which directly reads the original image information (i.e., whole storage and whole retrieval), this case adopts an image cutting model to perform regional reading of the original image information to obtain multiple original sub-images (i.e., whole storage and partial retrieval). The amount of information read from the external memory at a single time is smaller, which reduces the information pressure during reading. Even when faced with an original image that exceeds the rated resolution of the external memory of this case, image cutting can be used to perform regional reading to generate original sub-images and process them separately, thereby improving the applicability of the convolutional neural network accelerator.

[0015] Compared with the prior art, the on-chip memory stores the entire original image information for reading by the convolution operation unit in the convolution operation network, as well as the cache of the convolution operation network during operation, and directly stores it in the external memory after the convolution operation is completed. In this case, the on-chip memory only needs to store the original sub-image for reading by the convolution operation unit, and store the output of the convolution operation unit for reading by the next convolution operation unit. The amount of information that needs to be stored in the on-chip memory is greatly reduced, which significantly reduces the storage pressure of the on-chip memory, so that the on-chip memory can better serve the convolution operation, improve the operation efficiency, ensure the data communication between the convolution operation units of different network layers, support the collaborative operation of the convolution operation units of multiple network layers, and allow more precise convolution operations.

[0016] Compared with the prior art, the basic image blocks obtained by the preset segmentation model are directly processed by convolution operation, and the edges of the processed basic image blocks are distorted, which will cause distortion in the subsequent fusion operation. In this case, the image segmentation model fine-tunes the preset segmentation model according to the overlapping area specifications, and sets the concept of overlapping area outside the basic image block, so that even if the original sub-image obtained by the image segmentation model has edge distortion, the distorted edge will only be located in the overlapping area. Subsequently, it is only necessary to delete the overlapping area part of each target sub-image information and merge them into a large image to eliminate the adverse effects of edge distortion on the whole, thereby ensuring the accuracy of the convolution operation processing of the original image.

[0017] Furthermore, the original image information is stored in an external memory, the target sub-image is stored in the external memory, and the target sub-image information is used to merge into the final image information corresponding to the original image information after deleting the overlapping area part corresponding to the overlapping area specification.

[0018] Beneficial effect: Compared with the prior art, in which the external memory not only stores the entire original image information, but also needs to store the information output by each convolution operation unit and be read by the next convolution operation unit, in this case, the external memory stores the target sub-image information obtained by each original sub-image information through the convolution operation network, and the overall storage data volume is reduced, reducing the storage space occupied by the external memory. Even if the entire original image information is stored in the external memory, when the external memory is accessed, the bandwidth occupied is also reduced because the amount of data accessed each time is reduced (reading a single original sub-image information, accessing the target sub-image information). By deleting the overlapping area of ​​each target sub-image information and merging them into the final image information, the adverse effect of edge distortion that may occur during the convolution operation on the overall image can be eliminated, thereby ensuring the accuracy of the convolution operation processing of the original image information.

[0019] Further, sending the original sub-image information to a convolution operation network to obtain target sub-image information includes:

[0020] The convolution operation unit at the top generates intermediate sub-image information according to the original sub-image and stores it in the on-chip memory;

[0021] The convolution operation unit at the last position generates target sub-image information according to the intermediate sub-image information in the on-chip memory.

[0022] Beneficial effect: The convolution operation units are processed in a "pipeline" form. The intermediate sub-image information generated by the first convolution operation unit is processed by other convolution operation units behind; the last convolution operation unit generates the target sub-image information as the output of the entire convolution operation network. The previous convolution operation unit and the next convolution operation unit can pass through one on-chip memory or two on-chip memories.

[0023] Furthermore, before adjusting the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model, the method further includes:

[0024] The preset segmentation model is used to read the original image information to obtain basic image block information;

[0025] Setting a corresponding set of overlapping area values ​​according to the data volume of the basic image block information and the number of network layers in the convolution operation network;

[0026] According to the overlapping specification value in the overlapping area value set, adjusting the preset segmentation model to obtain a candidate segmentation model;

[0027] Determine whether the candidate segmentation model satisfies a verification condition; when the candidate segmentation model satisfies the verification condition, set the overlap specification value corresponding to the candidate segmentation model as the overlap area specification.

[0028] Beneficial effect: In this case, the numerical setting of the overlapping area specification depends on the data volume of the basic image block information and the number of network layers. The sizes of the overlapping area specifications corresponding to the basic image blocks of different sizes are also different. While ensuring that the edge distortion of image processing falls in the overlapping area, it can also reduce the operating pressure of the overlapping area for convolution image processing as much as possible, making the setting of the overlapping area specification more reasonable.

[0029] In addition, the setting of the overlap area specifications depends on whether the candidate segmentation models corresponding to the respective overlap specification values ​​meet the verification pass conditions. The overlap area specifications are set according to the accuracy of the final image processing to ensure the quality of image processing.

[0030] Further, the determining whether the candidate segmentation model satisfies the verification pass condition includes:

[0031] Obtain verification image information and corresponding verification target processing information;

[0032] Performing verification processing on each of the candidate segmentation models to obtain verification result image information;

[0033] Calculating the similarity between the verification result image information and the verification target processing information;

[0034] When the similarity satisfies the first preset range, the candidate segmentation model satisfies the verification pass condition; when the similarity does not satisfy the first preset range, the candidate segmentation model does not satisfy the verification pass condition;

[0035] The verification process includes: using the candidate segmentation model to cut the verification image information to obtain a plurality of verification sub-image information; and obtaining verification result image information by calculating all the verification sub-image information corresponding to the same candidate segmentation model.

[0036] Beneficial effect: The verification image information is processed by the candidate segmentation model to obtain the verification result image information, and according to the similarity between the verification result image information and the verification target processing information, the candidate segmentation model that meets the first preset range is selected, and the overlapping specification value corresponding to the candidate segmentation model is set as the overlapping area specification. The verification target processing information in this case is obtained by directly processing the verification image information as a whole. Through the verification image information and the verification target processing information, the overlapping specification value that can ensure the image quality is found as the overlapping area specification for this image processing.

[0037] Further, according to the overlap specification value in the overlap area value set, adjusting the preset segmentation model as a candidate segmentation model includes:

[0038] According to the overlap specification value in the overlap area value set corresponding to each basic image block information, adjusting the preset segmentation model as a partial segmentation model;

[0039] According to the position of the basic image block information relative to the original image information, the partial segmentation models are combined into the candidate segmentation model.

[0040] Beneficial effect: The overlapping area of ​​each basic image block information can be different, so the partial cutting model corresponding to each basic image block can also be different, and finally the candidate cutting model is combined according to the distribution of the basic image blocks.

[0041] Further, the preset segmentation model is adjusted according to the overlap area specification to obtain the image segmentation model, including:

[0042] According to the cutting configuration instructions, the corresponding preset cutting model is obtained;

[0043] The preset segmentation model is used to segment the original image information to generate a cutting line of a single basic image block as a reference cutting line set;

[0044] Taking the reference cutting line set as a reference, each cutting line is extended outward by a length corresponding to the overlapping area specification to obtain a newly created cutting line set;

[0045] The image cutting model is generated according to the newly created cutting line set.

[0046] Beneficial effect: The preset segmentation model is converted into a reference cutting line set facing each basic image block, and then the overlapping area specifications corresponding to the basic image block are expanded outward to generate a new cutting line set for each basic image block, and combined into a new image segmentation model according to the position of each basic image block relative to the original image information. After setting the overlapping area specifications, the preset segmentation model is adjusted to obtain the image segmentation model.

[0047] Further, obtaining a corresponding preset segmentation model according to the segmentation configuration instruction includes:

[0048] When the cutting configuration instruction is a parameter-based instruction, a corresponding preset cutting model is generated according to the parameter configuration in the cutting configuration instruction, wherein the parameter configuration includes the number of blocks, the length of each block, the height of each block, and the position coordinates of each block in the original image information; and / or,

[0049] When the cutting configuration instruction is a selection instruction, finding a corresponding preset cutting model from a preset table according to the cutting configuration instruction; and / or,

[0050] When the cutting configuration instruction is an image recognition instruction, image recognition is performed on the image type information of the original image information, and a prediction segmentation model is generated according to an image recognition rule corresponding to the type information; and / or,

[0051] When the cutting configuration instruction is a noise recognition instruction, the original image information is denoised to obtain denoised image information and noise image information, noise distribution information is obtained based on the noise image information combined with a clustering algorithm, and a preset cutting model is generated based on the noise distribution information.

[0052] Beneficial effects: Different methods are used to generate preset cutting models according to the type of cutting configuration instructions. First, when the cutting configuration instruction is a parameter-type instruction, a preset cutting model is generated according to the key parameter configuration of the cutting configuration instruction. Second, when the cutting configuration instruction is a selection-type instruction, the corresponding preset cutting model is found according to the cutting configuration instruction. Third, when the cutting configuration instruction is an image recognition-type instruction, a predictive cutting model is generated according to the image type of the image recognized by the image recognition rules. Fourth, a preset cutting model is generated according to the noise recognition result. Specifically, when the original image information is denoised, the noise image information is obtained, and then the noise distribution information is obtained by the clustering algorithm. Then, the original image is divided into regions according to the density of the noise distribution, and the preset cutting model is generated according to the region division rule.

[0053] Further, when the convolution operation network includes at least three convolution operation units, sending the original sub-image information to the convolution operation network to obtain the target sub-image information includes:

[0054] The convolution operation unit at the top generates intermediate sub-image information according to the original sub-image and stores it in the on-chip memory;

[0055] The convolution operation unit in the middle position generates new intermediate sub-image information according to the intermediate sub-image information stored in the on-chip memory by the previous convolution operation unit and stores it in the on-chip memory;

[0056] The convolution operation unit at the last position generates target sub-image information according to the intermediate sub-image information in the on-chip memory.

[0057] Beneficial effect: The convolution operation units are processed in the form of a "pipeline", and the pipeline has at least three convolution operation units. The first convolution operation unit generates the intermediate sub-image information and stores it in the on-chip memory of the convolution operation unit; the middle convolution operation unit reads the intermediate sub-image information stored in the corresponding on-chip memory of the previous convolution operation unit, performs convolution operation processing to obtain the intermediate sub-image information, and stores it in the corresponding on-chip memory; the last convolution operation unit generates the target sub-image information as the output of the entire convolution operation network. The previous convolution operation unit and the next convolution operation unit can pass through one on-chip memory or two on-chip memories.

[0058] Furthermore, one end of the same on-chip memory is connected to the output end of the previous convolution operation unit, and the other end of the on-chip memory is connected to the input end of the next convolution operation unit.

[0059] Beneficial effect: Compared with the case where the input end of each convolution operation unit is connected to a front on-chip memory and the output end is connected to a rear on-chip memory, the rear on-chip memory of the previous convolution operation unit in adjacent convolution operation units is connected to the front on-chip memory of the next convolution operation unit. In this case, one on-chip memory simultaneously serves as the output data storage of the previous convolution operation unit and the data reading of the next convolution operation unit. The overall structure is more concise, and the data handling process is reduced, thereby improving the overall efficiency.

[0060] Furthermore, the on-chip memory that meets the preset transfer requirements is also used to transfer the intermediate sub-image information of the previous convolution operation unit to the external memory, and the next convolution operation unit reads the intermediate sub-image information from the external memory.

[0061] Beneficial effect: Considering the limited storage capacity of the on-chip memory, after meeting the preset transfer requirements, the on-chip memory transfers the intermediate sub-image information stored in itself to the external storage, and the next convolution operation unit reads from the external storage, thereby ensuring the access stability of the on-chip memory. Usually, the preset transfer requirements are associated with the size of the original sub-image.

[0062] The present invention also provides a convolutional neural network accelerator, including an information transceiver module, a segmentation model configuration module, an image cutting module, a convolution operation module, and an on-chip memory;

[0063] The information transceiver module is used to select original image information from the external memory according to the selected instruction;

[0064] The segmentation model configuration module is used to adjust the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model;

[0065] The image cutting module is connected to the information transceiving module and the segmentation model configuration module, and is used to read the original image information using the image cutting model to obtain the original sub-image information;

[0066] The on-chip memory is connected to the image cutting module and the convolution operation module, and is used to store original sub-image information from the image cutting module;

[0067] The convolution operation module is connected to the on-chip memory and the information transceiver module, and is used to perform convolution operation through a convolution operation network according to the original sub-image information to obtain target sub-image information; the convolution operation module includes a plurality of convolution operation units, and the convolution operation units are used to generate intermediate sub-image information according to the original sub-image information and store it in the on-chip memory, and are also used to generate the target sub-image information according to the intermediate sub-image information in the on-chip memory;

[0068] The information transceiver module is also connected to the convolution operation module and is used to transfer the target sub-image information from the convolution operation module.

[0069] Beneficial effects: The on-chip memory is used to store cache during convolution operations and as a bridge for information exchange between convolution operation units of multiple network layers, ensuring data communication between convolution operation units of different network layers, supporting collaborative operations of convolution operation units of multiple network layers, and allowing more precise convolution operations. The image segmentation model fine-tunes the preset segmentation model according to the overlapping area specifications, and sets the concept of overlapping areas outside the basic image blocks, so that even if the original sub-image obtained by the image segmentation model has edge distortion, the distorted edge will only be located in the overlapping area. Subsequently, it is only necessary to delete the overlapping area part of each target sub-image information and merge them into a large image to eliminate the adverse effects of edge distortion on the overall image, thereby ensuring the accuracy of the convolution operation processing of the original image.

[0070] Furthermore, the information transceiver module selects original image information from an external memory according to a selected instruction; the information transceiver module is used to transfer the target sub-image information from the convolution operation module into the external memory; the target sub-image information is used to merge into the final image information corresponding to the original image information after deleting the overlapping area part corresponding to the overlapping area specification.

[0071] Beneficial effects: When the external memory is accessed, the bandwidth occupied is also reduced because the amount of data accessed each time is reduced (reading a single original sub-image information and accessing the target sub-image information), and the data in the convolution operation process is stored in the on-chip memory, reducing the data space occupied by the external memory. By deleting the overlapping area of ​​each target sub-image information and merging them into the final image information, the adverse effect of edge distortion that may occur during the convolution operation on the overall image can be eliminated, ensuring the accuracy of the convolution operation processing of the original image information.

[0072] Further, the segmentation model configuration module includes an overlap area debugging unit and a cutting model generation unit;

[0073] The overlapping area debugging unit is used to debug the obtained overlapping specification value according to the verification image information and the verification target processing information, in combination with the number of network layers and the data volume of the basic image block information;

[0074] The cutting model generating unit is used to adjust the specified preset cutting model according to the overlapping area specification to obtain the image cutting model;

[0075] The basic image block is generated by cutting the original image information with a preset segmentation model.

[0076] Beneficial effect: The debugging of the overlapping area specification depends on whether the candidate segmentation model meets the verification pass conditions to ensure the quality of image processing. The preset segmentation model is adjusted according to the overlapping area specification to obtain the image cutting model, and the overlapping area is set outside the basic image block to form the original sub-image, so as to ensure that the edge distortion that occurs during the convolution operation is located in the overlapping area.

[0077] In summary, the present application includes at least one of the following beneficial technical effects:

[0078] (1) The entire original image information in the external memory and the target sub-image information obtained by the convolution operation network are reduced in the overall storage data volume, thereby reducing the storage space occupied by the external memory.

[0079] (2) The image segmentation model is used to read the data of the original image information in the area to obtain multiple original sub-image information. In this process, the bandwidth required for reading the data information is the bandwidth corresponding to the original sub-image information. In addition, when the target sub-image information obtained by processing the original sub-image information through the convolutional network is stored in the external memory, the bandwidth occupied is the bandwidth corresponding to the target sub-image information. Since the data volume of the original sub-image information and the target sub-image information is small, the amount of data accessed by the external memory each time is small (reading a single original sub-image information and accessing the target sub-image information), and the bandwidth occupied is also reduced.

[0080] (3) The on-chip memory only needs to store the cache during the convolution operation (storing the original sub-image for the convolution operation unit to read, and storing the output of the convolution operation unit for the next convolution operation unit to read). The amount of information that needs to be stored in the on-chip memory is greatly reduced, which reduces the storage pressure of the on-chip memory, enables the on-chip memory to better serve the convolution operation, improves the operation efficiency, ensures data communication between the convolution operation units of different network layers, supports the collaborative operation of the convolution operation units of multiple network layers, and allows more precise convolution operations.

[0081] (4) When reading the original image information, the image cutting model is directly used for cutting, which reduces the information pressure during reading. Even when facing the original image that exceeds the rated resolution of the external memory in this case, the original sub-image can be generated by image cutting, and the image is processed by the convolution operation unit of the network layer, thereby improving the applicability of the convolutional neural network accelerator.

[0082] (5) The image segmentation model fine-tunes the preset segmentation model according to the specifications of the overlapping area. The concept of the overlapping area is set outside the basic image block, so that even if the original sub-image obtained by the image segmentation model has edge distortion, the distorted edge will only be located in the overlapping area. Subsequently, it is only necessary to delete the overlapping area of ​​each target sub-image information and merge them into a large image to eliminate the adverse effects of edge distortion on the overall image, thereby ensuring the accuracy of the convolution operation on the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A flowchart of an image data processing method provided by a first embodiment of the present invention;

[0084] Figure 2 for Figure 1 A partial flow chart of step 102;

[0085] Figure 3 for Figure 1 A partial flow chart of step 102;

[0086] Figure 4 for Figure 1 Schematic diagram of verification image information and verification target processing information;

[0087] Figure 5 yes Figure 3 Schematic diagram of the difference between the verification target processing information, the verification result image information, the verification image processing information and the verification image result information when the specification of the middle overlap area is set to be smaller;

[0088] Figure 6 yes Figure 3 Compared with the specifications of the overlap area Figure 5 A schematic diagram of the difference between the verification target processing information, the verification result image information, the verification image processing information and the verification image result information when the verification target processing information is larger;

[0089] Figure 7 yes Figure 3 A schematic diagram showing the difference between verification target processing information, verification result image information, verification image processing information and verification image result information that meets the verification passing condition;

[0090] Figure 8 yes Figure 1 A schematic diagram of the original image information in step 103;

[0091] Fig. 9 yes Figure 1 Schematic diagram of the working process of step 103;

[0092] Fig.10 yes Figure 1 Schematic diagram of the operation principle of the convolution operation network;

[0093] Fig.11 yes Fig.10 A schematic diagram of the operation of a convolutional operation network with two convolutional operation units;

[0094] Fig.12 yes Fig.10 Another operation diagram when the convolution operation network has two convolution operation units;

[0095] Fig.13 yes Fig.10 A schematic diagram of an operation when the convolution operation network has at least three convolution operation units;

[0096] Fig.14 yes Fig.10 Another operation diagram when the convolution operation network has at least three convolution operation units;

[0097] Fig.15 yes Fig.10 A schematic diagram of the operation of the convolution operation network having an on-chip memory of a convolution operation unit storing data into an external memory;

[0098] Fig.16 is a schematic diagram of the structure of a convolutional neural network accelerator provided by a second embodiment of the present invention;

[0099] Fig.17 for Fig.10 Schematic diagram of the process. DETAILED DESCRIPTION

[0100] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0101] First embodiment:

[0102] The first embodiment of the present invention provides an image data processing method for a convolutional neural network accelerator, the method comprising: selecting original image information according to a read instruction; adjusting a specified preset segmentation model according to an overlapping area specification to obtain an image segmentation model; reading original sub-image information from the original image information according to the image segmentation model; sending the original sub-image information to a convolutional operation network to obtain and store target sub-image information; the convolutional operation network comprises at least two convolutional operation units, and information output by the convolutional operation unit is stored in an on-chip memory for reading by other convolutional operation units.

[0103] The original image information is selected according to the reading instruction from the user. The source of the original image information can be sent to the convolutional neural network accelerator of this case by external equipment / software through communication methods such as data transmission lines and data transmission networks, can be stored in an external memory connected to the convolutional neural network accelerator of this case, or can be generated by other modules in the convolutional neural network accelerator of this case. The original image information is selected, and the preset segmentation model is adjusted according to the overlapping area specifications to generate an image segmentation model. The image segmentation model is used to read the original image information in blocks to obtain multiple original sub-image information. The convolution operation network operates on the original sub-image to generate the target sub-image information, and the output of the convolution operation unit of each network layer in the convolution operation network is stored in the on-chip memory for reading by other convolution operation units. The target sub-image information is then stored. The storage location of the target sub-image information can be an external memory or an internal memory, and can also be sent to other devices connected to the convolutional neural network accelerator of this case for storage.

[0104] Preferably, the original image information is stored in an external memory, the target sub-image is stored in the external memory, and the target sub-image information is used to merge into the final image information corresponding to the original image information after deleting the overlapping area part corresponding to the overlapping area specification.

[0105] In this case, the external memory is accessed based on the size of the original sub-image information and the target sub-image information, and the bandwidth required is lower than that of the prior art that accesses the original image information as a whole. Compared with the output of each convolution operation unit being stored in the external memory, this case stores the output of the convolution operation unit in the on-chip memory, which reduces the storage pressure of the external memory and also improves the computing efficiency. Compared with the on-chip memory accessing the original image information as a whole, the on-chip memory in this case only needs to store the original sub-image information that has just been generated and the information output by the convolution operation unit, which reduces the data storage capacity of the on-chip memory. The on-chip memory is used to store the cache during the convolution operation, and serves as a bridge for information exchange between the convolution operation units of multiple network layers, ensuring data communication between the convolution operation units of different network layers, supporting the collaborative operation of the convolution operation units of multiple network layers, and allowing more precise convolution operations.

[0106] Even when faced with an original image that exceeds the rated resolution of the external memory in this case, the original sub-image can be generated by image cutting, and the image is processed by the convolution operation unit of the network layer, which improves the applicability of the convolutional neural network accelerator. The image cutting model fine-tunes the preset segmentation model according to the overlapping area specifications, and sets the concept of overlapping area outside the basic image block, so that even if the original sub-image obtained by the image cutting model has edge distortion, the distorted edge will only be located in the overlapping area. Subsequently, it is only necessary to delete the overlapping area part of each target sub-image information and merge them into a large image to eliminate the adverse effects of edge distortion on the whole, thereby ensuring the accuracy of the convolution operation processing of the original image.

[0107] The following is a detailed description of the implementation details of the image data processing method of this embodiment. The following content is only provided for the convenience of understanding the implementation details, and is not necessary for the implementation of this solution. The specific process of this embodiment is as follows Figure 1 As shown, the following steps are included:

[0108] Step 101, selecting original image information according to a read instruction.

[0109] Specifically, the source of the reading instruction can be generated by manual input by the user, or generated by the user through a third-party device such as a trigger button or switch, or generated by a server connected to the convolutional neural network accelerator that applies the method, etc. The source of the original image information can be external input, such as external software or equipment directly imported through a data transmission network / data transmission line, etc., and another example is external sensors directly imported after acquisition through communication protocols or serial communication, etc.; the source of the original image information can be read from the outside, such as the convolutional neural network accelerator reading the original image information from a connected external memory; the source of the original image information can also be generated by the convolutional neural network accelerator itself, or by the device where the convolutional neural network accelerator is located.

[0110] In some examples, after receiving a read instruction input from a user, the convolutional neural network accelerator accesses an external memory connected to itself and selects original image information corresponding to the read instruction from the external memory.

[0111] Step 102, adjusting the specified preset segmentation model according to the overlap area specification to obtain an image segmentation model.

[0112] Specifically, usually before the implementation of step 102 , it is necessary to determine the overlap area specification and the specified preset segmentation model, and then execute step 102 .

[0113] (2.1) Determine the preset segmentation model:

[0114] In some examples, the preset segmentation model is a fixed segmentation model. After being input by a designer, the fixed preset segmentation model is used for segmentation regardless of the size, resolution, and amount of information of the original image information.

[0115] In some examples, a corresponding preset segmentation model is obtained according to the segmentation configuration instruction. Figure 2 As shown, specifically including:

[0116] S2-1-1, identify the type of cutting configuration instruction; if the cutting configuration instruction is a parameter type instruction, execute S2-1-2; if the cutting configuration instruction is a selection type instruction, execute S2-1-3; if the cutting configuration instruction is an image recognition type instruction, execute S2-1-4; if the cutting configuration instruction is a noise recognition type instruction, execute S2-1-5;

[0117] S2-1-2, generating a corresponding preset segmentation model according to the parameter configuration in the segmentation configuration instruction, wherein the parameter configuration includes the number of blocks, the length of each block, the height of each block, and the position coordinates of each block in the original image information;

[0118] S2-1-3, finding a corresponding preset segmentation model from a preset table according to the segmentation configuration instruction;

[0119] S2-1-4, image recognition of the image category information of the original image information, and generating a prediction segmentation model according to the image recognition rules corresponding to the category information;

[0120] S2-1-5, denoising the original image information to obtain denoised image information and noise image information, obtaining noise distribution information based on the noise image information in combination with a clustering algorithm, and generating a preset segmentation model based on the noise distribution information.

[0121] In this example, different preset segmentation model generation methods are selected according to the type of the cutting configuration instruction to achieve diverse generation of preset segmentation models.

[0122] In S2-1-2, a preset cutting model is generated according to the parameter configuration in the cutting configuration instruction. That is to say, the user can directly specify the number of blocks PN, the length PW of each block, the height PH of each block, and the position coordinates (PC, PR) of each block in the original image information by setting the cutting configuration instruction, importing the existing cutting configuration instruction, etc. This method directly generates a preset cutting model, wherein the position coordinates represent the position of each block relative to the original image information, and the position coordinates can be the center coordinates, the starting coordinates, or other coordinates representing the block position.

[0123] In S2-1-3, the preset table is a table preset by the user, and the user can store the preset segmentation model generated by other methods in the preset table for direct selection next time.

[0124] In S2-1-4, the image recognition rules are pre-set and associated with the category information. Each image corresponds to an image recognition rule, and the image recognition rules corresponding to each image information are pre-input and set. In the image recognition, the category information of the original image information is found according to the category information, and then the prediction segmentation model is generated according to the image recognition rule. For example, if the image category is a face selfie, then the image recognition rule will cut the original image information into the face image part and the background image part by image recognition, and the prediction segmentation model is the segmentation model corresponding to the image recognition rule.

[0125] In S2-2-5, when the original image information is denoised, the noise image information is obtained, and then the noise distribution information is obtained through a clustering algorithm according to the density of the noise distribution. The noise distribution area includes a high noise area, a medium noise area, and a low noise area. The original image information is divided into regions according to the noise distribution information. The parameters of each block are (GW, GH, GC, GR) which respectively represent the length, height and starting coordinates of the block in the original image, and a preset segmentation model is generated according to the rules of the region division. The rule of region division is usually that the more noise there is, the greater the amount of effective information in the region, and the smaller and denser the set blocks are.

[0126] (2.2) Determine the overlap area specifications:

[0127] In some examples, the overlap region specification is set to a fixed value that can be selected.

[0128] In this example, the image blocks generated by cutting the original image information may be set with the same overlap area specification or with different overlap area specifications.

[0129] In some examples, the overlap area specification is set to correspond to the data volume, size, and number of network layers of the base image block.

[0130] In detail, the method for determining the overlap area specification is as follows Figure 3 As shown, including:

[0131] S2-2-1, using a preset segmentation model to segment the original image information to obtain basic image block information;

[0132] S2-2-2, setting a corresponding overlapping area value set according to the data volume of the basic image block information and the number of network layers;

[0133] S2-2-3, according to the overlapping specification value in the overlapping area value set, adjusting the preset segmentation model as the candidate segmentation model, and judging whether the candidate segmentation model meets the verification pass condition;

[0134] S2-2-4, when the candidate segmentation model meets the verification pass condition, the overlap specification value corresponding to the candidate segmentation model is set as the overlap area specification.

[0135] The setting of the overlap area specification depends on the amount of data of the basic image block information and the number of network layers. The sizes of the overlap area specifications corresponding to the basic image blocks of different sizes are also different. While ensuring that the edge distortion of the image processing falls in the overlap area, it can also reduce the operating pressure of the overlap area for the convolution image processing as much as possible, making the setting of the overlap area specification more reasonable. And according to whether the candidate segmentation model corresponding to each overlap specification value in the overlap area value meets the verification pass condition, the qualified overlap specification value is selected as the overlap area specification, and the image processing quality is guaranteed by screening through verification.

[0136] In step S2-2-3, according to the overlapping specification value in the overlapping area value set, the preset segmentation model is adjusted as the candidate segmentation model, which specifically includes: S2-2-3-1, according to the overlapping specification value in the overlapping area value set corresponding to each basic image block information, the preset segmentation model is adjusted as a partial segmentation model; S2-2-3-2, according to the position of the basic image block information relative to the original image information, the partial segmentation model is combined into a candidate segmentation model. Considering that the overlapping area of ​​each basic image block information can be different, the partial cutting model corresponding to each basic image block can also be different, and finally combined into a candidate segmentation model according to the distribution of the basic image blocks.

[0137] In step S2-2-3, judging whether the candidate segmentation model meets the verification pass condition includes:

[0138] S2-2-3-3, obtaining verification image information and corresponding verification target processing information;

[0139] S2-2-3-4, performing verification processing on each candidate segmentation model to obtain verification result image information; the verification processing includes using the candidate segmentation model to cut the verification image information to obtain multiple verification sub-image information, and calculating the verification result image information based on all the verification sub-image information corresponding to the same candidate segmentation model;

[0140] S2-2-3-5, calculate the similarity between the verification result image information and the verification target processing information; when the similarity meets the first preset range, the candidate segmentation model meets the verification pass condition; when the similarity does not meet the first preset range, the candidate segmentation model does not meet the verification pass condition.

[0141] The verification image information is processed by the candidate segmentation model to obtain the verification result image information, and according to the similarity between the verification result image information and the verification target processing information, the candidate segmentation model that meets the first preset range is selected, and the overlapping specification value corresponding to the candidate segmentation model is set as the overlapping area specification. The verification target processing information in this case is obtained by directly processing the verification image information as a whole. Through the verification image information and the verification target processing information, the overlapping specification value that can ensure the image quality is found as the overlapping area specification for this image processing.

[0142] The necessity of setting the overlap area specification in this example can be demonstrated by the following cases:

[0143] like Figure 4 As shown in the figure on the left, the verification image information is the original image information used for verification, such as Figure 4 As shown in the figure on the right, the verification target processing information is the image obtained by directly convolving the verification image information as a whole using a high-performance convolutional neural network accelerator ( Figure 4 The noise of the people and various objects on the right has been eliminated). When the overlap area specification is set to a small value, Figure 5 The upper left picture shows the verification target processing information. Figure 5 The upper right picture is the final verification result image information obtained by processing the internal case. Although the naked eye cannot see the difference, the difference can be found by comparing the images. Figure 5 In the following figure, distortion appears in the red area, which is the edge of the original sub-image.

[0144] As the overlap area specification is set to increase, such as Figure 6 As shown, Figure 6 The upper left picture shows the verification target processing information. Figure 6 The upper right picture is the final verification result image information obtained by processing the internal case. Although the naked eye cannot see the difference, the difference can be found by comparing the images. Figure 6 The figure below shows Figure 5 For example, distorted content will appear at the edge of the original sub-image, and the area of ​​the distorted content will be further reduced.

[0145] In the overlap area, the specifications are set reasonably, such as Figure 7 As shown, Figure 7 The upper left picture shows the verification target processing information. Figure 7 The upper right picture is the final verification result image information obtained by processing the internal case, but by comparing the images, we can find the difference. Figure 7In the figure below, the image effect is the same to the naked eye, and the image data is consistent. Therefore, in this case, it can be achieved by constantly adjusting the overlap area specifications. The effect of image processing by first segmenting and then processing in this case is the same as that of directly processing the entire image.

[0146] (2.3) Adjust the specified preset segmentation model according to the overlap area specifications to obtain the image segmentation model:

[0147] In some examples, adjusting a preset segmentation model as a candidate segmentation model according to an overlap specification value in an overlap region value set includes:

[0148] S2-3-1, according to the cutting configuration instruction, obtain the corresponding preset cutting model;

[0149] S2-3-2, using a preset segmentation model to segment the original image information to generate a cutting line of a single basic image block as a reference cutting line set;

[0150] S2-3-3, taking the reference cutting line set as the reference, each cutting line is extended outward to a length corresponding to the overlapping area specification to obtain a new cutting line set;

[0151] S2-3-4, generating an image cutting model according to the newly created cutting line set.

[0152] In this example, the preset segmentation model is converted into a reference cutting line set facing each basic image block, and then a corresponding new cutting line set is generated outward based on the reference cutting line set according to the overlap area specifications, and the new cutting line set corresponding to each reference image block is combined into a new image segmentation model according to the position of each reference image block relative to the original image information. After setting the overlap area specifications, the preset segmentation model is adjusted to obtain the image segmentation model.

[0153] Step 103: read original sub-image information from the original image information according to the image segmentation model.

[0154] Specifically, the original sub-image information is read from the original image information using the image segmentation model in step 102. Here, the original sub-image information actually includes the reference image block and the overlapping area.

[0155] Original image information reference Figure 8 , the image cutting model divides the original image information into multiple regions, region 0, region 1, region 2, ... region N, and reads the data of the corresponding region range from the original image information according to the image cutting model, and obtains the original sub-image corresponding to the region range; for example, read the data in the region range of region 0 to obtain the original sub-image information 0, correspondingly, according to the image cutting model from Figure 8The original image information in can obtain original sub-image information N...original sub-image information 1, original sub-image information 0. Each original sub-image includes a reference image block (content within the frame) and an overlapping area (frame).

[0156] These image blocks are then transmitted to the convolutional operation units of each network layer ( Fig. 9 In the example above, Conv+BN+Relu is used. The size of each image block can be the same or different. The number of image blocks is determined according to the actual application.

[0157] In some examples, the original image information is divided into regions according to an image segmentation model, and the data corresponding to each region is read in sequence at a preset period to obtain the original image information.

[0158] In this example, the region division is performed according to the image cutting model, and the preset period is pre-designed by the staff. The setting of the preset period is often to ensure that there is no information interference between the original sub-image information, and the interference of the subsequent original sub-image information received during the calculation of the original sub-image information in the subsequent convolution operation network.

[0159] Furthermore, reading the original sub-image information from the original image information according to the image segmentation model includes: reading the next original sub-image information only after the previous original sub-image information is passed through the convolution operation network to obtain the target sub-image information.

[0160] refer to Fig. 9 When the convolution operation network calculates the original sub-image information 0 into the target sub-image information 0 and stores the target sub-image information 0 into the external memory, this step 103 will read the original sub-image information 1 from the original image information in the external memory according to the image cutting model; when the convolution operation network generates the target sub-image information 1 and stores the target sub-image information 1 into the external memory, this step 103 will read the original sub-image information 2 from the original image information in the external memory according to the image cutting model...; when the convolution operation network generates the target sub-image information n-1 and stores the target sub-image information n-1 into the external memory, this step 103 will read the original sub-image information n from the original image information in the external memory according to the image cutting model.

[0161] In this example, the image segmentation model reads the original sub-image information from the original image information sequentially to avoid information interference between the original sub-image information, ensuring that the convolution operation unit in the convolution operation network will only operate on the data of one original sub-image information, thereby ensuring the accuracy of the operation.

[0162] Step 104, sending the original sub-image information to the convolution operation network to obtain the target sub-image information and store it; the convolution operation network includes at least two convolution operation units, and the information output by the convolution operation unit is stored in the on-chip memory for reading by other convolution operation units.

[0163] Specifically, in step 103, the original sub-image information is read in sequence, and then the original sub-image information is passed to the convolution operation network in order for convolution operation, to obtain the target sub-image information corresponding to the original sub-image information, and the target sub-image information is stored. The target sub-image information obtained by the convolution operation network operation can be stored in the convolutional neural network accelerator, for example, in the on-chip memory of the convolutional neural network accelerator itself, or in a dedicated memory in the convolutional neural network accelerator; the target sub-image information can also be stored outside the convolutional neural network accelerator, for example, in an external memory connected to the convolutional neural network accelerator, or, for example, the target sub-image information is sent to an external device / server for storage.

[0164] like Fig. 9 As shown in the figure, the convolutional operation network has multiple network layers, each network layer is set with a corresponding convolutional operation unit, and the convolutional operation units of different network layers cannot communicate with each other. The convolutional operation units include "conv" (convolutional layer), "bn" (batch normalization layer), and "relu" (rectified linear unit activation layer). They are combined in the order of conv-bn-relu to form a valid network unit, that is, a network layer in the convolutional operation network.

[0165] After the original sub-image information is obtained in step 103, the original sub-image information 0, the original sub-image information 1, ... the original sub-image information N can be sent to the convolution operation unit (conv) of each network layer (conv-bn-relu) in order. Each network layer can have one convolution operation unit (conv) or multiple convolution operation units. The convolution operation network performs convolution image operation on the original sub-image information to obtain the target sub-image information, and the target sub-image information is then directly sent to the external memory for storage, or transferred to the on-chip memory and stored in the external memory (applicable to target sub-image information with a large amount of data).

[0166] In some examples, considering the large amount of data in the cut original sub-image information, the convolution operation network cannot perform convolution image operation on the original sub-image information as a whole at one time. Therefore, after reading the original sub-image information, it is first stored in the on-chip memory. Each convolution operation unit in the convolution operation network reads the data row it needs from the on-chip memory, performs convolution operation on the data row to obtain the result row, and stores the result row in its corresponding on-chip memory. The convolution operation unit will read all the data in the on-chip memory corresponding to the previous convolution operation unit and store it in its corresponding on-chip memory, so that the result rows corresponding to each data row in the same original sub-image information are collected by the last on-chip memory to obtain the target sub-image information.

[0167] For example, Fig.10 As shown, according to the image segmentation model, original sub-image information 0, original sub-image information 1, ... original sub-image information n can be read from the original image information in sequence; the original sub-image information 0 is stored in the on-chip memory 0, and the convolution operation unit A in the convolution operation network reads a part of the data (data row m1) in the original sub-image information 0 from the on-chip memory 0, performs convolution operation processing, obtains a part of the result (result row M1) and stores it in the on-chip memory 1; the convolution operation unit B reads the remaining part of the data in the original sub-image information 0 from the on-chip memory 0. According to (data row m2), convolution operation is performed to obtain the remaining result (result row M2), and stored in the on-chip memory 2. At the same time, convolution operation unit B also reads a part of the result (result row M1) from the on-chip memory 1 corresponding to the convolution operation unit A, and stores it in the on-chip memory 2 corresponding to itself. At this time, the on-chip memory 2 includes the result row M1 and the result row M2, and then the target sub-image information 0 (M1+M2) corresponding to the original sub-image information 0 (m1+m2) is obtained, and the target sub-image information 0 is stored in the external memory.

[0168] Regarding the convolution operation network, the convolution operation network includes at least two convolution operation units. The information output by the convolution operation unit is stored in the on-chip memory for reading by other convolution operation units. There are several specific examples:

[0169] In some examples, when there are only two convolution operation units in the convolution operation network, the first convolution operation unit generates intermediate sub-image information based on the original sub-image and stores it in the on-chip memory, and the last convolution operation unit generates target sub-image information based on the intermediate sub-image information in the on-chip memory and stores it in the external memory.

[0170] When each convolution operation unit is provided with a front-end on-chip memory and a rear-end on-chip memory, such as Fig.11As shown, the original sub-image information 0 is stored in the front on-chip memory A1 of the convolution operation unit A, the convolution operation unit A reads the original sub-image information 0 from the on-chip memory A1 to perform a convolution operation to obtain the intermediate sub-image information 0, and stores the intermediate sub-image information 0 in the rear on-chip memory A2; the front on-chip memory B1 of the convolution operation unit B reads the data (i.e., the intermediate sub-image information 0) of the rear on-chip memory A2 of the previous convolution operation unit A, the convolution operation unit B reads the intermediate sub-image information 0 from the on-chip memory B1 to perform a convolution operation to obtain the target sub-image information 0, and stores the intermediate sub-image information 0 in the rear on-chip memory B2, and the on-chip memory B2 stores the target sub-image information 0 as the output of the convolution operation network in the external memory.

[0171] When each convolution operation unit is provided with a unique on-chip memory, such as Fig.12 As shown, the convolution operation unit A reads the original image information 0 and performs a convolution operation to obtain the intermediate sub-image information 0, and stores the intermediate sub-image information 0 in its own on-chip memory A2. The convolution operation unit B reads the intermediate sub-image information 0 in the on-chip memory A2 of the previous convolution operation unit A, performs a convolution operation to obtain the target sub-image information 0, and stores the target sub-image information 0 in its own on-chip memory B2. The on-chip memory B2 stores the target sub-image information 0 as the output of the convolution operation network in the external memory.

[0172] In some examples, when the convolution operation network has at least three convolution operation units, the convolution operation unit at the first position generates intermediate sub-image information based on the original sub-image and stores it in the on-chip memory; the convolution operation unit at the middle position generates new intermediate sub-image information based on the intermediate sub-image information stored in the on-chip memory by the previous convolution operation unit and stores it in the on-chip memory; the convolution operation unit at the last position generates target sub-image information based on the intermediate sub-image information in the on-chip memory, and stores it in the external memory.

[0173] When each convolution operation unit is provided with a front-end on-chip memory and a rear-end on-chip memory, such as Fig.13As shown, the original sub-image information 0 is stored in the front on-chip memory A1 of the convolution operation unit A, the convolution operation unit A reads the original sub-image information 0 from the on-chip memory A1 to perform convolution operation to obtain the intermediate sub-image information 0, and stores the intermediate sub-image information 0 in the rear on-chip memory A2; the front on-chip memory C11 of the convolution operation unit C1 reads the data (i.e., the intermediate sub-image information 0) of the rear on-chip memory A2 of the previous convolution operation unit A, the convolution operation unit C reads the intermediate sub-image information 0 from the on-chip memory C11 to perform convolution operation to obtain the intermediate sub-image information 1, and stores the intermediate sub-image information 1 in the rear on-chip memory C12, ... the front on-chip memory Ck1 of the convolution operation unit Ck reads the data of the previous convolution operation unit C(k-1) The convolution operation unit Ck reads the data of the post-positioned on-chip memory C(k-1)2 (i.e., the intermediate sub-image information k-1), and the convolution operation unit Ck reads the intermediate sub-image information k-1 from the on-chip memory Ck1 to perform a convolution operation to obtain the intermediate sub-image information k, and stores the intermediate sub-image information k into the post-positioned on-chip memory Ck2; the front on-chip memory B1 of the convolution operation unit B reads the data of the post-positioned on-chip memory Ck2 of the previous convolution operation unit Ck (i.e., the intermediate sub-image information k), and the convolution operation unit B reads the intermediate sub-image information 0 from the on-chip memory B1 to perform a convolution operation to obtain the target sub-image information 0, and stores the intermediate sub-image information k into the post-positioned on-chip memory B2, and the on-chip memory B2 stores the target sub-image information 0 as the output of the convolution operation network into the external memory.

[0174] When each convolution operation unit is provided with a unique on-chip memory, such as Fig.14 As shown, the convolution operation unit A reads the original image information 0 and performs a convolution operation to obtain the intermediate sub-image information 0, and stores the intermediate sub-image information 0 in its own on-chip memory A2; the convolution operation unit C reads the intermediate sub-image information 0 from the on-chip memory A2 and performs a convolution operation to obtain the intermediate sub-image information 1, and stores the intermediate sub-image information 1 in the rear on-chip memory C12, ... the convolution operation unit Ck reads the intermediate sub-image information k-1 from the on-chip memory C(k-1)2 and performs a convolution operation to obtain the intermediate sub-image information k, and stores the intermediate sub-image information k in the rear on-chip memory Ck2; the convolution operation unit B reads the intermediate sub-image information k in the on-chip memory Ck2 of the previous convolution operation unit Ck, performs a convolution operation to obtain the target sub-image information 0, and stores the target sub-image information 0 in its own on-chip memory B2, and the on-chip memory B2 stores the target sub-image information 0 as the output of the convolution operation network in the external memory. In this case, an on-chip memory serves as both the output data storage of the previous convolution operation unit and the data reading of the next convolution operation unit. The overall structure is simpler, the data transfer process is reduced, and the overall efficiency is improved.

[0175] In some examples, since the convolution operation unit not only performs convolution operation on the regional data (data row) in the original sub-image corresponding to itself to obtain the regional result (result row) and stores it in the on-chip memory, the convolution operation unit also enriches the result of the previous convolution operation unit into its own corresponding on-chip memory. Therefore, as the convolution operation unit is further and further back in the convolution operation network, the data enriched in its on-chip memory becomes more and more. Considering that the storage capacity of the on-chip memory is limited, in this case, when the on-chip memory meets the preset transfer requirements, the data in the on-chip memory is transferred to the external memory. The next convolution operation unit reads from the external memory, which ensures the access stability of the on-chip memory. Usually, the preset transfer requirements are associated with the size of the original sub-image. When the original sub-image information is large, the number of on-chip memories that meet the transfer requirements is greater.

[0176] like Fig.15 As shown, the original sub-image information 0 is read from the original image information of the external memory and transmitted to the convolution operation unit A in the convolution operation network. The convolution operation unit A performs convolution operation to obtain the intermediate sub-image information 0 and stores it in the on-chip memory A2... The convolution operation unit C(k-1) performs convolution operation to obtain the intermediate sub-image information k-1 and stores it in the on-chip memory C(k-1)2. The on-chip memory meets the preset transfer requirements and stores the intermediate sub-image information k-1 in the external memory. The convolution operation unit Ck reads the intermediate sub-image information k-1 from the external memory, performs convolution operation to obtain the intermediate sub-image information k and stores it in the on-chip memory Ck2. The convolution operation unit B performs convolution operation to obtain the target sub-image information 0 and stores it in the on-chip memory B2. This case supports the temporary storage of the intermediate sub-image information in the on-chip memory to reduce the storage pressure of the on-chip memory.

[0177] The intermediate sub-image information generated by the convolution operation unit is usually shared with the next convolution operation unit through the on-chip memory, and can also be shared with the external memory through the on-chip memory. The on-chip memory is used to store the cache during the convolution operation and as a bridge for information exchange between the convolution operation units of multiple network layers, ensuring data communication between the convolution operation units of different network layers, supporting the collaborative operation of the convolution operation units of multiple network layers, and allowing more precise convolution operations.

[0178] Step 105 , deleting the overlapping area portion corresponding to the overlapping area specification in each target sub-image information to obtain intermediate image information, and merging all the intermediate image information into final image information.

[0179] Specifically, after step 102, under the condition of ensuring the operation efficiency, the overlapping area specifications are debugged according to the size of the basic image block and the number of network layers to ensure that even if the original sub-image obtained by the image cutting model has edge distortion, the distorted edge will only be located in the overlapping area. Subsequently, only the overlapping area of ​​each target sub-image information needs to be deleted and merged into a large image to eliminate the adverse effects of edge distortion on the whole, thereby ensuring the accuracy of the convolution operation processing on the original image. This step can be completed by a convolutional neural network accelerator, or it can be performed by other devices connected to the convolutional neural network accelerator.

[0180] The step division of the various methods described above is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0181] Second implementation method:

[0182] A second embodiment of the present invention provides a convolutional neural network accelerator 900, such as Fig.16 As shown, it includes an information receiving and sending module 901, a segmentation model configuration module 902, an image cutting module 903, a convolution operation module 904, and an on-chip memory 905.

[0183] The information transceiver module 901 is used to read the original image information from the external memory 800 according to the read instruction;

[0184] The segmentation model configuration module 902 is used to adjust the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model;

[0185] The image cutting module 903 is connected to the information transceiver module 901 and the segmentation model configuration module 902, and is used to cut the original image information using the image cutting model to obtain the original sub-image information, and send the original sub-image information to the convolution operation module 904 and the on-chip memory 905, or send the original sub-image information to the information transceiver module to be stored in the external memory 800;

[0186] An on-chip memory 905, connected to the image cutting module 903 and the convolution operation module 904, for storing original sub-image information from the image cutting module 903 and target sub-image information from the convolution operation unit in the convolution operation module 904;

[0187] The convolution operation module 904 is connected to the image cutting module 903, the on-chip memory 905 and the information transceiver module 901, and includes a convolution operation unit 9041 of multiple network layers; the convolution operation unit 9041 is used to perform a convolution operation on the original sub-image information from the image cutting module 903 or the on-chip memory 905 to obtain target sub-image information, and store the target sub-image information in the on-chip memory 905 or send it to the information transceiver module 901, and the convolution operation unit 9041 is also used to perform a convolution operation on the target sub-image information from the on-chip memory 905 to obtain new target sub-image information and update it to the on-chip memory 905, or send it to the information transceiver module 901;

[0188] The information transceiver module 901 is also connected to the convolution operation module 904 and is used to transfer the target sub-image information from the convolution operation module 904.

[0189] In some examples, such as Fig.16 As shown, the information transceiver module 901 selects the original image information from the external memory 800 according to the selected instruction; the information transceiver module 901 transfers the target sub-image information from the convolution operation module 904 to the external memory 800; the target sub-image information is used to merge into the final image information corresponding to the original image information after deleting the overlapping area part corresponding to the overlapping area specification.

[0190] In some examples, the segmentation model configuration module 902 includes an overlap area debugging unit 9021 and a cutting model generation unit 9022; the overlap area debugging unit 9021 is used to debug the overlap specification value obtained based on the verification image information and the verification target processing information, combined with the number of network layers and the data volume of the basic image block information; the cutting model generation unit 9022 is used to adjust the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model; wherein the basic image block is generated by cutting the original image information with the preset segmentation model.

[0191] Detailed, such as Fig.17 As shown, the original image data is stored in the external memory 800, and the image data and weights of the original image data are transferred to the on-chip memory 905 (including the feature map cache and the weight cache) through the DMA interface; the image configuration segmentation and image recognition segmentation in the figure are completed by the segmentation model configuration module 902 to generate an image segmentation model; the image segmentation model realizes the segmentation of the original image data read from the external memory through the MUX to obtain the original sub-image; the original sub-image reference Figure 8It is directly transmitted to the convolution operation unit 9041 for convolution operation, or it can be transmitted to the on-chip memory 905 for feature map caching, and then read from the on-chip memory 905 by the convolution operation unit 9041. During this period, the target sub-image information generated by the convolution operation unit 9041 can be stored in the on-chip memory 905 for temporary caching, and then other convolution operation units 9041 perform secondary convolution operations. The convolution operation unit 9041 can also store the target sub-image information in the on-chip memory 905 for temporary caching, and then after the output result is cached, it is transmitted to the external memory 800 through the DMA interface. The convolution operation unit can also directly transmit the target sub-image information to the external memory 800 through the information DMA interface.

[0192] Other implementation details and working methods of the convolutional neural network accelerator 900 disclosed in this application are the same or similar to the image data processing method described above and will not be repeated here.

[0193] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. An image data processing method for a convolutional neural network accelerator, characterized in that: The method comprises: Selecting original image information according to a reading instruction; Adjust the specified preset segmentation model according to the overlap area specifications to obtain the image segmentation model; Reading original sub-image information from the original image information according to the image segmentation model; The original sub-image information is sent to a convolution operation network to obtain and store target sub-image information; the convolution operation network includes at least two convolution operation units, and the information output by the convolution operation unit is stored in an on-chip memory for reading by other convolution operation units.

2. The image data processing method according to claim 1, characterized in that: The original image information is stored in an external memory, the target sub-image is stored in the external memory, and the target sub-image information is used to merge into the final image information corresponding to the original image information after deleting the overlapping area part corresponding to the overlapping area specification.

3. The image data processing method according to claim 1, characterized in that: The sending the original sub-image information to a convolution operation network to obtain and store target sub-image information includes: The convolution operation unit at the top generates intermediate sub-image information according to the original sub-image and stores it in the on-chip memory; The convolution operation unit at the last position generates target sub-image information according to the intermediate sub-image information in the on-chip memory.

4. The image data processing method according to claim 1, characterized in that: Before adjusting the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model, the method further includes: The preset segmentation model is used to read the original image information to obtain basic image block information; Setting a corresponding set of overlapping area values ​​according to the data volume of the basic image block information and the number of network layers in the convolution operation network; According to the overlapping specification value in the overlapping area value set, adjusting the preset segmentation model to obtain a candidate segmentation model; Determine whether the candidate segmentation model satisfies a verification condition; when the candidate segmentation model satisfies the verification condition, set the overlap specification value corresponding to the candidate segmentation model as the overlap area specification.

5. The image data processing method according to claim 4, characterized in that: The determining whether the candidate segmentation model satisfies the verification pass condition includes: Obtain verification image information and corresponding verification target processing information; Performing verification processing on each of the candidate segmentation models to obtain verification result image information; Calculating the similarity between the verification result image information and the verification target processing information; When the similarity satisfies the first preset range, the candidate segmentation model satisfies the verification pass condition; when the similarity does not satisfy the first preset range, the candidate segmentation model does not satisfy the verification pass condition; The verification process includes: using the candidate segmentation model to cut the verification image information to obtain a plurality of verification sub-image information; and obtaining verification result image information by calculating all the verification sub-image information corresponding to the same candidate segmentation model.

6. The image data processing method according to claim 4, characterized in that: According to the overlap specification value in the overlap area value set, adjusting the preset segmentation model as a candidate segmentation model includes: According to the overlap specification value in the overlap area value set corresponding to each basic image block information, adjusting the preset segmentation model as a partial segmentation model; According to the position of the basic image block information relative to the original image information, the partial segmentation models are combined into the candidate segmentation model.

7. The image data processing method according to claim 1, characterized in that: The step of adjusting the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model includes: According to the cutting configuration instructions, the corresponding preset cutting model is obtained; The preset segmentation model is used to segment the original image information to generate a cutting line of a single basic image block as a reference cutting line set; Taking the reference cutting line set as a reference, each cutting line is extended outward by a length corresponding to the overlapping area specification to obtain a newly created cutting line set; The image cutting model is generated according to the newly created cutting line set.

8. The image data processing method according to claim 7, characterized in that: The step of obtaining a corresponding preset segmentation model according to the segmentation configuration instruction includes: When the cutting configuration instruction is a parameter-based instruction, a corresponding preset cutting model is generated according to the parameter configuration in the cutting configuration instruction, wherein the parameter configuration includes the number of blocks, the length of each block, the height of each block, and the position coordinates of each block in the original image information; and / or, When the cutting configuration instruction is a selection instruction, finding a corresponding preset cutting model from a preset table according to the cutting configuration instruction; and / or, When the cutting configuration instruction is an image recognition instruction, image recognition is performed on the image type information of the original image information, and a prediction segmentation model is generated according to an image recognition rule corresponding to the type information; and / or, When the cutting configuration instruction is a noise recognition instruction, the original image information is denoised to obtain denoised image information and noise image information, noise distribution information is obtained based on the noise image information combined with a clustering algorithm, and a preset cutting model is generated based on the noise distribution information.

9. The image data processing method according to claim 1, characterized in that: When the convolution operation network includes at least three convolution operation units, sending the original sub-image information to the convolution operation network to obtain the target sub-image information includes: The convolution operation unit at the top generates intermediate sub-image information according to the original sub-image and stores it in the on-chip memory; The convolution operation unit in the middle position generates new intermediate sub-image information according to the intermediate sub-image information stored in the on-chip memory by the previous convolution operation unit and stores it in the on-chip memory; The convolution operation unit at the last position generates target sub-image information according to the intermediate sub-image information in the on-chip memory.

10. The image data processing method according to any one of claims 3 or 9, characterized in that: One end of the same on-chip memory is connected to the output end of the previous convolution operation unit, and the other end of the on-chip memory is connected to the input end of the next convolution operation unit.

11. The image data processing method according to any one of claims 3 or 9, characterized in that: The on-chip memory that meets the preset transfer requirements is also used to transfer the intermediate sub-image information of the previous convolution operation unit to the external memory, and the next convolution operation unit reads the intermediate sub-image information from the external memory.

12. A convolutional neural network accelerator, characterized in that: It includes information receiving and sending module, segmentation model configuration module, image cutting module, convolution operation module and on-chip memory; The information transceiver module is used to select the original image information according to the selection instruction; The segmentation model configuration module is used to adjust the specified preset segmentation model according to the overlap area specification to obtain the image segmentation model; The image cutting module is connected to the information transceiving module and the segmentation model configuration module, and is used to read the original image information using the image cutting model to obtain the original sub-image information; The on-chip memory is connected to the image cutting module and the convolution operation module, and is used to store original sub-image information from the image cutting module; The convolution operation module is connected to the on-chip memory and the information transceiver module, and is used to perform convolution operation through a convolution operation network according to the original sub-image information to obtain target sub-image information; the convolution operation module includes a plurality of convolution operation units, and the convolution operation units are used to generate intermediate sub-image information according to the original sub-image information and store it in the on-chip memory, and are also used to generate the target sub-image information according to the intermediate sub-image information in the on-chip memory; The information transceiver module is also connected to the convolution operation module and is used to transfer the target sub-image information from the convolution operation module.

13. The convolutional neural network accelerator according to claim 12, characterized in that: The information transceiver module selects the original image information from the external memory according to the selected instruction; the information transceiver module transfers the target sub-image information from the convolution operation module into the external memory; The target sub-image information is used to merge into the final image information corresponding to the original image information after deleting the overlapping area portion corresponding to the overlapping area specification.

14. The convolutional neural network accelerator according to claim 12, characterized in that: The segmentation model configuration module includes an overlap area debugging unit and a cutting model generation unit; The overlapping area debugging unit is used to debug and obtain the overlapping specification value according to the verification image information and the verification target processing information, combined with the number of network layers and the data volume of the basic image block information; The cutting model generating unit is used to adjust the specified preset cutting model according to the overlapping area specification to obtain the image cutting model; The basic image block is generated by cutting the original image information with a preset segmentation model.

Images