An address arrangement method based on a convolutional neural network

Abstract

The application relates to a convolutional neural network-based address arrangement method and belongs to the technical field of deep learning. The method comprises the following steps: traversing a neural network structure model according to the node order of a directed acyclic graph to extract structure information. Information integration is carried out according to the type of the structure information to obtain a data dictionary. Layer garbage collection verification is carried out, layer data meeting the recycling conditions are deleted, and the remaining layers are marked as valid layers. The next layer of the valid layer is verified, and sufficient memory space is reserved for special layers. The valid layer is addressed in the memory module to obtain a memory pointer position. The start-end position difference of the valid layer is calculated, and the address data size pointed to by the memory pointer is judged. The memory pointer is moved to the end position of the valid layer, and memory address allocation is carried out on the valid layer data according to the accurate address position of the pointer. Therefore, address arrangement with a relatively high address reuse rate is realized for most target detection network structures, and the method is universal for different network structures.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to an address arrangement method based on a convolutional neural network. Background Technology

[0002] Neural network address mapping refers to the process of mapping the various layers and neurons of a neural network to hardware resources (such as processors, memory, etc.) according to certain rules, so as to facilitate the deployment and application of neural network models on hardware.

[0003] Currently, the main methods for address arrangement in neural networks include weight address arrangement methods for convolutional neural networks suitable for parallel computing architectures. The main process includes: a memory arrangement optimization unit arranges the weights in memory according to the computation order; a task splitting control unit splits the neural network computation task into multiple executable subtasks; and a direct memory read / write control unit moves the weight coefficients from memory to the multi-core processor. In convolutional neural network parallel computing architectures, the need to split tasks into different processing cores according to the output channel necessitates designing an efficient memory arrangement method for weight coefficients across multiple tasks. This can significantly improve task execution efficiency, reduce on-chip storage space pressure, ensure continuous weight reading, reduce memory read latency, and optimize computation.

[0004] However, none of the currently published patents describe the address arrangement method for each convolutional layer in a convolutional neural network. The address arrangement methods in the currently published patents, such as the weight arrangement method applicable to parallel computing, have high hardware requirements and special structures, and are not universally applicable to address arrangements in other chips. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides an address arrangement method based on a convolutional neural network, implemented through the following steps:

[0006] S1: Obtain the neural network structure model, traverse the neural network structure model according to the node order defined by the directed acyclic graph, and extract the structural information of the neural network structure model;

[0007] S2: Based on the structural type of the structural information, integrate the layer data in the structural information to obtain a data dictionary;

[0008] S3: Perform garbage collection verification on the layer data according to the data dictionary, delete the layer data that meets the collection conditions, set the garbage collection flag of the layer data to 1, and mark the remaining layer data as valid layer data.

[0009] S4: Obtain the next_concat_name_list of the effective layer data. When the length of the next_concat_name_list is greater than 1, reserve memory space for the output size of all convolutional layers in the next_concat_name_list for the effective layer data. The next_concat_name_list is the list of names of the next connection layer of the effective layer data.

[0010] S5: When the length of the next_concat_name_list is not greater than 1, the memory pointer is calculated by performing memory addressing on the effective layer data in the memory module according to the characteristic end address of the previous layer of the effective layer data;

[0011] S6: Calculate the difference between the beginning and end positions of the effective layer data, and make a size judgment on the address data pointed to by the calculated position of the memory pointer based on the difference between the beginning and end positions. When the size judgment is correct, the memory pointer is used as the accurate address position of the effective layer data.

[0012] S7: Move the memory pointer to the end of the valid layer data, and allocate memory addresses for the valid layer data according to the accurate address position of the pointer.

[0013] Specifically, the structure types in S2 include: dense block, residual block, fpn, and inception block. The dense block represents a densely connected block structure, the residual block represents a residual block structure, the fpn represents a feature pyramid structure, and the inception block represents an inception block structure. The layer data includes convolutional layers and connection layers.

[0014] Specifically, when the structure type in S2 is the dense block and the inception block, the data dictionary includes: concat_info_dict, conv_list, stride_dict, concat_conv_offset_dict, and reference_count_dict;

[0015] When the structure type is the residual block, the data dictionary includes: resnet_layer_config_dict;

[0016] When the structure type is fpn, the data dictionary includes: fpn_horizontal_dict. The meanings of the data dictionary involved in S2 are shown in the following table:

[0017]

[0018] Specifically, the garbage collection verification in S3 includes:

[0019] When the structure type is the dense block and the inception block and the layer data is the connection layer, the reference count of the layer data in the reference_count_dict is decremented by 1, and the garbage collection flag is set to 0;

[0020] When the structure type is the residual block and the fpn and the layer data is the connection layer, decrement the reference count of the layer data in the reference_count_dict by 1;

[0021] When the structure type is the dense block and the inception block and the layer data is the convolutional layer, the reference count of the layer data in the reference_count_dict is decremented by 1;

[0022] When the reference count in the reference_count_dict is 0 and the garbage collection flag is empty, the layer data corresponding to the reference count is deleted, the garbage collection flag of the layer data is set to 1, and the remaining layer data is marked as valid layer data.

[0023] Specifically, S5 includes:

[0024] Obtain the feature end address of the previous layer of the valid layer data. When the feature end address is not 0, use the feature end address as the memory pointer.

[0025] When the end address of the feature is 0 and the remaining space of the memory module is 0, the memory pointer is marked as 0;

[0026] When the end address of the feature is 0 and the remaining space of the memory module is not 0, the end address of the upper-level feature is obtained until the end address of the upper-level feature is not 0, and then the end address of the upper-level feature is used as the memory pointer.

[0027] Specifically, S6 includes:

[0028] When the memory module occupancy is 0, the memory pointer is used as the accurate address location of the valid layer data;

[0029] When the memory module's occupancy is not zero, the remaining space of the memory module is compared with the difference between the beginning and end positions. When the remaining space is greater than the difference between the beginning and end positions, the memory pointer is used as the accurate address position of the valid layer data. When the remaining space is less than the difference between the beginning and end positions, the upper-layer feature end address of the feature end address is obtained. This process continues until the remaining space is greater than or equal to the difference between the beginning and end positions, at which point the upper-layer feature end address is used as the accurate address position.

[0030] Specifically, the formula for calculating the pointer movement position in S7 is as follows:

[0031]

[0032] Where addr is the current position of the memory pointer, base is the base address, offset is the size of the feature map of the effective layer data, and addr_size is the size of the address data. When this formula does not hold, the memory pointer is moved to the following position:

[0033]

[0034] The formula for calculating the size of the feature map is:

[0035] offset = featuremap = width × height × channel, where width, height, and channel are the length, width, and channel labels, respectively.

[0036] The beneficial effects of this invention are as follows:

[0037] It can perform address arrangement with high address reuse rate for most target detection network structures and has universality for different network structures. Attached Figure Description

[0038] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0039] Figure 1 This is a schematic diagram of the address arrangement method based on convolutional neural networks in this invention;

[0040] Figure 2 This is a schematic diagram illustrating the extraction and integration of data dictionaries based on different structure types in this invention;

[0041] Figure 3This is a flowchart illustrating the garbage collection verification, next-link layer verification, memory addressing, and address data size determination processes in this invention. Detailed Implementation

[0042] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0043] Please see Figure 1 The address arrangement method based on convolutional neural networks includes the following steps:

[0044] S1: Obtain the neural network structure model, traverse the neural network structure model according to the node order defined by the directed acyclic graph, and extract the structural information of the neural network structure model;

[0045] S2: Based on the structural type of the structural information, integrate the layer data in the structural information to obtain a data dictionary, such as... Figure 2 As shown;

[0046] S3: Perform garbage collection verification on the layer data according to the data dictionary, delete the layer data that meets the collection conditions, set the garbage collection flag of the layer data to 1, and mark the remaining layer data as valid layer data.

[0047] S4: Obtain the next_concat_name_list of the valid layer data. When the length of the next_concat_name_list is greater than 1, reserve memory space for the output size of all convolutional layers in the next_concat_name_list for the valid layer data. The next_concat_name_list is the list of names of the next connection layer of the valid layer data.

[0048] S5: When the length of the next_concat_name_list is not greater than 1, the memory pointer is calculated by performing memory addressing on the effective layer data in the memory module according to the characteristic end address of the previous layer of the effective layer data;

[0049] S6: Calculate the difference between the beginning and end positions of the effective layer data, and make a size judgment on the address data pointed to by the calculated position of the memory pointer based on the difference between the beginning and end positions. When the size judgment is correct, the memory pointer is used as the accurate address position of the effective layer data.

[0050] S7: Move the memory pointer to the end of the valid layer data, and allocate memory addresses for the valid layer data according to the accurate address position of the pointer.

[0051] Specifically, the structure types in S2 include: dense block, residual block, fpn, and inception block. The dense block represents a densely connected block structure, the residual block represents a residual block structure, the fpn represents a feature pyramid structure, and the inception block represents an inception block structure. The layer data includes convolutional layers and connection layers.

[0052] Specifically, when the structure type in S2 is the dense block and the inception block, the data dictionary includes: concat_info_dict, conv_list, stride_dict, concat_conv_offset_dict, and reference_count_dict;

[0053] When the structure type is the residual block, the data dictionary includes: resnet_layer_config_dict;

[0054] When the structure type is fpn, the data dictionary includes: fpn_horizontal_dict.

[0055] Please see Figure 2 The garbage collection verification in S3 specifically includes:

[0056] When the structure type is the dense block and the inception block and the layer data is the connection layer, the reference count of the layer data in the reference_count_dict is decremented by 1, and the garbage collection flag is set to 0;

[0057] When the structure type is the residual block and the fpn and the layer data is the connection layer, decrement the reference count of the layer data in the reference_count_dict by 1;

[0058] When the structure type is the dense block and the inception block and the layer data is the convolutional layer, the reference count of the layer data in the reference_count_dict is decremented by 1;

[0059] When the reference count in the reference_count_dict is 0 and the garbage collection flag is empty, the layer data corresponding to the reference count is deleted, the garbage collection flag of the layer data is set to 1, and the remaining layer data is marked as valid layer data.

[0060] Specifically, S5 includes:

[0061] Obtain the feature end address of the previous layer of the valid layer data. When the feature end address is not 0, use the feature end address as the memory pointer.

[0062] When the end address of the feature is 0 and the remaining space of the memory module is 0, the memory pointer is marked as 0;

[0063] When the end address of the feature is 0 and the remaining space of the memory module is not 0, the end address of the upper-level feature is obtained until the end address of the upper-level feature is not 0, and then the end address of the upper-level feature is used as the memory pointer.

[0064] Specifically, S6 includes:

[0065] When the memory module occupancy is 0, the memory pointer is used as the accurate address location of the valid layer data;

[0066] When the memory module's occupancy is not zero, the remaining space of the memory module is compared with the difference between the beginning and end positions. When the remaining space is greater than the difference between the beginning and end positions, the memory pointer is used as the accurate address position of the valid layer data. When the remaining space is less than the difference between the beginning and end positions, the upper-layer feature end address of the feature end address is obtained. This process continues until the remaining space is greater than or equal to the difference between the beginning and end positions, at which point the upper-layer feature end address is used as the accurate address position.

[0067] Specifically, the formula for calculating the pointer movement position in S7 is as follows:

[0068]

[0069] Where addr is the current position of the memory pointer, base is the base address, offset is the size of the feature map of the effective layer data, and addr_size is the size of the address data. When this formula does not hold, the memory pointer is moved to the following position:

[0070]

[0071] The formula for calculating the size of the feature map is:

[0072] offset=featuremap=width×height×channel,

[0073] Among them, width, height, and channel are the length, width, and channel labels, respectively.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An address arrangement method based on a convolutional neural network, characterized in that, Includes the following steps: S1: Obtain the neural network structure model, traverse the neural network structure model according to the node order defined by the directed acyclic graph, and extract the structural information of the neural network structure model; S2: Based on the structural type of the structural information, integrate the layer data in the structural information to obtain a data dictionary; S3: Perform garbage collection verification on the layer data according to the data dictionary, delete the layer data that meets the collection conditions, set the garbage collection flag of the layer data to 1, and mark the remaining layer data as valid layer data. S4: Obtain the next_concat_name_list of the valid layer data. When the length of the next_concat_name_list is greater than 1, reserve memory space for the output size of all convolutional layers in the next_concat_name_list for the valid layer data. The next_concat_name_list is the list of names of the next connection layer of the valid layer data. S5: When the length of the next_concat_name_list is not greater than 1, the memory pointer is calculated by performing memory addressing on the effective layer data in the memory module according to the characteristic end address of the previous layer of the effective layer data; S6: Calculate the difference between the beginning and end positions of the effective layer data, and make a size judgment on the address data pointed to by the calculated position of the memory pointer based on the difference between the beginning and end positions. When the size judgment is correct, the memory pointer is used as the accurate address position of the effective layer data. S7: Move the memory pointer to the end of the valid layer data, and allocate memory addresses for the valid layer data according to the accurate address position of the pointer.

2. The method according to claim 1, characterized in that, The structure types in S2 include: denseblock, residualblock, fpn, and inceptionblock. Denseblock represents a densely connected block structure, residualblock represents a residual block structure, fpn represents a feature pyramid structure, and inceptionblock represents an inception block structure. The layer data includes convolutional layers and connection layers.

3. The method according to claim 2, characterized in that, When the structure type in S2 is the denseblock and the inceptionblock, the data dictionary includes: concat_info_dict, conv_list, stride_dict, concat_conv_offset_dict, and reference_count_dict; When the structure type is the residual block, the data dictionary includes: resnet_layer_config_dict; When the structure type is fpn, the data dictionary includes: fpn_horizontal_dict.

4. The method according to claim 3, characterized in that, The garbage collection verification in S3 specifically includes: When the structure type is the denseblock and the inceptionblock and the layer data is the connection layer, the reference count of the layer data in the reference_count_dict is decremented by 1, and the garbage collection flag is set to 0; When the structure type is the residualblock and the fpn and the layer data is the connection layer, the reference count of the layer data in the reference_count_dict is decremented by 1; When the structure type is the denseblock and the inceptionblock and the layer data is the convolutional layer, the reference count of the layer data in the reference_count_dict is decremented by 1; When the reference count in the reference_count_dict is 0 and the garbage collection flag is empty, the layer data corresponding to the reference count is deleted, the garbage collection flag of the layer data is set to 1, and the remaining layer data is marked as valid layer data.

5. The method according to claim 3, characterized in that, S5 specifically includes: Obtain the feature end address of the previous layer of the valid layer data. When the feature end address is not 0, use the feature end address as the memory pointer. When the end address of the feature is 0 and the remaining space of the memory module is 0, the memory pointer is marked as 0; When the end address of the feature is 0 and the remaining space of the memory module is not 0, the end address of the upper-level feature is obtained until the end address of the upper-level feature is not 0, and then the end address of the upper-level feature is used as the memory pointer.

6. The method according to claim 5, characterized in that, S6 specifically includes: When the memory module occupancy is 0, the memory pointer is used as the accurate address location of the valid layer data; When the memory module's occupancy is not zero, the remaining space of the memory module is compared with the difference between the beginning and end positions. When the remaining space is greater than the difference between the beginning and end positions, the memory pointer is used as the accurate address position of the valid layer data. When the remaining space is less than the difference between the beginning and end positions, the upper-layer feature end address of the feature end address is obtained. This process continues until the remaining space is greater than or equal to the difference between the beginning and end positions, at which point the upper-layer feature end address is used as the accurate address position.

7. The method according to claim 1, characterized in that, The calculation formula for the pointer movement position in S7 is as follows: Where addr is the current position of the memory pointer, base is the base address, offset is the size of the feature map of the effective layer data, and addr_size is the size of the address data. When this formula does not hold, the memory pointer is moved to the following position: The formula for calculating the size of the feature map is: offset=featuremap=width×height×channel, Among them, width, height, and channel are the length, width, and channel labels, respectively.

Images