A neural network model firmware generation method and image recognition method

Abstract

The present invention relates to a neural network model firmware generation method and an image recognition method in the field of image processing technology. The method comprises the following steps: configuring several groups of different input resolutions as inputs of a neural network, parsing the neural network, and obtaining model parameters; generating a quantized picture set, executing a forward process on the quantized picture set, obtaining forward quantized data of each layer of the neural network, and acquiring quantized parameter data and fixed-point weight data; calculating the first running memory occupied by each layer of the neural network on an embedded board under each group of input resolutions, and obtaining a first memory occupancy set; configuring multiple groups of neural networks, calculating the second running memory occupied by each group of neural networks on the embedded board under each group of input resolutions, and obtaining a second memory occupancy set; and generating model firmware, which has the advantages of small memory occupancy and high performance, and solves the problem that one network model firmware can process images of multiple image collectors.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a neural network model firmware generation method and an image recognition method. Background Art

[0002] With the rise of smart transportation and smart homes, about 70% of deep learning applications are in image recognition. Combined with the current situation in the security field, with the popularization of big data and high-definition camera applications, the era of big data in security has arrived. Massive high-definition and above resolution video data has brought big data to security product technology, which makes deep learning in the security field inevitably develop rapidly.

[0003] Since the embedded end requires smaller external memory flash and internal memory DDR to save costs, it is particularly important for the network model to generate network firmware process. Under the same model, in different algorithm application scenarios, the smaller the generated firmware, the more cost-effective it is, and the same effect can be achieved.

[0004] In previous implementations, when the algorithm scenario requires detecting objects, faces, human figures, and vehicles at different distances, the image acquisition device of the device collects images with different aspect ratios. In order to save performance, a low-resolution model firmware is used for long-distance detection, and a high-resolution model firmware is used for close-range detection. At this time, different firmware models are stored in the memory, which increases the memory cost. Another solution, in order to save memory costs, generates a network model firmware for the network, uses the same close-range, that is, high-resolution network, and performs edge filling operations on the farthest distance image. This results in poor performance and reduced effects. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a neural network model firmware generation method and an image recognition method, which have the advantages of small memory usage and high performance, and solves the problem that one network model firmware can process images from multiple image collectors.

[0006] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0007] A method for generating model firmware of a neural network comprises the following steps:

[0008] configuring several sets of different input resolutions as inputs to a neural network, and parsing the neural network to obtain model parameters;

[0009] Generate a quantized picture set, perform a forward process on the quantized picture set to obtain forward quantization data of each layer of the neural network, and obtain quantization parameter data and fixed-point weight data based on the forward quantization data;

[0010] Calculate the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set;

[0011] Configuring multiple neural networks with different resolutions, and calculating the amount of second running memory occupied by each neural network on the embedded board under each input resolution, to obtain a second memory occupancy set;

[0012] Model firmware is generated based on the model parameters, the quantization parameter data, the fixed-point weight data, the first memory footprint set, and the second memory footprint set.

[0013] Optionally, generating a quantized picture set and performing a forward process on the quantized picture set to obtain forward quantized data for each layer of the neural network includes the following steps:

[0014] Select a set of neural networks with different input resolutions and perform a forward pass on the quantized image set to obtain the maximum pixel output data value, the maximum weight output data value, and the floating-point data of each layer of the neural network;

[0015] The forward quantization data includes the maximum pixel output data value, the maximum weight output data value and floating point data of each layer of the neural network.

[0016] Optionally, obtaining quantization parameter data and fixed-point weight data based on the forward quantization data includes the following steps:

[0017] Calculating the quantization parameters of each layer of the neural network based on the maximum values ​​of the pixel output data and the maximum values ​​of the weight output data, and summarizing them into quantization parameter data;

[0018] Based on the floating-point data, the fixed-point weights of each layer of the neural network are calculated and summarized into fixed-point weight data.

[0019] Optionally, calculating the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set includes the following steps:

[0020] Calculating a first output resolution of each layer of the neural network based on each set of input resolutions, and calculating a first running memory occupied by each layer of the neural network based on the first output resolution of each layer of the neural network;

[0021] The first running memory occupied by each layer of the neural network is collected to obtain the first memory occupation set.

[0022] Optionally, configuring multiple neural networks and calculating the amount of second running memory occupied by each neural network on the embedded board at each input resolution to obtain a second memory occupancy set includes the following steps:

[0023] Calculating a second output resolution of each group of neural networks based on each group of input resolutions, and calculating a second running memory occupied by each group of neural networks based on the second output resolution of each group of neural networks;

[0024] The second running memory occupied by each group of neural networks is aggregated to obtain a second memory occupation set.

[0025] Optionally, the generating model firmware includes the following steps:

[0026] Generate a binary file, write the model parameters, quantization parameter data, fixed-point weight data, the first memory occupancy set, and the second memory occupancy set into the binary file to obtain model firmware.

[0027] A neural network model firmware generation system includes a parsing unit, a forward execution unit, a first calculation unit, a second calculation unit and a firmware generation unit;

[0028] The parsing unit is used to configure several groups of different input resolutions as inputs of the neural network, and parse the neural network to obtain model parameters;

[0029] The forward execution unit is used to generate a quantized picture set, perform a forward process on the quantized picture set, obtain forward quantization data of each layer of the neural network, and obtain quantization parameter data and fixed-point weight data based on the forward quantization data;

[0030] The first calculation unit is used to calculate the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set;

[0031] The second computing unit is used to configure multiple groups of neural networks with different resolutions, and calculate the second running memory occupied by each group of neural networks on the embedded board under each group of input resolutions to obtain a second memory occupancy set;

[0032] The firmware generation unit is used to generate model firmware based on the model parameters, quantization parameter data, fixed-point weight data, the first memory occupancy set, and the second memory occupancy set.

[0033] An image recognition method, executing any one of the above-mentioned neural network model firmware generation methods to generate model firmware, further comprising the following steps:

[0034] Acquire multiple video images, store and parse the model firmware;

[0035] A forward process is performed on the video image based on the model firmware to obtain an image recognition result.

[0036] Optionally, the following steps are also included:

[0037] The image recognition result is displayed on a terminal device, wherein the terminal device is provided with more than one group.

[0038] Optionally, the following steps are also included:

[0039] A forward process is performed on the video image to perform accelerated processing.

[0040] An image recognition device, comprising one or more image collectors, a compilation processor, an embedded board, a hardware accelerator, a memory, and one or more terminal devices, wherein the memory and the hardware accelerator are both installed in the embedded board, and the number of the image collectors is the same as the number of the terminal devices;

[0041] One or more image collectors are used to collect video images;

[0042] The compiling processor is used to generate a model firmware, and perform a forward process on the video image based on the model firmware to obtain an image recognition result;

[0043] The embedded board is used to parse the firmware model;

[0044] The hardware accelerator is used to perform accelerated processing when the video image performs a forward process;

[0045] The memory is used to store the model firmware;

[0046] One or more terminal devices are used to display image recognition results.

[0047] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0048] By configuring multiple resolution combinations, a single model firmware is generated that shares the same weights, so that the single model firmware can still be used even under multiple image collectors without generating multiple different model firmwares, reducing costs and storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of a method for generating model firmware of a neural network proposed in Example 1;

[0051] Figure 2This is a flow chart of an image recognition method proposed in Example 3;

[0052] Figure 3 This is a module diagram of an image recognition device proposed in Example 4. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below with reference to the examples. The following examples are intended to explain the present invention but the present invention is not limited to the following examples.

[0054] Example 1

[0055] like Figure 1 As shown, a method for generating model firmware of a neural network includes the following steps: configuring several groups of different input resolutions as inputs of the neural network, and parsing the neural network to obtain model parameters, wherein the several groups of different input resolutions refer to the aspect ratios of the resolutions of multiple hardware image collectors, so that the aspect ratio obtained by the height and width of the configured neural network input resolution is consistent with the hardware image collector, thereby making the final model firmware applicable to multiple hardware image collectors with different resolutions, wherein the resolution aspect ratio supported by each hardware image collector is determined by the hardware itself.

[0056] After completing the input resolution configuration of the neural network, it is necessary to analyze the neural network model generated by the algorithm training platform to obtain model parameters, where the model parameters include network basic data and network execution data. The algorithm training platform can be a platform such as caffe, pytorch, tensorflow, etc. In this embodiment, taking the caffe platform as an example, the neural network parsing steps are further explained. Specifically, the maximum resolution among the input resolutions is selected to configure the neural network generated by the caffe platform, and then the neural network is parsed. By configuring the neural network with the maximum input resolution, the obtained model parameters in the final model firmware can meet the resolution requirements of the hardware image acquisition device.

[0057] Furthermore, by parsing the neural network, we can obtain the prototxt file and the caffemodel file, and then further parse the prototxt file and the caffemodel file. Thus, we can obtain the organizational structure of the neural network from the prototxt file, such as the convolutional layer, pooling layer, fully connected layer, residual layer, etc., and we can also obtain the layer order, layer information, etc., among which the layer information includes kernel size, span, grouping, constant size, etc. The floating-point weight files of each neural network layer can be obtained from the caffemodel file. It should be noted that the content parsed from the prototxt file is the file content of the network basic data, and the content parsed from the caffemodel file is the file content of the network execution data.

[0058] Generate a quantized picture set, perform a forward process on the quantized picture set, obtain forward quantization data of each layer of neural network, and obtain quantization parameter data and fixed-point weight data based on the forward quantization data, specifically including the following steps: select a set of neural networks with arbitrary input resolutions, and perform a forward process on the quantized picture set to obtain the maximum pixel output data value, the maximum weight output data value and floating-point data of each layer of neural network, wherein the forward quantization data includes the maximum pixel output data value, the maximum weight output data value and floating-point data of each layer of neural network; calculate the quantization parameters of each layer of neural network based on the maximum pixel output data value and the maximum weight output data value, and summarize them into quantization parameter data; calculate the fixed-point weights of each layer of neural network based on the floating-point data, and summarize them into fixed-point weight data.

[0059] Specifically, the quantized picture set is a certain number of pictures collected under different scenes, and the number of pictures collected can be 50 or 100. The specific number can be changed according to the actual needs of the staff. Then, a set of input resolutions is selected, and the forward process is performed on the quantized picture set, that is, after the quantized picture set is input into the neural network, all quantized pictures are converted into floating-point pixel values ​​and weight values, and then the maximum and minimum values ​​of the pixels of each layer of the neural network are counted, that is, the maximum value of the pixel output data, and the maximum and minimum values ​​of the weights of each layer of the neural network are counted, that is, the maximum value of the weight output data, and the corresponding floating-point values ​​are counted. At this time, the quantization parameters of each layer of the neural network are calculated through the maximum and minimum values ​​of the pixels of each layer of the neural network, and the maximum and minimum values ​​of the weights of each layer of the neural network. The calculation formula of the quantization parameters of each layer of the neural network is as follows:

[0060] Where q represents the quantization parameter, x′ max is the maximum value of the input floating point number, x′ minis the minimum value of the input floating-point number, bw is the bit width of the conversion, for example, currently supports 8-bit and 16-bit bit widths, 8-bit quantization bw is 8, 16-bit quantization bw is 16, max takes the maximum value of the two, min takes the minimum value of the two, is a built-in function of the system library, log2 is a built-in function of the system library, ceil rounds up, abs takes the absolute value, is a built-in function of the system library, x′ m is x′ max and x′ min The absolute maximum value.

[0061] Then calculate the fixed-point weights of each layer of the neural network, where the calculation formula for the fixed-point weights is as follows: d =round(x′ f / pow(2,q)); where x′ f Represents a floating-point value in floating-point data, x′ d is a fixed-point weight, and pow is a built-in function of the system library, which represents the qth power of 2.

[0062] Calculate the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set. Specifically, the method includes the following steps: based on each set of input resolutions, calculate the first output resolution of each layer of the neural network, and calculate the first running memory occupied by each layer of the neural network based on the first output resolution of each layer of the neural network; and aggregate the first running memory occupied by each layer of the neural network to obtain a first memory occupancy set.

[0063] Furthermore, after the forward process is performed on the quantized image set, the width and height of the input resolution of each layer of the neural network can also be obtained. At this time, the first output resolution of each layer of the neural network is calculated by the formula, so as to later confirm the address offset and the occupancy of the board memory. In this embodiment, the calculation of the first output resolution of the convolutional layer is used as an example for explanation.

[0064] Specifically, the calculation formula for the first output resolution of the convolutional layer is as follows: Among them, O w , O h Indicates the width and height of the first output resolution of this layer, I w , O h Indicates the width and height of the input resolution of this layer, K w , K h Represents the width and height of the convolution kernel, P w 、P h Indicates the horizontal and vertical filling of the convolution kernel, S w 、S hRepresents the horizontal and vertical spans of the convolution kernel. It should be noted that different neural network layers use different calculation methods. For specific reference, refer to the calculation methods of the network training platforms caffe, pytorch, and tensorflow.

[0065] After the first output resolution is calculated, the memory occupied is calculated by the formula as follows: M=W*H*C, where W is width, H is height, and C is the number of channels. In actual use, at least two network layers are required according to the neural network. The memory occupied by the two network layers is marked as M1 and M1. The pre-configured network occupies M1+M2 memory. The third network layer is run in sequence to check whether the third network layer and the first network layer have any intersection. If there is no intersection, the memory of the first network layer is reused. When M3 of the third network layer is greater than M1, the memory is reallocated. The memory size occupied by a single network is M1+M2+M3. Other network layers are run in sequence to finally obtain the memory size occupied by a single network.

[0066] Configure multiple groups of neural networks with different resolutions, and calculate the second running memory occupied by each group of neural networks on the embedded board under each group of input resolutions to obtain a second memory occupancy set, specifically including the following steps: based on each group of input resolutions, calculate the second output resolution of each group of neural networks, and based on the second output resolution of each group of neural networks, calculate the second running memory occupied by each group of neural networks; aggregate the second running memory occupied by each group of neural networks to obtain a second memory occupancy set.

[0067] First, at each set of input resolutions, calculate the first output resolution of each layer of a single neural network, then calculate the memory size occupied by the single neural network to obtain the second running memory, and perform calculations on each configured neural network in turn to obtain the memory size occupied by multiple neural networks, and summarize them to obtain the second memory occupation set. Count the maximum memory of the second memory occupation set, and use the maximum memory as the second memory occupation size.

[0068] Model firmware is generated based on model parameters, quantization parameter data, fixed-point weight data, a first memory occupancy set, and a second memory occupancy set, wherein generating the model firmware includes the following steps: generating a binary file, writing the model parameters, quantization parameter data, fixed-point weight data, the first memory occupancy set, and the second memory occupancy set into the binary file to obtain the model firmware.

[0069] Specifically, since the model firmware represents a binary universal file that can be recognized by a computer, the model parameters, quantization parameter data, fixed-point weight data, the first memory occupation set, and the second memory occupation set are mainly saved in a binary file in byte format. The saving order is as follows: first save the 64-word firmware header, including the network name, check code, scope, etc., then save the first layer input quantization parameters of the neural network, the quantization parameters of the last output layer of the neural network, the resolution height and width combination, and the selector parameter default maximum resolution configuration to facilitate algorithm application configuration. Finally, save the network name of each layer, the output resolution height and width combination, the execution order association, the quantization parameters, the fixed-point weight data, the first memory occupation and the second memory occupation, so that the final model firmware can achieve the purpose of multi-resolution combination and share a weight. Specifically, on the one hand, the weight of a neural network is fixed, indicating that the weight quantization parameter will not change. On the other hand, different resolution height and width combinations are always controlled at the same resolution, and the maximum and minimum values ​​of the image pixel values ​​remain unchanged. The data quantization parameters generated at the same resolution are applicable to all different resolutions, thereby saving storage space.

[0070] Example 2

[0071] A neural network model firmware generation system includes a parsing unit, a forward execution unit, a first computing unit, a second computing unit and a firmware generation unit. The parsing unit is used to configure several groups of different input resolutions as inputs of the neural network, and parse the neural network to obtain model parameters, wherein the several groups of different input resolutions refer to the aspect ratios of the resolutions of multiple hardware image collectors, so that the aspect ratio obtained by the height and width of the configured neural network input resolution is consistent with the hardware image collector, thereby making the final model firmware applicable to multiple hardware image collectors with different resolutions, wherein the resolution aspect ratio supported by each hardware image collector is determined by the hardware itself.

[0072] After completing the input resolution configuration of the neural network, it is necessary to analyze the neural network model generated by the algorithm training platform to obtain model parameters, where the model parameters include network basic data and network execution data. The algorithm training platform can be a platform such as caffe, pytorch, tensorflow, etc. In this embodiment, taking the caffe platform as an example, the neural network parsing steps are further explained. Specifically, the maximum resolution among the input resolutions is selected to configure the neural network generated by the caffe platform, and then the neural network is parsed. By configuring the neural network with the maximum input resolution, the obtained model parameters in the final model firmware can meet the resolution requirements of the hardware image acquisition device.

[0073] Furthermore, by parsing the neural network, we can obtain the prototxt file and the caffemodel file, and then further parse the prototxt file and the caffemodel file. Thus, we can obtain the organizational structure of the neural network from the prototxt file, such as the convolutional layer, pooling layer, fully connected layer, residual layer, etc., and we can also obtain the layer order, layer information, etc., among which the layer information includes kernel size, span, grouping, constant size, etc. The floating-point weight files of each neural network layer can be obtained from the caffemodel file. It should be noted that the content parsed from the prototxt file is the file content of the network basic data, and the content parsed from the caffemodel file is the file content of the network execution data.

[0074] The forward execution unit is used to generate a quantized picture set, execute a forward process on the quantized picture set, obtain forward quantization data of each layer of the neural network, and obtain quantization parameter data and fixed-point weight data based on the forward quantization data. The forward execution unit specifically includes the following steps: select a set of neural networks with an input resolution, and execute a forward process on the quantized picture set to obtain the maximum pixel output data value, the maximum weight output data value and the floating-point data of each layer of the neural network, wherein the forward quantization data includes the maximum pixel output data value, the maximum weight output data value and the floating-point data of each layer of the neural network; calculate the quantization parameters of each layer of the neural network based on the maximum pixel output data value and the maximum weight output data value, and summarize them into quantization parameter data; calculate the fixed-point weights of each layer of the neural network based on the floating-point data, and summarize them into fixed-point weight data.

[0075] Specifically, the quantized picture set is a certain number of pictures collected under different scenes, and the number of pictures collected can be 50 or 100. The specific number can be changed according to the actual needs of the staff. Then, a set of input resolutions is selected, and the forward process is performed on the quantized picture set, that is, after the quantized picture set is input into the neural network, all quantized pictures are converted into floating-point pixel values ​​and weight values, and then the maximum and minimum values ​​of the pixels of each layer of the neural network are counted, that is, the maximum value of the pixel output data, and the maximum and minimum values ​​of the weights of each layer of the neural network are counted, that is, the maximum value of the weight output data, and the corresponding floating-point values ​​are counted. At this time, the quantization parameters of each layer of the neural network are calculated through the maximum and minimum values ​​of the pixels of each layer of the neural network, and the maximum and minimum values ​​of the weights of each layer of the neural network. The calculation formula of the quantization parameters of each layer of the neural network is as follows:

[0076] Where q represents the quantization parameter, x′ max is the maximum value of the input floating point number, x′ minis the minimum value of the input floating-point number, bw is the bit width of the conversion, for example, currently supports 8-bit and 16-bit bit widths, 8-bit quantization bw is 8, 16-bit quantization bw is 16, max takes the maximum value of the two, min takes the minimum value of the two, is a built-in function of the system library, log2 is a built-in function of the system library, ceil rounds up, abs takes the absolute value, is a built-in function of the system library, x′ m is x′ max and x′ min The absolute maximum value.

[0077] Then calculate the fixed-point weights of each layer of the neural network, where the calculation formula for the fixed-point weights is as follows: d =round(x′ f / pow(2,q)); where x′ f Represents a floating-point value in floating-point data, x′ d is a fixed-point weight, and pow is a built-in function of the system library, which represents the qth power of 2.

[0078] The first calculation unit is used to calculate the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set. Specifically, the following steps are included: based on each set of input resolutions, calculating the first output resolution of each layer of the neural network, and based on the first output resolution of each layer of the neural network, calculating the first running memory occupied by each layer of the neural network; and collecting the first running memory occupied by each layer of the neural network to obtain a first memory occupancy set.

[0079] Furthermore, after the forward process is performed on the quantized image set, the width and height of the input resolution of each layer of the neural network can also be obtained. At this time, the first output resolution of each layer of the neural network is calculated by the formula, so as to later confirm the address offset and the occupancy of the board memory. In this embodiment, the calculation of the first output resolution of the convolutional layer is used as an example for explanation.

[0080] Specifically, the calculation formula for the first output resolution of the convolutional layer is as follows: Among them, O w , O h Indicates the width and height of the first output resolution of this layer, I w , O h Indicates the width and height of the input resolution of this layer, K w , K h Represents the width and height of the convolution kernel, P w 、P h Indicates the horizontal and vertical filling of the convolution kernel, S w 、S hRepresents the horizontal and vertical spans of the convolution kernel. It should be noted that different neural network layers use different calculation methods. For specific reference, refer to the calculation methods of the network training platforms caffe, pytorch, and tensorflow.

[0081] After the first output resolution is calculated, the memory occupied is calculated by the formula as follows: M=W*H*C, where W is width, H is height, and C is the number of channels. In actual use, at least two network layers are required according to the neural network. The memory occupied by the two network layers is marked as M1 and M1. The pre-configured network occupies M1+M2 memory. The third network layer is run in sequence to check whether the third network layer and the first network layer have any intersection. If there is no intersection, the memory of the first network layer is reused. When M3 of the third network layer is greater than M1, the memory is reallocated. The memory size occupied by a single network is M1+M2+M3. Other network layers are run in sequence to finally obtain the memory size occupied by a single network.

[0082] The second computing unit is used to configure multiple groups of neural networks with different resolutions, and calculate the second running memory occupied by each group of neural networks on the embedded board under each group of input resolutions to obtain a second memory occupancy set, specifically including the following steps: based on each group of input resolutions, calculating the second output resolution of each group of neural networks, and based on the second output resolution of each group of neural networks, calculating the second running memory occupied by each group of neural networks; and aggregating the second running memory occupied by each group of neural networks to obtain a second memory occupancy set.

[0083] First, at each set of input resolutions, calculate the first output resolution of each layer of a single neural network, then calculate the memory size occupied by the single neural network to obtain the second running memory, and perform calculations on each configured neural network in turn to obtain the memory size occupied by multiple neural networks, and summarize them to obtain the second memory occupation set. Count the maximum memory of the second memory occupation set, and use the maximum memory as the second memory occupation size.

[0084] The firmware generation unit is used to generate model firmware based on model parameters, quantization parameter data, fixed-point weight data, a first memory occupancy set, and a second memory occupancy set, wherein generating the model firmware includes the following steps: generating a binary file, writing the model parameters, quantization parameter data, fixed-point weight data, the first memory occupancy set, and the second memory occupancy set into the binary file to obtain the model firmware.

[0085] Specifically, since the model firmware represents a binary universal file that can be recognized by a computer, the model parameters, quantization parameter data, fixed-point weight data, the first memory occupation set, and the second memory occupation set are mainly saved in a binary file in byte format. The saving order is as follows: first save the 64-word firmware header, including the network name, check code, scope, etc., then save the first layer input quantization parameters of the neural network, the quantization parameters of the last output layer of the neural network, the resolution height and width combination, and the selector parameter default maximum resolution configuration to facilitate algorithm application configuration. Finally, save the network name of each layer, the output resolution height and width combination, the execution order association, the quantization parameters, the fixed-point weight data, the first memory occupation and the second memory occupation, so that the final model firmware can achieve the purpose of multi-resolution combination and share a weight. Specifically, on the one hand, the weight of a neural network is fixed, indicating that the weight quantization parameter will not change. On the other hand, different resolution height and width combinations are always controlled at the same resolution, and the maximum and minimum values ​​of the image pixel values ​​remain unchanged. The data quantization parameters generated at the same resolution are applicable to all different resolutions, thereby saving storage space.

[0086] Example 3

[0087] An image recognition method executes the model firmware generation method of the neural network as described in Example 1 to generate the model firmware, and further includes the following steps: acquiring multiple video images, storing and parsing the model firmware; performing a forward process on the video images based on the model firmware to obtain image recognition results, and further includes the following steps: displaying the image recognition results on a terminal device, wherein the terminal device is provided with more than one set; and performing accelerated processing on the video images by performing a forward process.

[0088] Specifically, the model firmware is first generated according to the method of Example 1, and then the CPU loads the model firmware and stores it in the memory. The video image CPU obtains the video stream from the acquisition module. At this time, if the acquired video image needs to be recognized, the model firmware must be parsed first, and then the parsed model firmware is used to perform a forward process on each frame of the acquired video to obtain the recognition result. Furthermore, the parsing of the model firmware file is to select a long-distance, medium-distance or short-distance mode according to the algorithm scenario, collect video streams in different modes, use a hardware accelerator to accelerate the forward process, and finally analyze the image recognition result and display it.

[0089] This embodiment takes human figure detection as an example for specific explanation. It is set that the hardware device for detection has three image collectors, and the three image collectors correspond to three channels to collect image frames. The three image collectors are equipped with different resolutions, namely, the long-distance image resolution is 1080P (width 1920, height 1080), and the aspect ratio is 1.777; the medium-distance image resolution is 1080 width, 720 height, and the aspect ratio is 1.50; the close-range image resolution is 720 width, 567 height, and the aspect ratio is 1.27. Correspondingly, three displays are required to display the three-channel images.

[0090] At this time, according to the three image collectors obtained, the input resolutions in the neural network model are configured as follows: one group with a width of 110 and a height of 62, with an aspect ratio of 1.777; one group with a width of 360 and a height of 240, with an aspect ratio of 1.50; and one group with a width of 720 and a height of 567, with an aspect ratio of 1.27. Then, the compiler generates a model firmware for detecting human figures according to the steps in Example 1 based on a set of 50 quantized human figures in different scenes.

[0091] On the other hand, the CPU on the embedded board performs channel scheduling management, using three threads to process each input video image. Different resolutions are selected according to the configuration selector, and the same model firmware is reused. Because only one hardware accelerator is set, the three channels need to be queued and executed in sequence. After obtaining the results, they are displayed separately on the monitor.

[0092] Example 4

[0093] like Figure 3 As shown, an image recognition device executes the image recognition method as described in Example 3, and the image recognition device includes one or more image collectors, a compilation processor, an embedded board, a hardware accelerator, a memory and one or more terminal devices, the memory and the hardware accelerator are both installed in the embedded board, and the number of image collectors is the same as the number of terminal devices; one or more image collectors are used to collect video images; the compilation processor is used to generate model firmware, and execute a forward process on the video image based on the model firmware to obtain an image recognition result, that is, the compilation processor has the function of compiling model firmware; the embedded board is used to parse the firmware model; the hardware accelerator is used to accelerate processing when the video image executes the forward process; the memory is used to store the model firmware; and one or more terminal devices are used to display the image recognition result.

[0094] Specifically, one or more image collectors are set on the same hardware device to obtain video streams and store them in DDR (internal memory). The embedded board CPU parses the model firmware file, reads the model firmware information, reads the video stream in DDR, uses the hardware accelerator, executes the neural network forward process, and outputs the image recognition result. Furthermore, the CPU of the embedded board reads the model firmware file, selects a long-distance, medium-distance or short-distance mode according to the algorithm scenario, thereby determining the currently running resolution firmware, and then the hardware accelerator accelerates the forward process, and then the CPU analyzes the image recognition result and displays it through the terminal device. The terminal device referred to in this embodiment is a hardware device with a display function, and the specific model is not limited here.

[0095] This embodiment takes human figure detection as an example for specific explanation. It is set that the hardware device for detection has three image collectors, and the three image collectors correspond to three channels to collect image frames. The three image collectors are equipped with different resolutions, namely, the long-distance image resolution is 1080P (width 1920, height 1080), and the aspect ratio is 1.777; the medium-distance image resolution is 1080 width, 720 height, and the aspect ratio is 1.50; the close-range image resolution is 720 width, 567 height, and the aspect ratio is 1.27. Correspondingly, three displays are required to display the three-channel images.

[0096] At this time, according to the three image collectors obtained, the input resolutions in the neural network model are configured as follows: a group of 110 width and 62 height, with an aspect ratio of 1.777; a group of 360 width and 240 height, with an aspect ratio of 1.50; a group of 720 width and 567 height, with an aspect ratio of 1.27. Then, the compilation processor generates a model firmware for detecting human figures according to the steps in Example 1 based on a set of 50 quantized human figures in different scenes.

[0097] On the other hand, the CPU on the embedded board performs channel scheduling management, using three threads to process each input video image. Different resolutions are selected according to the configuration selector, and the same model firmware is reused. Because only one hardware accelerator is set, the three channels need to be queued and executed in sequence. After obtaining the results, they are displayed separately on the monitor.

[0098] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, any one of the above methods for intercepting dirty data is executed.

[0099] More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wire segments, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, in this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, electrical, optical, RF, or any suitable combination thereof.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules, subassemblies, or units is merely a logical functional division. In actual implementation, other division methods may be used, such as multiple units, subassemblies, or components can be combined or integrated into another device, or some features can be ignored or not performed.

[0102] The units may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0103] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0104] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0105] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention shall be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for generating model firmware of a neural network, characterized in that: The following steps are involved: configuring several sets of different input resolutions as inputs to a neural network, and parsing the neural network to obtain model parameters; Generate a quantized picture set, perform a forward process on the quantized picture set to obtain forward quantization data of each layer of the neural network, and obtain quantization parameter data and fixed-point weight data based on the forward quantization data; Calculate the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set; Configuring multiple neural networks with different resolutions, and calculating the amount of second running memory occupied by each neural network on the embedded board under each input resolution, to obtain a second memory occupancy set; Model firmware is generated based on the model parameters, the quantization parameter data, the fixed-point weight data, the first memory footprint set, and the second memory footprint set.

2. The method for generating a neural network model firmware according to claim 1, wherein: Generating a quantized picture set, performing a forward process on the quantized picture set, and obtaining forward quantized data of each layer of the neural network, including the following steps: Select a set of neural networks with different input resolutions and perform a forward pass on the quantized image set to obtain the maximum pixel output data value, the maximum weight output data value, and the floating-point data of each layer of the neural network; The forward quantization data includes the maximum pixel output data value, the maximum weight output data value and floating point data of each layer of the neural network.

3. The method for generating a neural network model firmware according to claim 2, wherein: Acquiring quantization parameter data and fixed-point weight data based on the forward quantization data includes the following steps: Calculating the quantization parameters of each layer of the neural network based on the maximum values ​​of the pixel output data and the maximum values ​​of the weight output data, and summarizing them into quantization parameter data; Based on the floating-point data, the fixed-point weights of each layer of the neural network are calculated and summarized into fixed-point weight data.

4. The method for generating a neural network model firmware according to claim 1, wherein: Calculating the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain the first memory occupation set includes the following steps: Calculating a first output resolution of each layer of the neural network based on each set of input resolutions, and calculating a first running memory occupied by each layer of the neural network based on the first output resolution of each layer of the neural network; The first running memory occupied by each layer of the neural network is collected to obtain the first memory occupation set.

5. The method for generating a neural network model firmware according to claim 1, wherein: Configuring multiple neural networks and calculating the amount of second memory occupied by each neural network on the embedded board at each input resolution to obtain a second memory occupancy set includes the following steps: Calculating a second output resolution of each group of neural networks based on each group of input resolutions, and calculating a second running memory occupied by each group of neural networks based on the second output resolution of each group of neural networks; The second running memory occupied by each group of neural networks is aggregated to obtain a second memory occupation set.

6. The method for generating a neural network model firmware according to claim 1, wherein: The generation model firmware includes the following steps: Generate a binary file, write the model parameters, quantization parameter data, fixed-point weight data, the first memory occupancy set, and the second memory occupancy set into the binary file to obtain model firmware.

7. A neural network model firmware generation system, characterized in that: It includes a parsing unit, a forward execution unit, a first calculation unit, a second calculation unit and a firmware generation unit; The parsing unit is used to configure several groups of different input resolutions as inputs of the neural network, and parse the neural network to obtain model parameters; The forward execution unit is used to generate a quantized picture set, perform a forward process on the quantized picture set to obtain forward quantization data of each layer of the neural network, and obtain quantization parameter data and fixed-point weight data based on the forward quantization data; The first calculation unit is used to calculate the first running memory occupied by each layer of the neural network on the embedded board under each set of input resolutions to obtain a first memory occupancy set; The second computing unit is used to configure multiple groups of neural networks with different resolutions, and calculate the second running memory occupied by each group of neural networks on the embedded board under each group of input resolutions to obtain a second memory occupancy set; The firmware generation unit is used to generate model firmware based on the model parameters, quantization parameter data, fixed-point weight data, the first memory occupancy set, and the second memory occupancy set.

8. An image recognition method, characterized in that: Executing the neural network model firmware generation method according to any one of claims 1 to 6 to generate the model firmware further comprises the following steps: Acquire multiple video images, store and parse the model firmware; A forward process is performed on the video image based on the model firmware to obtain an image recognition result.

9. An image recognition method according to claim 8, characterized in that: The following steps are also included: The image recognition result is displayed on a terminal device, wherein the terminal device is provided with more than one group.

10. The image recognition method according to claim 8, characterized in that: The following steps are also included: A forward process is performed on the video image to perform accelerated processing.

11. An image recognition device, characterized in that: The system comprises one or more image collectors, a compilation processor, an embedded board, a hardware accelerator, a memory, and one or more terminal devices, wherein the memory and the hardware accelerator are installed in the embedded board, and the number of the image collectors is the same as the number of the terminal devices; One or more image collectors are used to collect video images; The compiling processor is used to generate a model firmware, and perform a forward process on the video image based on the model firmware to obtain an image recognition result; The embedded board is used to parse the firmware model; The hardware accelerator is used to perform accelerated processing when the video image performs a forward process; The memory is used to store the model firmware; One or more terminal devices are used to display image recognition results.

Images