A method, apparatus, device, and storage medium for neural network hardware and software collaborative detection.

Abstract

The application discloses a neural network software and hardware cooperative detection method and device, equipment and a storage medium, and relates to the technical field of deep learning, which comprises the following steps: inputting a first target image into a pre-trained neural network model for forward operation, and calculating the output resolution of the current network layer of the neural network; replacing the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution; after the replacement of all the network layer output resolutions stored in the network firmware is completed, reconfiguring the register command word in the network firmware, and starting the neural network hardware to accelerate detection. In the application, the resolution of the input neural network is variable, which can ensure that there is a detection object under the current resolution, and because the network resolution is smaller, the un-scaled image is clearer, the network performance is improved, and the bandwidth usage is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a neural network software and hardware cooperative detection method and device, equipment and a storage medium. BACKGROUND

[0002] The rise of intelligent transportation and smart home makes deep learning mainly applied in image recognition. With the popularization of big data and high-definition cameras, the security big data era has come, and the massive high-definition and above resolution videos bring big data to security technology, which makes deep learning in the security field inevitably develop rapidly.

[0003] The neural network algorithm is landed on the embedded chip end, and a neural network compiler is used to compile the algorithm model trained on the server into a binary firmware to facilitate computer recognition. The embedded chip end usually transplants a Linux operating system to facilitate the operation of the firmware through file reading and writing.

[0004] Since the forward calculation process of the neural network is realized by hardware, the convolutional neural network algorithm is realized on the chip with hardware intelligent acceleration, which can accelerate the landing of products, but the running performance of each frame is consistent at the same algorithm resolution, and the image detection distance of the device does not change.

[0005] In the prior art, embedded devices with hardware intelligent acceleration chips such as HiSilicon, Linkage and Chip Source do not realize the scheme of supporting variable resolution input on the hardware accelerator algorithm end. For pictures detected during the walking of a person or the movement of a vehicle, it is impossible to realize long-distance detection of people or vehicles, and the network performance cannot be accelerated. SUMMARY

[0006] The present application provides a neural network software and hardware cooperative detection method to solve the problem that the neural network hardware accelerator chip in the prior art can only use the same resolution to send network detection, resulting in low long-distance target detection performance and large memory occupation.

[0007] To achieve the above purpose, the following technical solutions are adopted in the present application:

[0008] The neural network software and hardware cooperative detection method provided by the present application comprises the following steps:

[0009] inputting a first target image into a pre-trained neural network model for forward running, and calculating the output resolution of the current network layer of the neural network;

[0010] replacing the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, wherein the network firmware is obtained by compiling the pre-trained neural network model;

[0011] After all the network layer output resolutions stored in the network firmware have been replaced, the register command words in the network firmware are reconfigured, and the neural network hardware is started to accelerate detection.

[0012] Preferably, the step of inputting the first target image into a pre-trained neural network model for forward processing and calculating the output resolution of the current network layer includes:

[0013] The current image to be detected is acquired, and a foreground extraction algorithm is used to extract the first target image and the position of the first target image from the current image to be detected. If the resolution of the first target image is greater than the preset maximum input resolution of the neural network model, the resolution of the first target image is adjusted to the maximum input resolution.

[0014] Preferably, the method further includes:

[0015] The next frame of the image to be detected is acquired and preprocessed to obtain the second target image and the position of the second target image;

[0016] Calculate the intersection-union ratio (IUR) between the first target image position and the second target image position. If the IUR is less than a preset threshold, then input the second target image into the pre-trained neural network model.

[0017] Preferably, the step of replacing the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, wherein the network firmware is obtained by compiling the pre-trained neural network model, and includes:

[0018] Obtain the output layer network name, number, and output resolution stored in the network firmware, wherein the network firmware is obtained by compiling the pre-trained neural network model using a neural network compiler;

[0019] Obtain the network name and number of the current network layer, and compare them with the network name and number of the output layer. If they match, assign the calculated output resolution to the output layer.

[0020] Otherwise, update the network layer output resolution stored in the network firmware layer by layer.

[0021] Preferably, the step of activating the neural network hardware to accelerate detection includes:

[0022] The calculation results after the current neural network acceleration are obtained, and the calculation results are post-processed to obtain the detection results.

[0023] A neural network hardware and software collaborative detection device, comprising:

[0024] The execution module is used to input the first target image into a pre-trained neural network model for forward processing and to calculate the output resolution of the current network layer of the neural network.

[0025] The replacement module is used to replace the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, wherein the network firmware is compiled from the pre-trained neural network model;

[0026] The configuration module is used to reconfigure the register command words in the network firmware and start the neural network hardware to accelerate detection after all the network layer output resolutions stored in the network firmware have been replaced.

[0027] Preferably, the replacement module includes:

[0028] The acquisition unit is used to acquire the output layer network name, number, and output resolution stored in the network firmware, wherein the network firmware is obtained by compiling the pre-trained neural network model using a neural network compiler;

[0029] The comparison unit is used to obtain the network name and number of the current network layer and compare them with the network name and number of the output layer. If they match, the calculated output resolution is assigned to the output layer.

[0030] An update unit is used to update the network layer output resolution stored in the network firmware layer by layer, unless otherwise specified.

[0031] Preferably, the device further includes:

[0032] The preprocessing module is used to acquire the current image to be detected and extract the first target image and the position of the first target image from the current image to be detected using a foreground extraction algorithm. If the resolution of the first target image is greater than the preset maximum input resolution of the neural network model, the resolution of the first target image is adjusted to the maximum input resolution.

[0033] An electronic device includes a memory and a processor, the memory being used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a neural network hardware-software co-detection method as described in any one of the above descriptions.

[0034] A computer-readable storage medium storing a computer program that, when executed by a computer, implements a neural network hardware-software co-detection method as described in any one of the preceding descriptions.

[0035] This application has the following beneficial effects:

[0036] The resolution of the input neural network in this application is variable, which ensures that there is a detection object at the current resolution. Since the network resolution is smaller, the unscaled image is clearer, which also improves network performance, reduces bandwidth usage, and allows the detection object to be further away. Small targets can be detected normally as long as they meet the network training resolution. This hardware and software co-detection method enables the acceleration hardware to achieve the best performance and usage effect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a neural network hardware and software collaborative detection method implemented in Embodiment 1 of this application;

[0039] Figure 2 This is a network firmware memory layout diagram in Embodiment 1 of this application;

[0040] Figure 3 This is a schematic diagram of a neural network hardware and software collaborative detection device implemented in Embodiment 3 of this application;

[0041] Figure 4 This is a schematic diagram of an electronic device that implements a neural network hardware and software collaborative detection method in Embodiment 4 of this application. Detailed Implementation

[0042] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] The terms “first,” “second,” etc., used in the claims and description of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate. This is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] Example 1

[0045] like Figure 1 As shown, a neural network hardware and software collaborative detection method includes the following steps:

[0046] S110. Input the first target image into the pre-trained neural network model for forward processing and calculate the output resolution of the current network layer of the neural network.

[0047] S120. Replace the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, wherein the network firmware is obtained by compiling the pre-trained neural network model;

[0048] S130. After all the network layer output resolutions stored in the network firmware have been replaced, the register command words in the network firmware are reconfigured, and the neural network hardware is started to accelerate detection.

[0049] Neural network accelerators can only accelerate the forward prediction process of neural networks. The process of back-training the neural network to obtain parameters is completed on the host computer. That is, the host computer sends the weights, images and instructions to the accelerator, and the accelerator returns the results to the host computer after the calculation is completed. Therefore, this embodiment only emphasizes the forward operation of the neural network.

[0050] In existing technologies, the input resolution of neural network models used for object detection is configured to a fixed value. Before inputting the image to be detected into the neural network model, the image must be processed so that its resolution exactly matches the input requirements of the neural network model. However, when the distance is too far, image scaling will make the target smaller. If the size of the target does not meet the model requirements, target detection cannot be performed. The hardware and software co-detection method for neural networks that supports variable resolution provided in this embodiment can solve this problem.

[0051] First, a neural network compiler is used to compile the pre-trained neural network model into network firmware. This is because the trained network model cannot be recognized by the computer, is in floating-point format, and cannot match the input format of the neural network accelerator. The network firmware stores firmware header information, input and output information, information of each intermediate layer, register command words, and quantization weights, such as... Figure 2 As shown, specifically:

[0052] The firmware header information includes the firmware's network name, checksum, and scope, totaling 64 bytes in size.

[0053] Input and output information includes the input layer name, input layer quantization parameters, input and output resolutions of the input layer, and relevant information about the output layer. Since a neural network can have multiple output layers, it includes the layer name of each output layer, the layer quantization parameters, and the output layer resolution.

[0054] The intermediate layer information is sorted by index value from 1 to N, where N represents the last intermediate layer. Each intermediate layer is executed in this order after sorting. It includes the layer name, layer index value, and layer type, and includes network layers with different functions such as convolutional layers, pooling layers, and residual layers.

[0055] Register command words: Each network layer has a set of register command words. Each set of command words contains 21 sub-command words of fixed size. Each sub-command word is 128 bytes long and represents different meanings. They store information such as the output resolution of each network layer, the current layer index value and the next layer index value, the layer type, the layer quantization parameters, the layer kernel size, span, padding, and grouping.

[0056] Quantization weights refer to the process of parsing and quantizing the network weights using a neural network compiler. To save storage space, the 32-bit weight data is quantized into 8-bit data storage using a non-linear quantization method, resulting in almost no loss of precision and a result close to that of floating-point operations.

[0057] The input and output feature data quantization method is as follows:

[0058] Qd = BW - ceil(log2(x) max )+1)

[0059] Where, x max For x′ max abs(x′) min The maximum value in ), x′ max The maximum value of the input floating-point number is x′. min The minimum value of the input floating-point number is given by 'abs', which is the absolute value and is a built-in function of the system library. 'BW' is the quantization bit width, which is currently only supported by 8-bit hardware acceleration. 'log2' is a built-in function of the system library, and 'ceil' is the rounding function, which is also a built-in function of the system library.

[0060] The method for converting floating-point to fixed-point is as follows:

[0061] X = round(x float *2 Qd )

[0062] Where `round` is a built-in function in the system library for rounding, and `x`... float X is a floating-point number, and Qd is a quantization parameter. When the value of X exceeds the upper limit of the bit width, the upper limit value is taken; when it is less than the lower limit of the bit width, the lower limit value is taken.

[0063] The method for converting fixed-point to floating-point is as follows:

[0064] x float = (X*1.0) / 2 Qd

[0065] Where, x float Qd is a floating-point number, Qd is the quantization parameter, and X represents fixed-point 8-bit data.

[0066] Secondly, based on the performance of the network accelerator in the chip and the actual application scenario of the algorithm, determine the maximum input resolution of the pre-trained neural network model. For example, if the chip acquires images at 25 frames per second in the current scenario, then 1 frame is 40ms. Test the performance of the current network on the chip to determine the maximum input resolution of the network.

[0067] Then, the current image to be detected is acquired, preprocessed, and then input into a pre-trained neural network model for forward propagation.

[0068] Specifically, the current image to be detected is acquired, and a foreground extraction algorithm is used to extract the first target image and the position of the first target image from the current image to be detected. If the resolution of the first target image is greater than the preset maximum input resolution of the neural network model, the resolution of the first target image is adjusted to the maximum input resolution.

[0069] The system acquires the current image to be detected, and uses methods such as histogram and foreground extraction to crop the target image from it. The system then obtains the position of the target image within the image to be detected. This position is approximate rather than precise. There is only one approximate position for each frame, which is the approximate region of the target detection result. There is also only one precise position, which is the location of the detected target. If the resolution of the target image is less than or equal to the maximum input resolution of the network, it is directly fed into the network. If it is greater than the maximum input resolution, its resolution is reduced to the size of the maximum input resolution before being fed into the network for detection.

[0070] After the current image to be detected is sent to the network for detection, the next frame of the image to be detected is obtained and preprocessed to obtain the second target image and the position of the second target image;

[0071] Calculate the intersection-union ratio (IUR) between the first target image position and the second target image position. If the IUR is less than a preset threshold, then input the second target image into the pre-trained neural network model.

[0072] After preprocessing the current image to be detected and feeding it into a pre-trained neural network model for detection, the next frame image is acquired as the current image to be detected. The same preprocessing is performed on the current frame image as on the previous frame image to obtain the target image and its position within the image. Then, the IOU (Intersection over Union) is calculated between the target image positions obtained from the current frame image and those obtained from the previous frame image. If the result is less than 0.9 and the resolution of the current target image is less than or equal to the maximum input resolution, the current target image is directly fed into the neural network. If the overlap ratio is less than 0.9 but the resolution of the current target image is greater than the maximum input resolution, the target image is not directly fed into the neural network. When adjusting the resolution, the current target image resolution is first reduced to the maximum input resolution before being sent to the neural network; otherwise, the original result is used to reduce network computational performance. IOU (Intersection over Union) is a concept used in object detection, which is the overlap ratio between the generated candidate box and the original labeled box, i.e., the ratio of their intersection to their union. When the ratio is 1, it means that the candidate box and the original labeled box completely overlap. In this embodiment, the original labeled box is the position of the target image in the previous frame, and the candidate box is the position of the target image in the next frame. Simply put, when the target is stationary or the target movement range in each image is small, it is not sent to the network for detection, thereby saving network performance.

[0073] Next, based on the image input to the pre-trained neural network model and the original network model parameter information stored in the network firmware, the output resolution of each layer of the current neural network is calculated. Except for the input layer, the input resolution of other network layers is the output resolution of the previous layer, so there is no need to calculate their input resolution. For convolutional layers, the padding size, kernel size, and kernel span are obtained to calculate their output resolution. For pooling layers, different formulas are used to calculate their output resolution for different training platforms such as Caffe and PyTorch. For residual layers, their input resolution and output resolution are consistent. The formula for calculating the output resolution of convolutional layers is as follows:

[0074] Ow=(Iw–Kw+2*Pw) / Sw+1

[0075] Oh = (Ih – Kh + 2 * Ph) / Sh + 1

[0076] Where Ow and Oh represent the width and height of the output resolution of the layer, respectively; Iw and Ih represent the width and height of the input resolution of the layer, respectively; Pw and Ph represent the horizontal and vertical padding of the convolution kernel, respectively; Sw and Sh represent the horizontal and vertical span of the convolution kernel, respectively; and Kw and Kh represent the width and height of the convolution kernel, respectively.

[0077] The formula for calculating the output resolution of the pooling layer in the Caffe platform is as follows:

[0078] Ow=ceil((Iw–Kw+2*Pw) / Sw)+1

[0079] Oh=ceil((Ih–Kh+2*Ph) / Sh)+1

[0080] The formula for calculating the output resolution of the pooling layer in the PyTorch platform is as follows:

[0081] Ow=floor((Iw–Kw+2*Pw) / Sw)+1

[0082] Oh=floor((Ih–Kh+2*Ph) / Sh)+1

[0083] Where Ow and Oh represent the width and height of the output resolution of the layer, respectively; Iw and Ih represent the width and height of the input resolution of the layer, respectively; Pw and Ph represent the horizontal and vertical padding of the convolution kernel, respectively; Sw and Sh represent the horizontal and vertical span of the convolution kernel, respectively; Kw and Kh represent the width and height of the convolution kernel, respectively; floor represents floor; and ceil represents ceil.

[0084] The formula for calculating the output resolution of the residual layer is as follows:

[0085] Ow = Iw

[0086] Oh = Ih

[0087] Where Ow and Oh represent the width and height of the output resolution of this layer, respectively, and Iw and Ih represent the width and height of the input resolution of this layer, respectively.

[0088] Then, the output layer network name, number, and output resolution stored in the network firmware are obtained. The network firmware is obtained by compiling the pre-trained neural network model using a neural network compiler.

[0089] Obtain the network name and number of the current network layer, and compare them with the network name and number of the output layer. If they match, assign the calculated output resolution to the output layer.

[0090] Otherwise, update the network layer output resolution stored in the network firmware layer by layer.

[0091] The layer name and number of the current layer in the neural network are compared with the layer name and number of the output layer. If they match, the current layer is the output layer, and the output layer resolution stored in the network firmware is replaced with the calculated output resolution. If they do not match, the current layer is another layer, and the output resolution of the network layer with the same layer name and number stored in the network firmware is replaced with the calculated output resolution. If the pre-trained neural network model has multiple output layers, the output layer here refers to the last output layer, until the output resolutions of each layer stored in the network firmware have been replaced.

[0092] Based on the output resolution of each layer of the replaced network firmware, the register command words stored in the network firmware are updated, and the startup information is configured to start the neural network hardware for forward acceleration. An interrupt is generated after the forward operation is completed.

[0093] Finally, the calculation results after the current neural network acceleration are obtained, and the calculation results are post-processed to obtain the detection results.

[0094] After the hardware calls the interrupt function, the host obtains the forward execution result of the current neural network, that is, the calculation result of the output layer of the last layer of the current neural network. Based on the calculation result, image post-processing is performed. The image post-processing uses the Non-Maximum Suppression (NMS) algorithm. The essence of this algorithm is to search for local maxima and suppress non-maximum elements. In object detection, it is to filter out redundant candidate boxes and retain the candidate box with the best detection position. This candidate box is the detection result. Although there is only one approximate position of the target image, there are multiple target candidate boxes generated during the network operation. It is necessary to select the candidate box that best meets the requirements from multiple candidate boxes, so as to realize the hardware and software collaborative detection of the neural network.

[0095] In this embodiment, a maximum input resolution is set for the neural network model to ensure that the memory does not exceed the limit for different combinations of resolutions. At the same time, the resolution of the input neural network is variable, which ensures that there is a detection object at the current resolution. Since the network resolution is smaller, the unscaled image is clearer, which improves network performance, reduces bandwidth usage, and allows the detection object to be further away. Small targets can be detected normally as long as they meet the network training resolution. This hardware and software co-detection method enables the acceleration hardware to achieve the best performance and usage effect.

[0096] Example 2

[0097] This embodiment uses human detection as an example to illustrate the neural network hardware and software collaborative detection method:

[0098] Based on the specific application scenario of the neural network, a maximum input resolution is set for the pre-trained neural network model to ensure that the input resolution of each image in the later stages is controlled within this maximum input resolution. This ensures that the output feature memory of each layer does not exceed the limit during the forward operation of the network. Because the chip's internal SRAM has limitations, the hardware operation requires importing data (such as the output resolution) from the external DDR memory into the internal memory for computation. At this time, the data at such a high output resolution does not meet the computational requirements and needs to be imported in blocks. For example, if the SRAM size is 16*16, then conv_w=(16-kw) / ks tep+1, where kw is the kernel width, kstep is the kernel span, conv_w is the output convolution width, block_w_num = ceil(out_wid / conv_w), where block_w_num represents the number of horizontal blocks and out_wid represents the output width resolution. Since a high resolution would consume too much memory and storage space, this embodiment uses a width of 640 and a height of 320 as the maximum value, that is, the maximum input resolution is 640*320. The product of the width and height of the variable resolution is less than or equal to 640*320, which can detect human figures within 20 meters.

[0099] Using 50 quantized images from different scenarios, relevant quantization parameters are inferred using an existing neural network compiler, and network firmware is generated. The firmware memory layout is the same as in Example 1.

[0100] Next, a sensor (image acquisition device) is used to acquire a frame of image with a resolution of 1080P (1920 wide, 1080 high). Histogram and foreground feature extraction methods are used to extract the image and approximate position of the current human figure at a resolution of 1080P. If the resolution of the target image, i.e. the human figure image, is less than or equal to 640*320, the human figure image is directly input into the neural network for processing. Otherwise, the resolution of the human figure image is reduced to 640*320 before being fed into the neural network.

[0101] Then, based on the image input to the pre-trained neural network model and the original network model parameter information stored in the network firmware, the output resolution of each layer of the current neural network is calculated. The layer name and number of the current layer are compared with the layer name and number of the output layer stored in the network firmware. If they match, the current layer is the output layer, and the output layer resolution stored in the network firmware is replaced with the calculated output resolution. If they do not match, the current layer is another layer, and the output resolution of the network layer with the same layer name and number stored in the network firmware is replaced with the calculated output resolution. The pre-trained neural network model has multiple output layers, and the output layer here refers to the last output layer. This continues until all the output resolutions of each layer stored in the network firmware have been replaced and the register command words stored in the network firmware have been updated. The neural network hardware is then started to accelerate forward. After the forward acceleration is completed, the hardware generates an interrupt, and the host obtains the forward running results. Based on the forward running results, image post-processing is performed to obtain the output box position and confidence score of the target. The confidence score represents the probability of a human figure. When it is greater than 0.8, it is considered a human figure. The CPU is used to draw lines to display the target, thereby realizing the hardware and software collaborative detection of the neural network.

[0102] The process involves preprocessing the current image to be detected and feeding it into the neural network. Then, the next frame is captured, and histogram and foreground extraction methods are used to extract the approximate position of the current human figure at 1080P resolution. This position is then compared with the human figure position in the previous frame, using the Intersection over Union (IOU) method. For example, if the human figure position in the previous frame is represented by A and the current human figure position by B, the IOU overlap ratio is the intersection of A and B divided by the union of A and B. If the overlap ratio is less than 0.9 and the resolution of the current human figure image is less than or equal to 640*320, the current human figure image is directly fed into the neural network. If the overlap ratio is less than 0.9 but the resolution of the current human figure image is greater than 640*320, the resolution of the current human figure image is reduced to 640*320 before being fed into the neural network. Otherwise, the original result is used, meaning the current human figure image is not fed into the network to reduce network computational performance. At the same time, a smaller resolution can accelerate network operation, and the scaled and cropped image is clear. It is only necessary to ensure that the aspect ratio of the image is within the network training constraints.

[0103] Example 3

[0104] like Figure 3 As shown, a neural network hardware and software collaborative detection device includes:

[0105] The running module 10 is used to input the first target image into a pre-trained neural network model for forward processing and to calculate the output resolution of the current network layer of the neural network.

[0106] Replacement module 20 is used to replace the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, wherein the network firmware is obtained by compiling the pre-trained neural network model;

[0107] The configuration module 30 is used to reconfigure the register command words in the network firmware and start the neural network hardware to accelerate detection after all the network layer output resolutions stored in the network firmware have been replaced.

[0108] One embodiment of the above-mentioned device may be as follows: the running module 10 inputs the first target image into a pre-trained neural network model for forward processing and calculates the output resolution of the current network layer of the neural network; the replacement module 20 replaces the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, wherein the network firmware is obtained by compiling the pre-trained neural network model; after all the network layer output resolutions stored in the network firmware have been replaced, the configuration module 30 reconfigures the register command words in the network firmware and starts the neural network hardware to accelerate detection.

[0109] Example 4

[0110] like Figure 4 As shown, an electronic device includes a memory 401 and a processor 402. The memory 401 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 402 to implement the above-described neural network hardware and software collaborative detection method.

[0111] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the electronic device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] A computer-readable storage medium storing a computer program that, when executed by a computer, enables a neural network hardware-software co-detection method as described above.

[0113] For example, a computer program can be divided into one or more modules / units. One or more modules / units are stored in memory 401 and executed by processor 402. Data I / O interface transmission is completed by input interface 405 and output interface 406 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions. The instruction segments are used to describe the execution process of the computer program in the computer device.

[0114] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a memory 901 and a processor 902. Those skilled in the art will understand that this embodiment is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include an input device 407, a network access device, a bus, etc.

[0115] The processor 402 can be a Central Processing Unit (CPU), or other general-purpose processors 402, digital signal processors 402 (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 402 can be a microprocessor 402, or any conventional processor 402, etc.

[0116] The memory 401 can be an internal storage unit of a computer device, such as a hard disk or memory. The memory 401 can also be an external storage device of a computer device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 401 can include both internal and external storage units. The memory 401 is used to store computer programs and other programs and data required by the computer device. The memory 401 can also be used for temporary storage in the output device 408. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM) 403, random access memory (RAM) 404, discs, or optical discs.

[0117] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A neural network hardware and software collaborative detection method, characterized in that, The method comprises the following steps: inputting a first target image into a pre-trained neural network model for forward running, and calculating the output resolution of the current network layer of the neural network; replacing the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, the network firmware being obtained by compiling the pre-trained neural network model; after all the network layer output resolutions stored in the network firmware are replaced, reconfiguring the register command word in the network firmware, and starting the neural network hardware to accelerate detection; the step of replacing the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, the network firmware being obtained by compiling the pre-trained neural network model, comprises: obtaining the network name, number and output resolution of the output layer stored in the network firmware, the network firmware being obtained by compiling the pre-trained neural network model by using a neural network compiler; obtaining the network name and number of the current network layer, and comparing them with the network name and number of the output layer, if they are consistent, assigning the calculated output resolution to the output layer; otherwise, updating the network layer output resolutions stored in the network firmware layer by layer.

2. The method of claim 1, wherein, The method further comprises the following steps before the step of inputting the first target image into the pre-trained neural network model for forward running, and calculating the output resolution of the current network layer of the neural network: collecting a current image to be detected, and extracting a first target image and the position of the first target image from the current image to be detected by using a foreground extraction algorithm, if the resolution of the first target image is greater than a preset maximum input resolution of the neural network model, adjusting the resolution of the first target image to the maximum input resolution.

3. The method of claim 1 or 2, wherein, The method further comprises the following steps: obtaining a next frame of image to be detected, and pre-processing the next frame of image to be detected to obtain a second target image and the position of the second target image; calculating the intersection over union of the position of the first target image and the position of the second target image, if the intersection over union is less than a preset threshold, inputting the second target image into the pre-trained neural network model.

4. The method of claim 1, wherein, The method further comprises the following steps after the step of starting the neural network hardware to accelerate detection: obtaining the calculation result of the current neural network after acceleration, and performing image post-processing on the calculation result to obtain a detection result.

5. A neural network hardware-software co-detection apparatus, comprising: The method comprises: a running module, configured to input a first target image into a pre-trained neural network model for forward running, and calculate the output resolution of the current network layer of the neural network; a replacing module, configured to replace the output resolution of the corresponding network layer stored in the network firmware with the calculated output resolution, the network firmware being obtained by compiling the pre-trained neural network model; a configuration module, configured to reconfigure the register command word in the network firmware after all the network layer output resolutions stored in the network firmware are replaced, and start the neural network hardware to accelerate detection; the replacing module comprises: An acquisition unit is configured to acquire an output layer network name, a number, and an output resolution stored in a network firmware, wherein the network firmware is obtained by compiling the pre-trained neural network model using a neural network compiler; A comparison unit is configured to acquire a network name and a number of the current network layer, and compare the network name and the number of the current network layer with those of the output layer. If the network name and the number of the current network layer are consistent with those of the output layer, the calculated output resolution is assigned to the output layer. An update unit is configured to update the output resolution of the network layer stored in the network firmware layer by layer if the network name and the number of the current network layer are not consistent with those of the output layer.

6. The neural network hardware / software co-detection apparatus of claim 5, wherein, The device further comprises: A preprocessing module is configured to acquire a current image to be detected, and extract a first target image and a position of the first target image from the current image to be detected using a foreground extraction algorithm. If a resolution of the first target image is greater than a preset maximum input resolution of a neural network model, the resolution of the first target image is adjusted to the maximum input resolution.

7. An electronic device, comprising: The computer program enables a computer to implement the neural network software and hardware collaborative detection method according to any one of claims 1-4 when executed.

8. A computer readable storage medium storing a computer program, characterized in that, The computer program enables a computer to implement the neural network software and hardware collaborative detection method according to any one of claims 1-4 when executed.

Images