Object detection method and device based on lookup table

Abstract

The embodiment of the invention provides an object detection scheme based on a lookup table, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the unified quantification of weight parameters of convolution kernels of all convolution layers in a pre-trained object detection model and an activation value of a generated feature map; creating a two-dimensional multiplication lookup table; inputting a to-be-detected image into the object detection model, and when a calculation process enters a target convolutional layer, obtaining a feature map corresponding to the to-be-detected image after unified quantization and a convolution kernel in the target convolutional layer after quantization; aiming at each element in the feature map, regarding the element as a first multiplier, and regarding the corresponding element of the element in the convolution kernel as a second multiplier; inputting the first multiplier and the second multiplier into a two-dimensional multiplication lookup table to obtain binary intermediate products, performing shift zero filling on the binary intermediate products, and then performing summation to obtain a target product; and generating a detection tensor based on each obtained target product. Through the object detection method, the detection efficiency of the object in the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for object detection based on a lookup table. Background Art

[0002] Object detection is a core technology in computer vision, widely used in scenarios such as autonomous driving, intelligent surveillance, and robotic navigation. YOLO (You Only Look Once), a high-performance object detection and image segmentation model, is currently one of the mainstream real-time object detection technologies, widely used for its exceptional speed and good accuracy. In YOLO, an input image x is first predefined as a logical partition consisting of s×s grids. The image is then processed by a CNN (Convolutional Neural Network), ultimately outputting a feature tensor of dimensions s×s×(B×5+C). In this tensor, s represents the number of grid partitions in the horizontal and vertical directions. B represents the number of bounding boxes predicted for each grid. Each bounding box is typically described by five values: the center point x and y coordinates, width w, height h, and confidence score. C represents the total number of object types predefined in the dataset.

[0003] This output tensor contains the location, size, and category information of all potential objects in the image. However, the convolutional neural network that implements this process is computationally intensive, especially in its core convolution operation. This limits the responsiveness of YOLO technology in systems with extremely high real-time requirements. The core operation of a convolutional neural network is the convolution operation. This operation requires the calculation of two matrices: one matrix A represents the input image or the feature map output by the previous layer, and the other matrix K represents the convolution kernel or filter. The convolution process can be described as sliding the convolution kernel K across matrix A, performing element-wise multiplications at each position and summing them to generate a new output matrix. During the forward propagation of the entire YOLO network, this multiplication operation is performed billions or even trillions of times. Therefore, improving the efficiency of the multiplication operation between the convolution kernel K and each element in matrix A is key to accelerating the entire YOLO object detection process. Summary of the Invention

[0004] In order to solve one of the above technical defects, an object detection method and device based on a lookup table are provided in an embodiment of the present application.

[0005] An embodiment of the present invention provides an object detection method based on a lookup table, wherein the method includes:

[0006] Performing unified quantization on weight parameters of convolution kernels of all convolutional layers in a pre-trained object detection model and activation values ​​of feature maps generated during inference, wherein the unified quantization includes: representing 32-bit floating-point numbers as fixed-point integers with a first number being an exponent of 2;

[0007] Creating a two-dimensional multiplication lookup table; wherein the two-dimensional multiplication lookup table includes products of a plurality of first-number-bit fixed-point binary number pairs;

[0008] Input the image to be detected into the object detection model. When the calculation process enters the target convolution layer, obtain the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer;

[0009] For each element in the feature map, consider the element as a first multiplier, and consider the element corresponding to the element in the convolution kernel as a second multiplier;

[0010] Inputting the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and performing zero-padding on each of the binary intermediate products and then summing them to obtain a target product;

[0011] Generate a detection tensor based on the obtained target products;

[0012] The detection tensor is decoded to extract detection information of each object in the image.

[0013] Optionally, the step of uniformly quantizing the weight parameters of the convolution kernels of all convolutional layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process includes:

[0014] Using a quantization tool to convert the weight parameters of the convolution kernels of all convolutional layers in the object detection model from 32-bit floating point numbers to 8-bit fixed point integers;

[0015] The input feature map activation values ​​generated during the object detection model inference process are converted into 8-bit fixed-point integers.

[0016] Optionally, the step of inputting the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and performing shifting and zero-padding on each of the binary intermediate products and then summing them to obtain a target product includes:

[0017] Dividing the first multiplier into groups starting from the right side to obtain first groups, wherein each first group contains the first number of binary numbers;

[0018] Dividing the second multiplier into groups starting from the right side to obtain second groups, wherein each second group contains the first number of binary numbers;

[0019] Extracting one group from each of the first group and the second group according to a preset rule and inputting them into a lookup table to obtain a binary intermediate product, wherein a second number of binary numbers included in the binary intermediate product is twice the first number;

[0020] The binary intermediate products are shifted and zero-filled, and then summed to obtain a target product.

[0021] Optionally, the step of dividing the first multipliers into groups starting from the right side to obtain a first group includes:

[0022] determining whether the number of binary digits included in the first multiplier is an integer multiple of the first number;

[0023] If not, padding the end of the first multiplier with zeros until the number of binary digits included is an integer multiple of the first number, and then performing group division starting from the right side of the first multiplier after the zero padding to obtain a first group;

[0024] If so, group division is performed directly starting from the right side of the first multiplier to obtain a first group.

[0025] Optionally, the step of extracting one group from each of the first group and the second group according to a preset rule and inputting the groups into a lookup table to obtain a binary intermediate product includes:

[0026] When both the first group and the second group contain multiple subgroups, each subgroup in the first group and each subgroup in the second group are respectively combined into an intermediate multiplier pair and input into a lookup table to obtain a binary intermediate product.

[0027] Optionally, the step of performing shifting and zero-padding on each of the binary intermediate products and then summing the products to obtain a target product includes:

[0028] For each of the binary intermediate products, determining the number of left shifts and the number of zero paddings for the binary intermediate product according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product, wherein the number of left shifts and the number of zero paddings corresponding to the same binary intermediate product are the same;

[0029] The binary intermediate products after shifting and zero-filling are summed to obtain a target product.

[0030] Optionally, when the first number is 4, the first group includes a first upper four-bit subgroup and a first lower four-bit subgroup, and the second group includes a second upper four-bit subgroup and a second lower four-bit subgroup, the step of respectively forming intermediate multiplier pairs from each subgroup in the first group and each subgroup in the second group and inputting them into a lookup table to obtain a binary intermediate product includes:

[0031] Inputting the first lower four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a first intermediate product;

[0032] Inputting the first low-order four-bit subgroup and the second high-order four-bit subgroup into a lookup table to obtain a second intermediate product;

[0033] Inputting the first upper four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a third intermediate product;

[0034] The first upper four-bit subgroup and the second upper four-bit subgroup are input into a lookup table to obtain a fourth intermediate product.

[0035] Optionally, for each binary intermediate product, the step of determining the number of left shifts and the number of zero paddings for the binary intermediate product according to the high and low bit levels of two subgroups corresponding to the binary intermediate product includes:

[0036] For the first intermediate product, determining that both subgroups corresponding to the first intermediate product are low bits, and determining that the first intermediate product is not to be left-shifted or padded with zeros;

[0037] For the second intermediate product and the third intermediate product, determining that the two subgroups corresponding to the second intermediate product and the third intermediate product both have one high bit and one low bit, determining that the second intermediate product and the third intermediate product are both shifted left by four bits, and padding the four shifted bits with zeros;

[0038] For the fourth intermediate product, it is determined that both subgroups corresponding to the fourth intermediate product are high bits, and after the fourth intermediate product is left-shifted by eight bits, the eight bits shifted left are padded with zeros.

[0039] An embodiment of the present invention provides an object detection device based on a lookup table, wherein the device includes:

[0040] A unified quantization module is configured to uniformly quantize the weight parameters of the convolution kernels of all convolutional layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process, wherein the unified quantization processing includes: representing the 32-bit floating-point number as a first number of fixed-point integers; the first number is an exponent of 2;

[0041] A lookup table creation module, configured to create a two-dimensional multiplication lookup table; wherein the two-dimensional multiplication lookup table includes products of a plurality of first-number-bit fixed-point binary number pairs;

[0042] An acquisition module is used to input the image to be detected into the object detection model, and when the calculation process enters the target convolution layer, obtain the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer;

[0043] a multiplier determination module, configured to, for each element in the feature map, regard the element as a first multiplier and regard the element corresponding to the element in the convolution kernel as a second multiplier;

[0044] a calculation module, configured to input the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and perform zero-padding on each of the binary intermediate products and then sum them to obtain a target product;

[0045] A detection tensor generation module, configured to generate a detection tensor based on the obtained target products;

[0046] The decoding module is used to decode the detection tensor and extract the detection information of each object in the image.

[0047] Optionally, the unified quantization module is specifically configured to:

[0048] The weight parameters of the convolution kernels of all convolutional layers in the object detection model are converted from 32-bit floating-point numbers to 8-bit fixed-point integers using a quantization tool; and the activation values ​​of the input feature maps generated during the inference process of the object detection model are converted into 8-bit fixed-point integers.

[0049] Optionally, the calculation module includes:

[0050] a first grouping submodule, configured to group the first multiplier starting from the right side to obtain first groups, wherein each first group contains the first number of binary numbers;

[0051] A second grouping submodule, configured to group the second multiplier starting from the right side to obtain second groups, wherein each second group contains the first number of binary numbers;

[0052] a table lookup submodule, configured to extract one group from each of the first group and the second group according to a preset rule and input the group into a lookup table to obtain a binary intermediate product, wherein a second number of binary numbers included in the binary intermediate product is twice the first number;

[0053] The shift processing submodule is used to perform shifting and zero-filling on each of the binary intermediate products and then sum them to obtain a target product.

[0054] Optionally, the first grouping submodule includes:

[0055] a judging unit, configured to judge whether the number of binary numbers included in the first multiplier is an integer multiple of the first number;

[0056] The execution unit is configured to, if the answer is no, pad the end of the first multiplier with zeros until the number of binary digits contained therein is an integer multiple of the first number, and then perform group division starting from the right side of the first multiplier after the zero padding to obtain a first group; and if the answer is yes, directly perform group division starting from the right side of the first multiplier to obtain a first group.

[0057] Optionally, the table lookup submodule is specifically used to:

[0058] When both the first group and the second group contain multiple subgroups, each subgroup in the first group and each subgroup in the second group are respectively combined into an intermediate multiplier pair and input into a lookup table to obtain a binary intermediate product.

[0059] Optionally, the shift processing submodule includes:

[0060] a zero-padding unit configured to determine, for each binary intermediate product, a number of left shift bits and a number of zero-padding bits of the binary intermediate product according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product, wherein the number of left shift bits and the number of zero-padding bits corresponding to the same binary intermediate product are the same;

[0061] The summing unit is used to sum the binary intermediate products after shifting and filling with zeros to obtain a target product.

[0062] Optionally, when the first number is 4, the first group includes a first upper four-bit subgroup and a first lower four-bit subgroup, and the second group includes a second upper four-bit subgroup and a second lower four-bit subgroup, the table lookup submodule is specifically configured to:

[0063] Inputting the first lower four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a first intermediate product;

[0064] Inputting the first low-order four-bit subgroup and the second high-order four-bit subgroup into a lookup table to obtain a second intermediate product;

[0065] Inputting the first upper four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a third intermediate product;

[0066] The first upper four-bit subgroup and the second upper four-bit subgroup are input into a lookup table to obtain a fourth intermediate product.

[0067] Optionally, the zero-padding unit is specifically configured to:

[0068] For the first intermediate product, determining that both subgroups corresponding to the first intermediate product are low bits, and determining that the first intermediate product is not to be left-shifted or padded with zeros;

[0069] For the second intermediate product and the third intermediate product, determining that the two subgroups corresponding to the second intermediate product and the third intermediate product both have one high bit and one low bit, determining that the second intermediate product and the third intermediate product are both shifted left by four bits, and padding the four shifted bits with zeros;

[0070] For the fourth intermediate product, it is determined that both subgroups corresponding to the fourth intermediate product are high bits, and after the fourth intermediate product is left-shifted by eight bits, the eight bits shifted left are padded with zeros.

[0071] In another aspect of the present invention, an electronic device is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the above-mentioned lookup table-based object detection methods when executing the programs stored in the memory.

[0072] The object detection scheme based on lookup table disclosed in the present application uniformly quantizes the weight parameters of the convolution kernels of all convolution layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process; creates a two-dimensional multiplication lookup table; inputs the image to be detected into the object detection model, and when the calculation process enters the target convolution layer, obtains the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer; for each element in the feature map, regards the element as the first multiplier, and regards the element corresponding to the element in the convolution kernel as the second multiplier; inputs the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and shifts and fills each binary intermediate product with zeros and then sums them to obtain a target product; generates a detection tensor based on each target product obtained; decodes the detection tensor to extract the detection information of each object in the image. With the lookup table-based object detection solution disclosed in this application, the multiplication process of elements in the feature map with elements in the convolution kernel only involves a lookup table, left shift and zero padding, and binary addition. Compared to using traditional multiplication rules to calculate the product of two numbers, the computer's execution of this series of binary-related operations can increase the computing speed, and thus the multiplication speed. Because the multiplication speed is increased, the speed of generating detection tensors is also indirectly increased, ultimately speeding up object detection in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0074] Figure 1is a flowchart showing the steps of a lookup table-based object detection method according to an embodiment of the present application;

[0075] Figure 2 is a flowchart showing a method of implementing multiplication based on a lookup table and binary addition according to an embodiment of the present application;

[0076] Figure 3 is a schematic diagram showing a lookup table for calculating an intermediate product according to an embodiment of the present application;

[0077] Figure 4 is a structural block diagram of an object detection device based on a lookup table according to an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the technical solutions and advantages of the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, and are not an exhaustive list of all the embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other unless they conflict.

[0079] Multiplication has many application scenarios in computing, especially in today's deep learning, where there are a large number of matrix multiplications. The present invention proposes a method for implementing the multiplication of two integers using only a lookup table, shift padding, and binary addition, without the need for traditional complex multiplication calculations. On the one hand, it can improve the implementation speed of multiplication, and on the other hand, it is suitable for parallel computing and can make full use of the computing resources of computers such as multi-core CPUs and GPUs. The multiplication calculation method provided by the present invention can be applied not only to the scenario of finding the product of two integers, but also to real number multiplication and division. The reason why it can be applied to real number multiplication is that the difference between real number multiplication and integer multiplication is the shift of the decimal point. After obtaining the product of the two integers, we can obtain the result of real number multiplication by shifting the decimal point. The reason why it can be applied to division is that the implementation process of division on a computer involves the multiplication of two numbers, so the present invention can be applied to the intermediate process of division to improve the operation speed of division.

[0080] The object detection solution based on the lookup table provided in the embodiment of the present application is described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0081] As attached Figure 1 As shown, the object detection method based on the lookup table of the embodiment of the present application includes the following steps:

[0082] Step 101: uniformly quantize the weight parameters of the convolution kernels of all convolutional layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process.

[0083] In actual implementation, a pre-trained object detection model can be loaded onto the computer system's local hard drive or a remote server. The object detection model can be a YOLO model, which is widely used for its exceptional speed and accuracy in real-time object detection. The object model can be stored in a format supported by deep learning frameworks such as PyTorch, TensorFlow, and ONNX. The object model consists of multiple convolutional layers, each of which includes an input feature map A, a convolution kernel K, and an output feature map C. The model is then loaded into main memory for subsequent inference.

[0084] The unified quantization processing includes: representing the 32-bit floating point number as a first number of fixed-point integers; the first number is an exponent of 2.

[0085] In an optional embodiment, the weight parameters of the convolution kernels of all convolution layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process are uniformly quantized as follows: the weight parameters of the convolution kernels of all convolution layers in the object detection model are converted from 32-bit floating point numbers to 8-bit fixed-point integers using a quantization tool; the activation values ​​of the input feature maps generated during the inference process of the object detection model are converted to 8-bit fixed-point integers. The 8-bit fixed-point integer can be represented as INT8. The quantized value range is [0, 255] or [-128, 127], depending on whether a signed integer table is used; the quantization results can be recorded as K q (convolution kernel) and A q (Input feature map. The quantized object model parameters can be saved in memory for use in the inference phase.

[0086] It should be noted that this optional embodiment only exemplifies that a 32-bit floating-point number can be represented as an 8-bit fixed-point integer. In the specific implementation process, it is not limited to this. A 32-bit floating-point number can be represented as any fixed-point integer with an exponent of 2, for example: 4 bits, 12 bits, etc.

[0087] Step 102: Create a two-dimensional multiplication lookup table.

[0088] The two-dimensional multiplication lookup table includes products of a plurality of first-number-bit fixed-point binary number pairs.

[0089] Taking the representation of 32-bit floating-point numbers as 8-bit fixed-point integers as an example, since the data has been quantized to INT8, its value range is [0, 255] (or [-128, 127]), with a total of 256 possible values. Therefore, a 256×256 lookup table is constructed. This table precomputes and stores the products of all INT8 integer pairs. The value of the element LUT(val1, val2) in the lookup table (which can be represented as a LUT) is the multiplication result of the integers val1 and val2. This lookup table is stored in a high-speed cache, low-latency memory such as SRAM or CPU cache.

[0090] The cache can be level 1 (L1), level 2 (L2), or on-chip SRAM, ensuring low latency for table lookups. The lookup table remains resident during inference and is not released when the inference task completes.

[0091] Step 103: Input the image to be detected into the object detection model. When the calculation process enters the target convolution layer, obtain the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer.

[0092] Steps 103 to 107 are the online detection stage of the image. When the system enters the real-time detection of objects in the image, the image x is input and the object detection model starts forward propagation.

[0093] Step 104: For each element in the feature map, the element in the feature map is regarded as a first multiplier, and the element in the convolution kernel corresponding to the element in the feature map is regarded as a second multiplier.

[0094] When the calculation process enters any convolution layer, the quantized input feature map A is obtained q And the quantized convolution kernel K q For each position (i, j) of the output feature map C, the sliding convolution kernel K q In A q For each pair of matching A q (i+u, j+v) and K q (u, v) can be regarded as both the first multiplier and the second multiplier.

[0095] Step 105: Input the first multiplier and the second multiplier into a two-dimensional multiplication lookup table to obtain binary intermediate products, and perform zero-padding on each binary intermediate product and then sum them to obtain a target product.

[0096] For the input feature map A q And the quantized convolution kernel K qWhen performing convolution calculations, for each element C(i, j) of the output matrix C, the core multiplication term A(i+u,j+v)·K(u,v) in the calculation process is replaced by the following efficient table lookup operation:

[0097] Get the quantized value A q (i+u, j+v) and K q (u, v); Use these two INT8 integer values ​​directly as indexes to access the lookup table LUT, that is, execute Product = LUT (A q (i+u,j+v),K q (u, v)); the target product of the two multipliers can be quickly calculated through table lookup, completely avoiding the need to call the hardware multiplier. The value of C(i, j) is obtained by summing all the target product terms.

[0098] The specific process of calculating the target product can refer to the method for implementing multiplication based on a lookup table and binary addition provided in the following embodiment, which will not be repeated in detail in the embodiments of this application. The convolution operation provided in the embodiments of this application completely avoids the use of a hardware multiplier and relies only on memory access to complete the multiplication operation. The convolution operation can be performed on a general-purpose CPU, an embedded processor, or an AI coprocessor such as a GPU.

[0099] During the forward propagation and convolution computations of the object detection model, frequently used input feature map tiles are cached in a high-speed cache. In practical implementations, the principle of locality can be exploited to reduce the number of external memory accesses. Alternatively, a block-wise feature map read strategy combined with a lookup table access pattern can be employed to improve overall throughput. In a more preferred embodiment, prefetch instructions can be used to preload upcoming feature map data, further reducing memory access latency.

[0100] Step 106: Generate a detection tensor based on the obtained target products.

[0101] Repeat steps 104 to 105 to obtain the input feature map A q The C value corresponding to each element in , based on which a detection tensor can be generated.

[0102] The detection tensor is a feature tensor of dimension s×s×(B×5+C). The detection tensor contains information such as bounding box coordinates, confidence, and category probability. The CPU can be used to decode the detection tensor and extract the detection information of each object in the image.

[0103] Step 107: Decode the detection tensor to extract the detection information of each object in the image.

[0104] Steps 104 and 105 can increase the speed of convolution operations, thereby shortening the overall calculation time of the object detection model. Ultimately, computer vision-based object detection can be completed at a higher speed, which can improve the response speed and safety and reliability of systems such as autonomous driving.

[0105] After inference is completed, the occupied memory resources can be released according to the operating system scheduling mechanism; the lookup table LUT can be retained in the cache as needed for rapid reuse in the next inference; if it is a multi-threaded or multi-tasking system, access synchronization and consistency of the LUT must be guaranteed; the system can support dynamic loading of different versions of object detection models and lookup tables to improve flexibility.

[0106] In the embodiments of this application, the convolution multiplication operation, which is the most computationally intensive in the YOLO model, is transformed from relying on the complex instruction execution of the processor's arithmetic logic unit (ALU) to simple, fast memory access to pre-calculated results. This greatly reduces the latency of a single multiplication operation. When this acceleration is accumulated over the billions of multiplication operations in the entire YOLO model, it can significantly shorten the total time from the model receiving the image to outputting the s×s×(B×5+C) detection tensor.

[0107] The object detection method based on a lookup table provided in an embodiment of the present application uniformly quantizes the weight parameters of the convolution kernels of all convolution layers in a pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process; creates a two-dimensional multiplication lookup table; inputs the image to be detected into the object detection model, and when the calculation process enters the target convolution layer, obtains the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer; for each element in the feature map, regards the element as a first multiplier, and regards the element corresponding to the element in the convolution kernel as a second multiplier; inputs the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain a binary intermediate product, and shifts and fills each binary intermediate product with zeros and then sums them to obtain a target product; generates a detection tensor based on each target product obtained; decodes the detection tensor to extract detection information of each object in the image. With the lookup table-based object detection method disclosed in this application, the multiplication operation of elements in the feature map with elements in the convolution kernel only involves a lookup table, left shift and zero padding, and binary addition. Compared to using traditional multiplication rules to calculate the product of two numbers, the computer's execution of this series of binary-related operations can increase the computing speed and, consequently, the multiplication speed. Because the multiplication speed is increased, the speed of generating detection tensors is also indirectly increased, ultimately speeding up object detection in images.

[0108] When the object detection method based on the lookup table provided in the embodiment of the present application is applied in the field of autonomous driving, when the autonomous driving vehicle starts, the autonomous driving system calls the camera to shoot images, the autonomous driving system loads the YOLO model and the multiplication lookup table, and uses the multiplication based on the lookup table and binary addition provided in the embodiment of the present application to complete the convolution calculation to obtain the detection tensor, and decodes and extracts the detection information of each object in the image, thereby achieving the purpose of judging the object in the image. When the object in the image is judged, the vehicle's driving path is controlled according to the judgment result to effectively avoid obstacles during driving. Based on the object detection method provided in the embodiment of the present application, a faster reasoning speed can be achieved, which means that in autonomous driving applications, the system can more quickly identify obstacles such as pedestrians and vehicles in front, and gain valuable reaction time for the decision-making and control system, thereby effectively improving the safety and reliability of the autonomous driving system.

[0109] As attached Figure 2 As shown, the method for implementing multiplication based on a lookup table and binary addition in an embodiment of the present application includes the following steps:

[0110] Step 201: Divide the first multiplier into groups starting from the right side to obtain a first group.

[0111] In a preferred embodiment, the level of computing capability of the data processing device may be determined first, and the first number of binary numbers contained in each group may be determined based on the level.

[0112] Among them, the first number is an index of 2, and the first number can be 2, 4, 8, etc.

[0113] The method for implementing multiplication based on a lookup table and binary addition provided in an embodiment of the present application can be applied to electronic devices. The specific data processing function of the electronic device can also be called a data processing device. A storage medium is provided in the electronic device, and the storage medium stores a data processing computer program for implementing multiplication based on a lookup table and binary addition. When the computer program is executed by a processor, the data processing method flow for implementing multiplication based on a lookup table and binary addition is executed.

[0114] The method for implementing multiplication based on lookup tables and binary addition provided in the embodiments of the present application can be applied to fields such as deep learning and image processing. In deep learning, convolution operations are often used extensively, and the essence of convolution operations is matrix multiplication, such as matrix multiplication of an image and a convolution kernel; at the same time, in feedforward neural networks and other neural networks, multiplication is also widely used between the input feature matrix and the weights of each neuron, so the proposal of this application can be effectively applied to deep learning. In image processing, many image transformations and processing rely on multiplication operations, including Fourier transforms in the frequency domain, image compression and feature extraction, etc. These aspects involve a large number of multiplication operations, so this proposal can also be directly applied to image processing. In the embodiments of this application, the application of the method for implementing multiplication based on lookup tables and binary addition to the scene of object detection in an image is used as an example to illustrate.

[0115] It should be noted that the above is only an illustrative list of some scenarios in which the method for implementing multiplication based on lookup tables and binary addition provided in the embodiments of the present application can be applied. In the actual implementation process, it is not limited to this. This method can also be applied to any other appropriate scenario where multiplication calculations exist.

[0116] Before implementing the multiplication process based on a lookup table and binary addition, it is necessary to first determine the first multiplier and the second multiplier of the product to be calculated. The electronic device includes a computer hard disk, a memory, an I / O interface, and a read-write device. During use, the computer reads two binary multipliers to be multiplied through the read-write device or the hard disk. The multipliers can be matrices, vectors, or scalars. In the embodiment of the present application, the elements in the feature map are regarded as the first multiplier, and the corresponding elements in the convolution kernel are regarded as the second multiplier. The first multiplier and the second multiplier are both binary numbers. Each first group contains a first number of binary numbers.

[0117] If the number of binary digits contained in the first multiplier is greater than or equal to the first number, then all the binary digits in the first multiplier can be directly divided into one subgroup. If the number of binary digits contained in the first multiplier is greater than the first number, then the binary digits in the first multiplier must be divided into two or more subgroups. For example, if the first multiplier contains 8 binary digits and the first number is 4, then the first multiplier must be divided into 2 subgroups.

[0118] In an optional embodiment, the method of dividing the first multiplier into groups starting from the right side to obtain the first group may include the following sub-steps:

[0119] Sub-step 1: determining whether the number of binary digits included in the first multiplier is an integer multiple of the first number;

[0120] Sub-step 2: If not, pad the end of the first multiplier with zeros until the number of binary digits contained therein is an integer multiple of the first number, and then perform group division starting from the right side of the zero-padded first multiplier to obtain a first group;

[0121] Sub-step 3: If yes, perform group division starting from the right side of the first multiplier to obtain the first group.

[0122] This optional manner of dividing the multipliers into groups can ensure that each subgroup obtained by the division contains a binary number of the first value.

[0123] Step 202: Divide the second multiplier into groups starting from the right side to obtain second groups.

[0124] Each second group contains a first number of binary numbers.

[0125] The group division rule for the second multiplier is the same as the group division rule for the first multiplier, and will not be repeated here.

[0126] Step 203: extract one group from each of the first group and the second group according to a preset rule and input them into a lookup table to obtain a binary intermediate product.

[0127] The second number of binary numbers included in the binary intermediate product is twice the first number.

[0128] In a preferred embodiment, according to a preset rule, one group is extracted from each of the first group and the second group and input into the lookup table, and a binary intermediate product is obtained in a manner as follows: when both the first group and the second group contain multiple subgroups, each subgroup in the first group and each subgroup in the second group are respectively formed into an intermediate multiplier pair and input into the lookup table to obtain a binary intermediate product.

[0129] Figure 3 Schematic diagram showing a lookup table for calculating intermediate products in an embodiment of the present application. Figure 3 As shown, by inputting two four-bit binary multipliers into the lookup table, an eight-bit binary intermediate product can be obtained. It should be noted that in this case, if both the first and second multipliers are four-bit binary numbers, then the target product can be directly obtained by inputting them into the lookup table. If both the first and second multipliers are larger than four bits, then the first and second multipliers need to be divided into groups of four bits, and the intermediate products are calculated for each of the resulting subgroups. The intermediate products are then shifted and zero-padded, and then binary summed to obtain the target product.

[0130] An exemplary method of inputting each subgroup in the first group and each subgroup in the second group into an intermediate multiplier pair into a lookup table to obtain a binary intermediate product is:

[0131] When the first number is 4, the first group includes the first high-four-bit subgroup and the first low-four-bit subgroup, and the second group includes the second high-four-bit subgroup and the second low-four-bit subgroup, the first low-four-bit subgroup and the second low-four-bit subgroup are input into the lookup table to obtain a first intermediate product; the first low-four-bit subgroup and the second high-four-bit subgroup are input into the lookup table to obtain a second intermediate product; the first high-four-bit subgroup and the second low-four-bit subgroup are input into the lookup table to obtain a third intermediate product; and the first high-four-bit subgroup and the second high-four-bit subgroup are input into the lookup table to obtain a fourth intermediate product.

[0132] Step 204: Shift and fill with zeros on each binary intermediate product and then sum them to obtain the target product.

[0133] An optional method of shifting and zero-padding each binary intermediate product and then summing them to obtain the target product can be as follows:

[0134] First, for each binary intermediate product, the number of left shifts and the number of zero paddings of the binary intermediate product are determined according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product, wherein the number of left shifts and the number of zero paddings corresponding to the same binary intermediate product are the same; the binary intermediate products after shifting and padding are summed to obtain the target product.

[0135] For example: if the first multiplier and the second multiplier both contain 12 binary bits, and the first value is 4, then the first multiplier can be divided into three subgroups: low, middle, and high, and the second multiplier can also be divided into three subgroups: low, middle, and high. For another example: if the first multiplier and the second multiplier both contain 8 binary bits, and the first value is 4, then the first multiplier can be divided into two subgroups: low and high, and the second multiplier can also be divided into two subgroups: low and high.

[0136] In an optional embodiment, for each binary intermediate product, the number of left shifts and the number of zero paddings for the binary intermediate product may be determined according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product as follows:

[0137] For the first intermediate product, determining that both subgroups corresponding to the first intermediate product are low bits, and determining that the first intermediate product is not left-shifted or zero-padded;

[0138] For the second intermediate product and the third intermediate product, determining that the two subgroups corresponding to the second intermediate product and the third intermediate product are both a high bit and a low bit, determining that the second intermediate product and the third intermediate product are both shifted left by four bits, and padding the four shifted bits with zeros;

[0139] For the fourth intermediate product, it is determined that both subgroups corresponding to the fourth intermediate product are high bits, and after the fourth intermediate product is left-shifted by eight bits, the eight bits shifted left are padded with zeros.

[0140] If the first multiplier is divided into subgroups of three levels: low, middle, and high, and the second multiplier is divided into subgroups of three levels: low, middle, and high, then the subgroups corresponding to the first multiplier and the second multiplier are input into the lookup table in pairs, and 6 intermediate products can be obtained. After the 6 intermediate products are shifted and padded with zeros, binary summation is performed to obtain the target product.

[0141] The target product of a multiplier and a second multiplier can be calculated through the above steps 201 to 204. The target product of each element in the feature map and the element corresponding to the convolution kernel can be calculated by repeating or executing the above process in parallel. A detection tensor is generated based on the obtained target products. The generated detection tensor can be used as basic data for object detection in the image.

[0142] The embodiment of the present application provides a method for implementing multiplication based on a lookup table and binary addition, which divides the first multiplier into groups starting from the right side to obtain a first group; divides the second multiplier into groups starting from the right side to obtain a second group; extracts one group from each of the first group and the second group according to a preset rule and inputs it into a lookup table to obtain a binary intermediate product; shifts and fills each binary intermediate product with zeros and then sums them to obtain a target product; and performs data processing based on the target product. Through the method for implementing multiplication based on a lookup table and binary addition provided by the embodiment of the present application, the entire product calculation operation process only involves a lookup table, left shift and zero filling, and binary addition, and the computer executing the above series of binary related operations can improve the operation speed and thus the multiplication operation speed compared to using traditional multiplication calculation rules to calculate the product of two numbers. Not only that, since there is no need to introduce a complex product calculation circuit, the hardware requirements for data processing equipment can be reduced, saving computing resources.

[0143] The following two specific examples illustrate the method for implementing multiplication based on a lookup table and binary addition provided in the embodiments of the present application.

[0144] In this example, we use the English letters a, b, c, ... to represent the decimal digits 0, ..., 9, and the Greek letters α, β, etc. to represent the binary digits 0 and 1. In the first step, we create a lookup table, such as Figure 3As shown. The output of the lookup table is an eight-bit binary number γ7γ6γ5γ4γ3γ2γ1γ0, which is used to represent the 256 decimal numbers from 0 to 255. The input of the lookup table is two four-bit binary multipliers α3α2α1α0 and β3β2β1β0, which are used to represent the sixteen decimal integers from 0 to 15. The lookup table is set up so that its output value corresponds to the product of the two decimal numbers represented by α3α2α1α0 and β3β2β1β0. For example: if α3α2α1α0 = "0101" and β3β2β1β0 = "1110", they represent 5 and 14 in decimal, respectively. The product of the product is 70, which is "01000110" in binary. Therefore, the output of the lookup table is γ7γ6γ5γ4γ3γ2γ1γ0 = "01000110". The symbol φ represents a query to the lookup table, that is, the product calculation of this lookup table can be expressed as:

[0145] γ7γ6γ5γ4γ3γ2γ1γ0=φ(α3α2α1α0, β3β2β1β0). (1)

[0146] In the design of the lookup table, the lookup table alone can accomplish the situation corresponding to the multiplication of two single-digit numbers in the decimal system.

[0147] For more complex multipliers, this specific example relies on Figure 3 It is completed by the lookup table and binary shift and zero padding. Assume that the two multipliers to be multiplied are α7α6α5α4α3α2α1α0 and β7β6β5β4β3β2β1β0, that is, each multiplier is an eight-bit binary number (corresponding to a decimal number from 0 to 255). We regard the above binary multipliers as a group of four bits starting from the rightmost bit, that is, [α7α6α5α4][α3α2α1α0] and [β7β6β5β4][β3β2β1β0]. Define [α7α6α5α4] and [β7β6β5β4] as the upper four bits, and define [α3α2α1α0]] and [β3β2β1β0] as the lower four bits. Perform four queries on the lookup table to obtain four intermediate products, that is,

[0148]

[0149] The superscript of each intermediate product represents whether it comes from the upper or lower four bits of the two multipliers, such as HH, which means that the product comes from the upper four bits of the two multipliers. Because each four bits in binary corresponds to the fourth power of 2, that is, 16, in order to obtain the final product, the above four intermediate results, namely the intermediate products, need to be added, but they need to be shifted and padded with zeros before adding. Among them, formula (2) does not require shifting and padded with zeros because it is itself the multiplication of the lower bits of the two multipliers. Formulas (3) and (4) are the multiplication of the lower and upper bits, so the results should be multiplied by 16 in decimal, which is expressed in binary as shifting four bits to the left and padded with four zeros. Formula (5) is the multiplication of two high bits, so the result should be multiplied by 16 twice in decimal, that is, 16×16, which is expressed in binary as shifting eight bits to the left and padded with zeros, so the final product is shown in the following formula (6):

[0150]

[0151] The above operation involves four lookup table queries, two left shifts and zero padding, and three binary addition operations, but no multiplication, so it can be calculated efficiently. At the same time, by analyzing formula (6), it can be found that its first term is an eight-bit binary number, while the last term is a 16-bit binary number but the rightmost eight bits are zero. Therefore, the right eight bits of the last term can be directly filled with the first term, thus making formula (6) more concise as the following formula (7):

[0152]

[0153] Compared with formula (6), formula (7) saves one binary addition operation.

[0154] For more complex multiplication, such as a multiplier with a 12-bit binary α 11 α 10 α9α8α7α6α5α4α3α2α1α0 and another multiplier is an eight-bit binary number β7β6β5β4β3β2β1β0. In this specific example, starting from the rightmost bit of each multiplier, every four bits are divided into a group, that is, [α 11 α 10 α9α8][α7α6α5α4][α3α2α1α0] and [β7β6β5β4][β3β2β1β0], and the lookup table is queried six times, i.e.

[0155] φ(α3α2α1α0, β3β2β1β0) (8)

[0156] φ(α3α2α1α0, β7β6β5β4) (9)

[0157] φ(α7α6α5α4,β3β2β1β0) (10)

[0158] φ(α7α6α5α4,β7β6β5β4) (11)

[0159] φ(α 11 α 10 α9α8,β3β2β1β0) (12)

[0160] φ(α 11 α 10 α9α8,β7β6β5β4) (13)

[0161] In this specific example, φ(.) is directly used to represent the table lookup result. Then, the six intermediate results, i.e., intermediate products, are left-shifted and zero-filled. Among them, equation (8) is directly retained. After adding equations (9) and (10), they are left-shifted four bits and zero-filled. After adding equations (11) and (12), they are left-shifted eight bits and zero-filled. After adding equation (13), they are left-shifted 12 bits and zero-filled. The final product result (i.e., the target product) is shown in equation (14):

[0162]

[0163] More preferably, the number of addition operations in formula (14) can be further reduced. Specifically, the eight zeros on the right side of the third term can be directly replaced by the first term, and the 12 zeros on the right side of the fourth term can be directly replaced by the second term, thereby obtaining a faster operation as shown in the following formula (15):

[0164]

[0165] By analyzing equation (15), we can see that the calculation of equation (15) involves six lookup table queries, one left shift and zero padding, and three binary addition operations, but no multiplication. Furthermore, for equations (6), (7), (14), and (15), the lookup table queries and left shift and zero padding steps can be computed in parallel, further improving the computation speed.

[0166] This specific example illustrates multiplication using a lookup table and binary addition, using binary multipliers with 4, 8, and 16 bits. In actual implementation, if the binary multiplier's digits are not a power of 2, such as 4, 8, or 12, the binary multiplier can be padded with zeros on the left to bring it to a power of 2, such as 8, 12, or 16, to facilitate processing. Similarly, for multipliers with more digits, such as 32 bits, the above principles can still be followed to calculate the target product through table lookup, left shift padding, and binary addition.

[0167] This specific example provides a method for implementing multiplication based on a lookup table and binary addition. No multiplication operations are required during the entire calculation process. Essential operations only involve table lookup, left shift with zero padding, and binary addition. These operations are all very fast in computers, and therefore can improve the speed of multiplication operations overall.

[0168] Figure 4 A structural block diagram of an object detection device based on a lookup table is provided to implement an embodiment of the present application.

[0169] The object detection device based on the lookup table provided in the embodiment of the present application includes the following functional modules:

[0170] A unified quantization module 401 is configured to perform unified quantization processing on weight parameters of convolution kernels of all convolutional layers in a pre-trained object detection model and activation values ​​of feature maps generated during inference, wherein the unified quantization processing includes: representing a 32-bit floating-point number as a first number of fixed-point integers; the first number is an exponent of 2;

[0171] A lookup table creation module 402 is configured to create a two-dimensional multiplication lookup table, wherein the two-dimensional multiplication lookup table includes products of a plurality of first-number-bit fixed-point binary number pairs;

[0172] An acquisition module 403 is configured to input the image to be detected into the object detection model, and when the calculation process enters the target convolution layer, obtain the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel included in the quantized target convolution layer;

[0173] a multiplier determination module 404 configured to, for each element in the feature map, regard the element as a first multiplier and regard the element corresponding to the element in the convolution kernel as a second multiplier;

[0174] a calculation module 405 configured to input the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and perform zero-padding on each of the binary intermediate products before summing them to obtain a target product;

[0175] A detection tensor generation module 406 is configured to generate a detection tensor based on the obtained target products;

[0176] The decoding module 407 is used to decode the detection tensor and extract the detection information of each object in the image.

[0177] Optionally, the unified quantization module is specifically configured to:

[0178] The weight parameters of the convolution kernels of all convolutional layers in the object detection model are converted from 32-bit floating-point numbers to 8-bit fixed-point integers using a quantization tool; and the activation values ​​of the input feature maps generated during the inference process of the object detection model are converted into 8-bit fixed-point integers.

[0179] Optionally, the calculation module includes:

[0180] a first grouping submodule, configured to group the first multiplier starting from the right side to obtain first groups, wherein each first group contains the first number of binary numbers;

[0181] A second grouping submodule, configured to group the second multiplier starting from the right side to obtain second groups, wherein each second group contains the first number of binary numbers;

[0182] a table lookup submodule, configured to extract one group from each of the first group and the second group according to a preset rule and input the group into a lookup table to obtain a binary intermediate product, wherein a second number of binary numbers included in the binary intermediate product is twice the first number;

[0183] The shift processing submodule is used to perform shifting and zero-filling on each of the binary intermediate products and then sum them to obtain a target product.

[0184] Optionally, the first grouping submodule includes:

[0185] a judging unit, configured to judge whether the number of binary numbers included in the first multiplier is an integer multiple of the first number;

[0186] The execution unit is configured to, if the answer is no, pad the end of the first multiplier with zeros until the number of binary digits contained therein is an integer multiple of the first number, and then perform group division starting from the right side of the first multiplier after the zero padding to obtain a first group; and if the answer is yes, directly perform group division starting from the right side of the first multiplier to obtain a first group.

[0187] Optionally, the table lookup submodule is specifically used to:

[0188] When both the first group and the second group contain multiple subgroups, each subgroup in the first group and each subgroup in the second group are respectively combined into an intermediate multiplier pair and input into a lookup table to obtain a binary intermediate product.

[0189] Optionally, the shift processing submodule includes:

[0190] a zero-padding unit configured to determine, for each binary intermediate product, a number of left shift bits and a number of zero-padding bits of the binary intermediate product according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product, wherein the number of left shift bits and the number of zero-padding bits corresponding to the same binary intermediate product are the same;

[0191] The summing unit is used to sum the binary intermediate products after shifting and filling with zeros to obtain a target product.

[0192] Optionally, when the first number is 4, the first group includes a first upper four-bit subgroup and a first lower four-bit subgroup, and the second group includes a second upper four-bit subgroup and a second lower four-bit subgroup, the table lookup submodule is specifically configured to:

[0193] Inputting the first lower four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a first intermediate product;

[0194] Inputting the first low-order four-bit subgroup and the second high-order four-bit subgroup into a lookup table to obtain a second intermediate product;

[0195] Inputting the first upper four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a third intermediate product;

[0196] The first upper four-bit subgroup and the second upper four-bit subgroup are input into a lookup table to obtain a fourth intermediate product.

[0197] Optionally, the zero-padding unit is specifically configured to:

[0198] For the first intermediate product, determining that both subgroups corresponding to the first intermediate product are low bits, and determining that the first intermediate product is not to be left-shifted or padded with zeros;

[0199] For the second intermediate product and the third intermediate product, determining that the two subgroups corresponding to the second intermediate product and the third intermediate product both have one high bit and one low bit, determining that the second intermediate product and the third intermediate product are both shifted left by four bits, and padding the four shifted bits with zeros;

[0200] For the fourth intermediate product, it is determined that both subgroups corresponding to the fourth intermediate product are high bits, and after the fourth intermediate product is left-shifted by eight bits, the eight bits shifted left are padded with zeros.

[0201] The object detection device based on a lookup table provided in an embodiment of the present application uniformly quantizes the weight parameters of the convolution kernels of all convolution layers in a pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process; creates a two-dimensional multiplication lookup table; inputs the image to be detected into the object detection model, and when the calculation process enters the target convolution layer, obtains the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer; for each element in the feature map, regards the element as a first multiplier, and regards the element corresponding to the element in the convolution kernel as a second multiplier; inputs the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain a binary intermediate product, and shifts and fills each binary intermediate product with zeros and then sums them to obtain a target product; generates a detection tensor based on each target product obtained; decodes the detection tensor to extract detection information of each object in the image. With the lookup table-based object detection device disclosed in this application, the product operation of elements in the feature map and elements in the convolution kernel only involves a lookup table, left shift and zero padding, and binary addition. Compared to using traditional multiplication rules to calculate the product of two numbers, the computer's execution of this series of binary-related operations can increase the computing speed and, consequently, the multiplication speed. Because the multiplication speed is increased, the speed of generating detection tensors is also indirectly increased, ultimately speeding up object detection in images.

[0202] An embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0203] Memory for storing computer programs;

[0204] The processor is configured to implement the object detection method based on the lookup table shown in the above method embodiment when executing the program stored in the memory.

[0205] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0206] The communication interface is used for communication between the above terminal and other devices.

[0207] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0208] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, which stores instructions. When the computer-readable storage medium is executed on an electronic device, the electronic device implements the object detection method based on the lookup table described in any of the above embodiments.

[0209] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on an electronic device, the electronic device implements the object detection method based on the lookup table described in any one of the above embodiments.

[0210] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, C language, VHDL language, Verilog language, object-oriented programming language Java, and directly interpreted scripting language JavaScript, etc.

[0211] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0212] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0213] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0214] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0215] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0216] In this application, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be understood broadly. For example, they may refer to fixed connections, detachable connections, or integration; mechanical connections, electrical connections, or communication; direct connections or indirect connections through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of these terms in this application based on specific circumstances.

[0217] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0218] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for object detection based on a lookup table, characterized in that: The method comprises: Performing unified quantization on weight parameters of convolution kernels of all convolutional layers in a pre-trained object detection model and activation values ​​of feature maps generated during inference, wherein the unified quantization includes: representing 32-bit floating-point numbers as fixed-point integers with a first number being an exponent of 2; Creating a two-dimensional multiplication lookup table; wherein the two-dimensional multiplication lookup table includes products of a plurality of first-number-bit fixed-point binary number pairs; Input the image to be detected into the object detection model. When the calculation process enters the target convolution layer, obtain the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer; For each element in the feature map, consider the element as a first multiplier, and consider the element corresponding to the element in the convolution kernel as a second multiplier; Inputting the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and performing zero-padding on each of the binary intermediate products and then summing them to obtain a target product; Generate a detection tensor based on the obtained target products; The detection tensor is decoded to extract detection information of each object in the image.

2. The method according to claim 1, characterized in that The steps for uniformly quantizing the weight parameters of the convolution kernels of all convolutional layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during inference include: Using a quantization tool to convert the weight parameters of the convolution kernels of all convolutional layers in the object detection model from 32-bit floating point numbers to 8-bit fixed point integers; The input feature map activation values ​​generated during the object detection model inference process are converted into 8-bit fixed-point integers.

3. The method according to claim 1, characterized in that The step of inputting the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and performing shifting and zero-padding on each of the binary intermediate products and then summing them to obtain a target product comprises: Dividing the first multiplier into groups starting from the right side to obtain first groups, wherein each first group contains the first number of binary numbers; Dividing the second multiplier into groups starting from the right side to obtain second groups, wherein each second group contains the first number of binary numbers; Extracting one group from each of the first group and the second group according to a preset rule and inputting them into a lookup table to obtain a binary intermediate product, wherein a second number of binary numbers included in the binary intermediate product is twice the first number; The binary intermediate products are shifted and zero-filled, and then summed to obtain a target product.

4. The method according to claim 3, characterized in that The step of dividing the first multipliers into groups starting from the right side to obtain first groups includes: determining whether the number of binary digits included in the first multiplier is an integer multiple of the first number; If not, padding the end of the first multiplier with zeros until the number of binary digits included is an integer multiple of the first number, and then performing group division starting from the right side of the first multiplier after the zero padding to obtain a first group; If so, group division is performed directly starting from the right side of the first multiplier to obtain a first group.

5. The method according to claim 3, characterized in that The step of extracting one group from each of the first group and the second group according to a preset rule and inputting the groups into a lookup table to obtain a binary intermediate product includes: When both the first group and the second group contain multiple subgroups, each subgroup in the first group and each subgroup in the second group are respectively combined into an intermediate multiplier pair and input into a lookup table to obtain a binary intermediate product.

6. The method according to claim 3, characterized in that The step of performing shifting and zero-padding on each of the binary intermediate products and then summing the products to obtain a target product comprises: For each of the binary intermediate products, determining the number of left shifts and the number of zero paddings for the binary intermediate product according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product, wherein the number of left shifts and the number of zero paddings corresponding to the same binary intermediate product are the same; The binary intermediate products after shifting and zero-filling are summed to obtain a target product.

7. The method according to any one of claims 5 or 6, characterized in that When the first number is 4, the first group includes a first upper four-bit subgroup and a first lower four-bit subgroup, and the second group includes a second upper four-bit subgroup and a second lower four-bit subgroup, the step of respectively forming intermediate multiplier pairs from each subgroup in the first group and each subgroup in the second group and inputting them into a lookup table to obtain a binary intermediate product includes: Inputting the first lower four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a first intermediate product; Inputting the first low-order four-bit subgroup and the second high-order four-bit subgroup into a lookup table to obtain a second intermediate product; Inputting the first upper four-bit subgroup and the second lower four-bit subgroup into a lookup table to obtain a third intermediate product; The first upper four-bit subgroup and the second upper four-bit subgroup are input into a lookup table to obtain a fourth intermediate product.

8. The method according to claim 7, characterized in that For each of the binary intermediate products, the step of determining the number of left shifts and the number of zero paddings for the binary intermediate product according to the high and low bit levels of the two subgroups corresponding to the binary intermediate product includes: For the first intermediate product, determining that both subgroups corresponding to the first intermediate product are low bits, and determining that the first intermediate product is not to be left-shifted or padded with zeros; For the second intermediate product and the third intermediate product, determining that the two subgroups corresponding to the second intermediate product and the third intermediate product both have one high bit and one low bit, determining that the second intermediate product and the third intermediate product are both shifted left by four bits, and padding the four shifted bits with zeros; For the fourth intermediate product, it is determined that both subgroups corresponding to the fourth intermediate product are high bits, and after the fourth intermediate product is left-shifted by eight bits, the eight bits shifted left are padded with zeros.

9. An object detection device based on a lookup table, characterized in that: The device comprises: A unified quantization module is configured to uniformly quantize the weight parameters of the convolution kernels of all convolutional layers in the pre-trained object detection model and the activation values ​​of the feature maps generated during the inference process, wherein the unified quantization processing includes: representing the 32-bit floating-point number as a first number of fixed-point integers; the first number is an exponent of 2; A lookup table creation module, configured to create a two-dimensional multiplication lookup table; wherein the two-dimensional multiplication lookup table includes products of a plurality of first-number-bit fixed-point binary number pairs; An acquisition module is used to input the image to be detected into the object detection model, and when the calculation process enters the target convolution layer, obtain the feature map corresponding to the uniformly quantized image to be detected and the convolution kernel contained in the quantized target convolution layer; a multiplier determination module, configured to, for each element in the feature map, regard the element as a first multiplier and regard the element corresponding to the element in the convolution kernel as a second multiplier; a calculation module, configured to input the first multiplier and the second multiplier into the two-dimensional multiplication lookup table to obtain binary intermediate products, and perform zero-padding on each of the binary intermediate products and then sum them to obtain a target product; A detection tensor generation module, configured to generate a detection tensor based on the obtained target products; The decoding module is used to decode the detection tensor and extract the detection information of each object in the image.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; The processor is configured to implement the object detection method based on the lookup table according to any one of claims 1 to 8 when executing the program stored in the memory.

Images