A deep learning target detection method, device and system

Abstract

The application provides a deep learning target detection method based on raster scanning or video acquisition image, including: sequentially scanning full scene information row by row to obtain full frame target image data; the full frame target image data includes static target image data and moving target image data in the scene; sequentially dividing the full frame target image data into N blocks to obtain N block data; wherein there is no overlapping area between each block of the N blocks, and N is a natural number greater than 2; sequentially and alternately writing adjacent two block data in the N block data into a first cache area and a second cache area; a target perception backbone network sequentially and alternately reads adjacent two block data in the first cache area and the second cache area; the target perception backbone network extracts N block feature vectors according to the N block data; S15, sequentially fusing the N block feature vectors to obtain a fusion feature; and performing post-processing based on the fusion feature to obtain the class and position information of the target.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a deep learning-based object detection method, apparatus, and system. Background Technology

[0002] Object classification or recognition is a fundamental research topic in the field of deep learning. Since AlexNet topped the ImageNet classification accuracy charts in 2012, numerous deep learning optimization techniques have been developed based on image classification tasks, achieving significant results in many areas such as scene classification and object recognition. Furthermore, object detection is also a key technology in deep learning. Since the advent of R-CNN in 2014, deep learning-based object detection technology has developed rapidly, and many deep learning-based object perception technologies have been successfully implemented in numerous fields such as security monitoring, autonomous driving, and robotics.

[0003] The biggest problem with deep learning-based target perception technology is the huge demand for computational and storage resources. Many network miniaturization techniques, such as network pruning, network sparsity, and low-bit quantization, have emerged to make it possible for deep learning technology to be applied. However, in the field of low-power edge devices, especially in the design of low-power chips at the milliwatt and microwatt levels, existing neural network optimization techniques are still difficult to meet the computational resource and perception accuracy requirements of practical application scenarios. Summary of the Invention

[0004] To address the aforementioned problems, embodiments of this application provide a deep learning-based object detection method, apparatus, and system.

[0005] In a first aspect, embodiments of this application provide a deep learning-based target detection method based on images captured by raster scanning or video acquisition. The method includes: sequentially scanning the entire scene information line by line to obtain full-frame target image data; the full-frame target image data includes static target image data and moving target image data at any location in the scene; sequentially dividing the full-frame target image data into N blocks to obtain N block data; wherein there is no overlapping area between each of the N blocks, and N is a natural number greater than 2; sequentially and alternately writing adjacent blocks of the N blocks into a first buffer and a second buffer; simultaneously, a target perception backbone sequentially and alternately reading the adjacent blocks in the first and second buffers; the target perception backbone extracting N block feature vectors based on the N block data; sequentially fusing the N block feature vectors to obtain fused features; and performing post-processing based on the fused features to obtain the target's category and location information. Therefore, this application decouples block feature extraction from feature fusion. The first stage performs independent block feature extraction, and the second stage further abstracts the fused features, which can effectively recover the spatial correlation lost during the independent block feature extraction process in the first stage. This architecture can approximate the target perception accuracy of inference from directly input high-resolution data to the greatest extent possible while significantly reducing data caching.

[0006] In some embodiments, the full-scene information includes foreground information and background information. The step of sequentially scanning the full-scene information line by line to obtain full-frame target image data includes: using a raster scanning mode to scan the foreground and background information line by line from top to bottom to obtain the full-frame target image data. This allows for the effective perception of humanoid targets, both stationary and moving, at any location within the scene.

[0007] In some embodiments, dividing the full-frame target image data into N blocks according to the specified order to obtain N block data includes: dividing the full-frame target image data into N blocks at equal intervals from top to bottom along the scanning direction, with no overlapping areas between adjacent blocks, and each of the N blocks including H rows of target image data. Therefore, the end-to-end computational load does not increase with the increase in the number of blocks during network inference.

[0008] In some embodiments, the step of alternately writing two adjacent blocks of data from the N blocks of data to the first cache and the second cache in the stated order includes: writing two adjacent blocks of data from the N blocks of data to the first cache and the second cache respectively in the stated top-down order; and after the blocks of data in the first cache and the second cache are read, writing the next two adjacent blocks of data to the first cache and the second cache respectively in the stated order. Thus, by using a "ping-pong buffer" mechanism to alternately write and read two adjacent blocks of data from the two caches, the problem of caching large-resolution data is solved. During network inference, the additional storage capacity does not increase with the number of blocks, greatly alleviating the data caching problem.

[0009] In some embodiments, the target perception backbone sequentially and alternately reads two adjacent blocks of data in the first buffer and the second buffer, including: after writing the i-th block of data in the first buffer, the target perception backbone reads the i-th block of data in the first buffer; after writing the (i+1)-th block of data in the second buffer, the target perception backbone reads the (i+1)-th block of data in the second buffer. Thus, since the input data resolution for a single inference iteration of the neural network is only 1 / N of the original data, the operating memory and intermediate layer data cache of the neural network are also only 1 / N of the original resolution, significantly reducing the end-to-end additional memory capacity and computational load.

[0010] In some embodiments, the target perception backbone network includes a first subnetwork. The target perception backbone network extracts N block feature vectors based on the N block data, including: the first subnetwork downsampling the N block data to reduce the resolution of the N blocks and extracting N block feature vectors. In this way, the target perception backbone network is segmented in the intermediate layer, and the first subnetwork reduces the input buffer and computational load of the second subnetwork while maintaining perception accuracy.

[0011] In some embodiments, fusing the N block feature vectors sequentially to obtain fused features includes: fusing the N block feature vectors in a top-down order to obtain fused features. In this way, the N block features can be sequentially concatenated along an H-dimensional sequence from 0 to N-1 to approximate the effect of directly inputting high-resolution target perception.

[0012] In some embodiments, the target perception backbone network includes a second subnetwork. The step of post-processing based on the fused features to obtain the target's category and location information includes: the second subnetwork performing convolution and pooling operations at the junctions of the fused features to obtain optimized features; and post-processing the optimized features to obtain the target's category and location information. Thus, the second subnetwork's convolutional operations perform further convolution and pooling operations at the junctions of the fused feature blocks, patching the "gaps" between feature blocks to restore spatial correlation, which helps improve perception accuracy.

[0013] In some embodiments, the target perception backbone network is one of the following network architectures: visual neural network architecture, audio / video neural network architecture, text recognition neural network architecture, and speech recognition neural network architecture. This allows for the development of deep learning network architectures that meet specific scenario requirements, including constraints such as power consumption, chip area, storage, and accuracy.

[0014] In some embodiments, the target includes one of the following: a semantic-level target, or a pixel-level target. This can be extended to any deep learning task.

[0015] In some embodiments, the method is applied to low-power edge devices such as low-power home monitoring, home robots, or IoT visual monitoring devices. This enables milliwatt-level or even microwatt-level feature extraction and pattern recognition system architectures, significantly reducing power consumption compared to existing visual sensor chips, and offering advantages such as high specialization, small area, and ultra-low power consumption.

[0016] Secondly, embodiments of this application provide a deep learning-based target detection apparatus for performing the method described in any one of the first aspects. The apparatus includes at least: an image scanning module for sequentially scanning the entire scene information line by line to obtain full-frame target image data; the full-frame target image data includes stationary target image data and moving target image data at any location in the scene; a block partitioning module for sequentially dividing the full-frame target image data into N blocks to obtain N block data; wherein there is no overlap between each of the N blocks, and N is a natural number greater than 2; an alternating buffering module for sequentially and alternately writing adjacent blocks of the N block data into a first buffer and a second buffer; a target perception backbone network for sequentially and alternately reading the adjacent blocks of data in the first buffer and the second buffer; extracting N block feature vectors based on the N block data; a fusion module for sequentially fusing the N block feature vectors to obtain fused features; and a post-processing module for performing post-processing based on the fused features to obtain the target's category and location information. Its beneficial effects are as described in the first aspect and will not be repeated here.

[0017] Thirdly, embodiments of this application provide a deep learning-based object detection system, the system including apparatus for deep learning-based object detection, wherein the apparatus is used to perform the method as described in any one of the first aspects to obtain the category and location information of the object. Its beneficial effects are as described in the first aspect, and will not be repeated here.

[0018] Fourthly, embodiments of this application provide a computer device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of the first aspects. Its beneficial effects are as described in the first aspect and will not be repeated here.

[0019] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any of the first aspects. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0022] Figure 1 A schematic diagram illustrating the method for reducing network computing resources provided in the first scheme;

[0023] Figure 2 A schematic diagram of an embedded real-time human detection method provided for the second scheme;

[0024] Figure 3 A schematic diagram of a preprocessing method for ultra-high resolution image training data proposed in the third scheme;

[0025] Figure 4 A flowchart of a deep learning-based object detection method is provided in this application embodiment;

[0026] Figure 5 A schematic diagram of a deep learning-based object detection method provided in Embodiment 1 of this application;

[0027] Figure 6 A schematic diagram of the deep learning-based object detection method provided in Embodiment 2 of this application;

[0028] Figure 7 A schematic diagram of a deep learning-based object detection apparatus provided in an embodiment of this application;

[0029] Figure 8 A schematic diagram of a deep learning-based object detection system is provided as an embodiment of this application;

[0030] Figure 9 A schematic diagram of a computing device provided in an embodiment of this application;

[0031] Figure 10 A schematic diagram of a computing device cluster provided in an embodiment of this application;

[0032] Figure 11 This is a schematic diagram illustrating a connection method between computing device clusters provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0034] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0035] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0036] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0037] In the description of the embodiments in this application, "some embodiments" are mentioned, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0038] In the description of the embodiments of this application, the terms "first, second, third, etc." or module A, module B, module C, etc. are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that, where permitted, a specific order or sequence can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In the description of the embodiments of this application, the reference numerals for the steps involved, such as tables for S110, S120, etc., do not indicate that the steps will necessarily be executed. Where permissible, the order of the steps can be interchanged or executed simultaneously.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0041] Figure 1 A schematic diagram illustrating the method for reducing network computing resources provided by the first solution. (See diagram below.) Figure 1 As shown, inputting the original sampled data of large, medium, and small targets into the target perception network yields relatively stable perception results for large, medium, and small targets. However, the original image resolution is too high, resulting in a large amount of sampled data and high computational and storage requirements. The first approach reduces the sampled data through downsampling. After downsampling, the image resolution is reduced, and the original large, medium, and small targets are transformed into medium, small, and extremely small targets.

[0042] In particular, extremely small targets may be missed due to their low image resolution, making them undetectable. Small targets, with their already limited sampling information after image resolution reduction, will suffer further feature loss after passing through several pooling layers, leading to unstable final perception results and affecting perception accuracy. Multiple downsampling processes significantly weaken the ability of deep learning-based target perception networks to perceive medium- and small-scale targets.

[0043] Figure 2 A schematic diagram of an embedded real-time human detection method provided for the second scheme. (See diagram below.) Figure 2As shown, the second approach involves defining one or more regions of interest (ROIs) within a full-scene image. ROIs are typically determined by prior human observation of locations where targets are likely to appear, such as entrances / exits or stairwells. A specific algorithm is used to roughly estimate areas in the scene where human figures might exist, such as areas where movement is possible. Before being fed into the perceptual network, the ROIs undergo a certain degree of downsampling and scaling.

[0044] The second approach involves pre-training a neural network-based human detection model. This model is then used to acquire raw images within the target space, which can include large, medium, and small targets. Regions of Interest (ROIs) are calculated, resulting in ROI matting. Based on the ROI matting, a detection phase is selected, dynamically adjusting and reducing the image resolution for downsampling. The downsampled image data is then input into the human detection model. Using the human detection model, a multi-core parallel algorithm and a fast convolution algorithm are combined to perform convolution operations on the input image to obtain feature data. Multiple candidate human bounding boxes are obtained based on the feature data, and after filtering and fusion, the final human target is determined.

[0045] The second approach suffers from the problem of missing appearance information for large and medium-sized targets. This method typically involves manually setting up or using specific algorithms to detect Regions of Interest (ROIs) in the entire scene image where human figures are likely to be present. The human detection model only needs to process the data within the ROI, which can effectively reduce a large amount of useless interference data. However, in some doorbell and access control scenarios, human targets are usually close to the camera. Manually or algorithmically set ROIs can easily cause human targets to be truncated, resulting in the loss of human appearance and contour information, which seriously affects the accuracy of target perception.

[0046] The second approach suffers from uncertainty in ROI selection. This method utilizes SMD motion detection to perceive regions in the image where there are pixel differences between consecutive frames, using these ROIs as input to the target perception network. While this can remove a large number of useless background regions to some extent, SMD can only perceive regions with pixel jumps in the image. It cannot handle misjudgments caused by non-human moving targets or pixel differences between consecutive frames due to lighting changes. A large number of misjudged regions will put pressure on the computational resources and storage of subsequent human detection processes, affecting the overall end-to-end computational efficiency.

[0047] The second approach cannot detect stationary human targets. This method mainly relies on SMD to detect moving areas in the scene and further detect and locate the moving areas. For stationary human targets in the scene, since they do not cause pixel jumps in the human area between consecutive frames, they cannot be detected by SMD, resulting in missed detection of stationary human targets.

[0048] Figure 3This is a schematic diagram of a preprocessing method for ultra-high resolution image training data proposed in the third scheme. Figure 3 As shown, this method uses ultra-high resolution image overlapping and slicing technology to reduce network technology resources, thereby improving the neural network computation efficiency in ultra-high resolution target detection scenarios, while achieving the function of accurate detection and localization of weak targets.

[0049] The third approach sets a scale factor for the block ratio and a block overlap ratio based on the size of the ultra-high resolution image. The ultra-high resolution image to be detected is divided into blocks according to the block scale factor and the overlap ratio to obtain multiple block images. The labels of the original image are used to calculate new labels for the block images based on the block scale and the overlap ratio to set the intersection threshold T between specific targets and sliding crop boxes. Targets with overlapping edges are filtered based on the calculated T between the block images and the original labels, ultimately obtaining new multi-scale block image data that is easy to train and infer.

[0050] The third approach relies on a large data cache. In principle, this method reduces the storage capacity and computational efficiency in a single calculation, but it requires a certain proportion of overlap between adjacent image blocks.

[0051] The third approach requires a larger additional storage capacity, which increases significantly with the number of neural network layers. Furthermore, this method needs to store the entire original high-resolution image before slicing, relying on a large data buffer.

[0052] The third approach involves significant end-to-end computation. While this method reduces the computational cost and intermediate buffering of a single network inference by slicing the high-resolution image into blocks, the actual amount of data processed by the network is much larger than the original high-resolution image due to the overlap between blocks. Moreover, the computational cost increases linearly with the number of image blocks, and the end-to-end computational efficiency is not substantially reduced. In fact, it is higher than that of directly inputting the original high-resolution image for inference.

[0053] The slicing method provided by the third scheme is incompatible with general data scanning methods. The slicing mode in the third scheme is incompatible with the video image acquisition method based on raster scanning (row buffer scanning) commonly used in the industry. Moreover, in some extremely low-power edge application scenarios, there is often no DDR to cache data, so dynamic pipeline scanning and calculation of data cannot be achieved.

[0054] In the first, second, and third schemes, whether performing calculations by scaling the image or slicing the image horizontally and vertically, there are issues with storing the original full-frame data and the large data volume. For some ultra-low-power edge chips, there is no dedicated DDR for data storage, making it impossible to apply deep learning perception technology in such scenarios.

[0055] To save computational overhead and storage resources, the first, second and third schemes usually do not perform detection based on the whole frame image. Instead, they use manual priors or SMD to extract regions where human figures may exist in advance. Although a certain proportion of useless interference information is removed, the set ROI region will inevitably truncate human figures or miss human figures outside the ROI region, resulting in the loss of some or all of the target information.

[0056] The first, second, and third schemes mainly use overlapping block processing to alleviate the problem of low computational efficiency in high-resolution inference. Although the computational efficiency of single inference is significantly improved compared to full-frame image inference, there are overlapping areas between adjacent blocks, and the amount of data processed end-to-end is greater than the amount of data in the original full frame. Therefore, the overall computational efficiency has not been substantially improved, and the network computational efficiency is low.

[0057] Figure 4 This document provides a flowchart of a deep learning-based object detection method as an embodiment of this application. Figure 4 As shown, the image based on raster scanning or video acquisition includes: S11, sequentially scanning the entire scene information line by line to obtain full-frame target image data; the full-frame target image data includes static target image data and moving target image data at any location in the scene; S12, dividing the full-frame target image data into N blocks sequentially to obtain N block data; wherein there is no overlapping area between each of the N blocks, and N is a natural number greater than 2; S13, sequentially and alternately writing two adjacent blocks of data from the N blocks into the first buffer and the second buffer; simultaneously, the target perception backbone sequentially and alternately reads two adjacent blocks of data from the first buffer and the second buffer; S14, the target perception backbone extracts N block feature vectors based on the N block data; S15, sequentially fusing the N block feature vectors to obtain fused features; and performing post-processing based on the fused features to obtain the target's category and location information.

[0058] The deep learning-based object detection method provided in this application employs a row-buffered dynamic pipelined scanning mode for data reading. During the top-down, row-by-row scanning process, a "ping-pong caching" mechanism provides dual buffers for data writing and reading. Each buffer stores only a single block of data from the entire frame at a time, and this single block can consist of a fixed number of H rows, with no overlap between blocks. This prevents the additional storage capacity from increasing with the number of blocks during network inference, significantly alleviating the data caching problem.

[0059] The deep learning-based target detection method provided in this application scans the foreground and background information of the entire scene line by line from top to bottom, which can effectively perceive humanoid targets, including stationary and moving ones, at any location in the scene.

[0060] This application proposes a non-overlapping raster segmentation method that divides high-resolution full-frame data into multiple blocks from top to bottom. There are no overlapping areas between each block, and the end-to-end computational load does not increase with the increase of the number of blocks during network inference.

[0061] Example 1

[0062] Figure 5 This is a schematic diagram of a deep learning-based object detection method provided in Embodiment 1 of this application. Figure 4 As shown, this method uses images scanned by a non-overlapping network raster to perform block processing, including the following steps:

[0063] S501 scans the entire scene information line by line in sequence to obtain the full frame target image data.

[0064] In some embodiments, a raster scanning mode can be used to scan the foreground and background information line by line from top to bottom to obtain full-frame target image data. The full-frame target image data includes static target image data and moving target image data at any location in the scene.

[0065] For example, the target can be a humanoid target. A raster scanning mode is used to scan the foreground and background information line by line from top to bottom to perceive the humanoid target at any position in the scene and obtain full-frame humanoid target image data. The humanoid target image data includes static humanoid target image data and moving humanoid target image data.

[0066] S502, divide the full frame target image data into N blocks in sequence to obtain N block data; where there is no overlap between each block of the N blocks, and N is a natural number greater than 2.

[0067] In some embodiments, the entire frame of target image data can be divided into N equal parts in the image raster scanning direction to obtain N block data. There are no overlapping areas between the N blocks, and N is a natural number greater than 2.

[0068] In some embodiments, the full-frame target image data can be divided into N blocks, patch0, patch1, ..., patchN, at equal intervals along the raster scanning direction. Each block of image includes H rows, and there is no overlapping area between adjacent blocks. Using the raster scanning mode, the image of each block is scanned line by line from top to bottom to obtain a total of N blocks of data, patch0, patch1, ..., patchN-1.

[0069] For example, the full-frame target image is a high-resolution image. The high-resolution image is divided into 11 equal blocks, patch0, patch1, ..., patch10, along the raster scanning direction. Each block of the image includes 5 rows. There is no overlapping area between adjacent blocks. Using the raster scanning mode, the image of each block is scanned line by line from top to bottom to obtain 11 sets of data.

[0070] S503, sequentially and alternately writes two adjacent blocks of data from N blocks of data into the first buffer and the second buffer; at the same time, the target perception backbone sequentially and alternately reads two adjacent blocks of data from the first buffer and the second buffer.

[0071] In some embodiments, two buffers, buffer1 and buffer2, are set up and referred to as the first buffer and the second buffer, respectively. Two adjacent blocks of data from the N blocks are written into the first buffer and the second buffer in a top-down order.

[0072] After writing the i-th block of data to the first buffer, the target perception backbone reads the i-th block of data from the first buffer; i = 0, 1, ..., N. After writing the (i+1)-th block of data to the second buffer, the target perception backbone reads the (i+1)-th block of data from the second buffer.

[0073] After the block data in the first and second caches is read, the next two adjacent blocks of data are written into the first and second caches respectively.

[0074] For example, the "ping-pong buffer" mechanism can be used to write all H rows of data of patch0 to buffer1, and at the same time write all H rows of data of pathc1 to buffer2. After the data in buffer1 and buffer2 are read, all H rows of data of patch2 are written to buffer1 in sequence, all H rows of data of pathc3 are written to buffer2, ..., until all H rows of data of patch(N-1) are written to buffer1 and all H rows of data of pathcN are written to buffer buffer2.

[0075] S504, the target perception backbone network extracts N block feature vectors based on the N block data.

[0076] In some embodiments, the target-aware backbone reads data from patch 0 in buffer 1 to extract regional features and obtains block feature vector 1. After buffer 2 continues to receive row buffer scan data from path c1 online, it reads data from patch 1 to extract regional features and obtains block feature vector 2.

[0077] For example, after all the data in patch 0 has been written, the target perception backbone will read the data from patch 0 in buffer 1 to perform regional feature extraction and obtain block feature vector 1. At the same time, buffer 2 continues to receive row buffer scan data from path c1 online. After all the data in patch 1 has been written, the target perception backbone will read the data from patch 1 in buffer 2 to perform regional feature extraction and obtain block feature vector 2.

[0078] After all the data for patch 2 has been written, the target awareness backbone will read the data from patch 2 in buffer 1 to perform regional feature extraction and obtain block feature vector 3. At the same time, buffer 2 continues to receive row buffer scan data from path c3 online. After all the data for patch 3 has been written, the target awareness backbone will read the data from patch 3 in buffer 2 to perform regional feature extraction and obtain block feature vector 4.

[0079] Similarly, buffer1 and buffer2 extract features from N blocks one by one in a dynamic pipeline manner from top to bottom, resulting in N block feature vectors.

[0080] Since the target in the original high-resolution image has been cut into several incomplete local regions by image slicing, S501 to S504 mainly complete the image slicing and block feature extraction process. By this point, N small-resolution block features have been obtained, but target perception cannot be directly performed on the N small-resolution block features.

[0081] In order to approximate the effect of direct input of high resolution for target perception, S505 is executed.

[0082] S505, the feature vectors of N blocks are fused sequentially to obtain fused features, and post-processing is performed based on the fused features to obtain the category and location information of the target.

[0083] In some embodiments, the block feature vectors of N blocks are fused in a top-down order to obtain fused features, and post-processing is performed based on the fused features to obtain the category and location information of a specific target.

[0084] Feature fusion refers to concatenating N block features sequentially from 0 to N-1 along the H-dimensional plane. For example, two N*C*H*W feature tensors are concatenated along the H-dimensional plane to form one N*C*(2H)*W feature tensor.

[0085] Post-processing mainly includes: coordinate transformation of the network's fused feature map, NMS (non-maximum suppression), threshold filtering, and result output. Coordinate transformation includes sigmoid transformation, calculating the coordinates of the detection box center point, converting the center point coordinates to the coordinates of the top-left and bottom-right corners, and calculating the detection box confidence and class ID.

[0086] The deep learning object detection method provided in Embodiment 1 of this application does not require caching all N blocks of data. Instead, it uses a "ping-pong buffer" mechanism to alternately write and read two adjacent blocks of data in two buffer areas, thus solving the problem of caching raw large-resolution data.

[0087] The deep learning-based object detection method provided in Embodiment 1 of this application only needs to store two adjacent blocks of data in buffer1 and buffer2 for a single inference. Compared with the high-resolution data of the original image, the data cache is only 2 / N of the original, significantly reducing the intermediate data cache. In addition, since the input data resolution of the neural network for a single inference is only 1 / N of the original data, the operating memory of the neural network and the intermediate layer data cache are also only 1 / N of the original resolution. Furthermore, there are no overlapping areas between the blocks, so there is no increase in end-to-end additional memory capacity and computational load.

[0088] Example 2

[0089] Figure 6 This is a schematic diagram of the deep learning-based object detection method provided in Embodiment 2 of this application. Figure 6 As shown, the solution in Example 2 is an optimization based on the solution in Example 1, which aims to improve the accuracy of target perception.

[0090] Given that the scheme in Implementation 1 extracts features independently from each block and then fuses them after abstracting the features of the complete backbone network, the highly abstracted block features have completely lost their spatial correlation. Directly fusing the output feature map of the last convolutional layer is no longer sufficient to represent the original complete target appearance and shape information, resulting in a significant drop in target perception accuracy.

[0091] The deep learning object detection method provided in Embodiment 2 of this application performs image raster segmentation and "ping-pong buffer" mechanism construction in steps S601 to S603, which are the same as S501 to S503 in Embodiment 1, and will not be described again here.

[0092] The following is a detailed introduction to S604 to 606.

[0093] S604, the target perception backbone network includes a first subnet and a second subnet, wherein the first subnet performs downsampling processing on N blocks of data to reduce the resolution of N blocks and extracts feature vectors from N blocks.

[0094] In some embodiments, the target perception backbone is split in the intermediate layer to obtain two subnets, stage1 and stage2. Stage1 can be denoted as the first subnet, and stage2 as the second subnet.

[0095] In some embodiments, the stage1 subnet downsamples the input N block data multiple times to reduce the block resolution and extract N block feature vectors.

[0096] For example, the stage1 subnet can downsample the 11 read blocks 3 times to reduce the block resolution and extract N block feature vectors.

[0097] S605, the feature vectors of N blocks are fused in a top-down order to obtain the fused feature.

[0098] In S606, the second subnet performs convolution and pooling operations at the junction of the fused features to obtain optimized features; the optimized features are then post-processed to obtain the target category and location information.

[0099] For example, the stage2 subnet performs further convolution and pooling operations at the feature splicing points of the fused blocks to obtain the category and location information of the target.

[0100] The above steps S604-606 divide the target perception backbone network into two sub-networks, stage1 and stage2, in the intermediate layer. Stage1 downsamples the input blocks, rapidly reducing the block resolution to maintain perception accuracy while reducing the input buffer and computational load of stage2. Stage2's convolutional operations perform further convolution and pooling operations at the feature concatenation points of the fused blocks, patching the "gaps" between feature blocks to restore spatial correlation and improve perception accuracy. The output features of stage2 are then post-processed to obtain the final target category and location information.

[0101] The deep learning object detection method provided in this application has extremely low data caching requirements. A single neural network inference only needs to store two adjacent blocks of data in buffer1 and buffer2. Compared with the original high-resolution data (a total of N blocks), the data caching is only 2 / N of the original, which greatly reduces the intermediate data caching.

[0102] In addition, since the input data resolution for a single inference of a neural network is only 1 / N of the original data, the operating memory and intermediate layer data cache of the neural network are also only 1 / N of the original resolution, and there are no overlapping areas between the blocks, so there is no increase in end-to-end additional memory capacity and computational load.

[0103] The deep learning-based object detection method provided in this application offers high object perception accuracy. This application proposes a two-stage network raster segmentation and feature fusion architecture, decoupling block feature extraction from feature fusion. The first stage extracts independent block features, while the second stage further abstracts the fused features, effectively recovering the spatial correlation lost during the first stage's independent block feature extraction. This architecture can approximate the object perception accuracy of inference from directly input high-resolution data to the greatest extent possible while significantly reducing data caching.

[0104] It is worth mentioning that the deep learning-based target detection method provided in this application has great potential commercial value in some low / extremely low power edge devices.

[0105] The deep learning-based target detection method provided in this application can achieve a feature extraction and pattern recognition system architecture at the milliwatt or even microwatt level. Compared with existing visual sensor chips, it significantly reduces power consumption and features strong specialization, small size, and ultra-low power consumption. It can be applied to various visual sensing application fields such as ultra-low power home monitoring, home robots, and ubiquitous IoT visual monitoring.

[0106] The deep learning-based object detection method provided in this application is compatible with mainstream raster scanning modes. It is compatible with mainstream raster scanning-based image, audio, and video acquisition systems in the industry, enabling data acquisition, storage, and processing to be performed in a pipelined manner with lower storage capacity, significantly reducing system costs.

[0107] The technical solutions of this application can be extended to any deep learning task, including semantic-level target perception tasks and pixel-level perception tasks.

[0108] Furthermore, the design scheme of this application is not limited to target classification and detection in the field of vision, but can also be flexibly extended to fields related to neural network architecture design, such as audio and video, and text recognition. Deep learning network architectures that meet specific scenario requirements can be developed according to constraints such as power consumption, chip area, storage, and accuracy.

[0109] Figure 7This is a schematic diagram of a deep learning-based target detection apparatus provided in an embodiment of this application. The deep learning-based target detection apparatus provided in this application executes any of the methods described above. The apparatus includes an image scanning module 71, a block partitioning module 72, an alternating buffer module 73, a target perception backbone network 74, a fusion module 75, and a post-processing module 76. The image scanning module 71 scans the entire scene information line by line in sequence to obtain full-frame target image data. The full-frame target image data includes static target image data and moving target image data at any location in the scene. The block division module 72 divides the full-frame target image data into N blocks in sequence to obtain N block data. There is no overlap between each of the N blocks, and N is a natural number greater than 2. The alternating buffer module 73 sequentially and alternately writes two adjacent blocks of data from the N blocks into the first buffer and the second buffer. The target perception backbone network 74 sequentially and alternately reads two adjacent blocks of data from the first buffer and the second buffer. N block feature vectors are extracted based on the N block data. The fusion module 75 sequentially fuses the N block feature vectors to obtain fused features. The post-processing module 76 performs post-processing based on the fused features to obtain the target's category and location information.

[0110] Figure 8 This is a schematic diagram of a deep learning-based object detection system provided as an embodiment of this application. Figure 8 As shown, the device includes: a deep learning-based target detection device 70, a raster scanning device 81, and / or a video acquisition device 82. The video acquisition device 82 acquires full-scene video information, the raster scanning device 81 scans the full-scene information line by line sequentially, and the deep learning-based target detection device 70 obtains full-frame target image data based on the scan information. The full-frame target image data includes static target image data and moving target image data at any location in the scene. The full-frame target image data is sequentially divided into N blocks to obtain N block data. There is no overlap between the N blocks, and N is a natural number greater than 2. Adjacent blocks are sequentially and alternately written into a first buffer and a second buffer. Simultaneously, the target perception backbone sequentially and alternately reads adjacent blocks from the first and second buffers. The target perception backbone extracts N block feature vectors from the N block data. The N block feature vectors are sequentially fused to obtain fused features. Post-processing is performed based on the fused features to obtain the target's category and location information.

[0111] The deep learning-based object detection device 70, raster scanning device 81, and / or video acquisition device 82 can be implemented in software or in hardware. For example, the implementation of the deep learning-based object detection device 70 will be described below.

[0112] As an example of a software functional unit, the deep learning object detection device 70 may include code running on a computing instance. This computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing device may be one or more. For example, the deep learning object detection device 70 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0113] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0114] As an example of a hardware functional unit, the deep learning object detection device 70 may include at least one computing device, such as a server. Alternatively, the deep learning object detection device 70 may also be a device implemented using an ASIC or a PLD. The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0115] The deep learning object detection device 70 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the deep learning object detection device 70 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the deep learning object detection device 70 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0116] This application also provides a computing device 100. For example... Figure 9As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0117] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0118] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0119] Memory 106 may include volatile memory, such as random access memory (RAM). Memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0120] The memory 106 stores executable program code, which the processor 104 executes to implement the functions of the aforementioned image scanning module 71, block partitioning module 72, alternating buffer module 73, target perception backbone network 74, fusion module 75, and post-processing module 76, thereby realizing the deep learning-based target detection method. In other words, the memory 106 stores instructions for executing the deep learning-based target detection method.

[0121] Alternatively, the memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned deep learning object detection apparatus 70, thereby implementing the deep learning object detection method. That is, the memory 106 stores instructions for executing the deep learning object detection method.

[0122] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0123] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0124] like Figure 10 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for performing a deep learning object detection method.

[0125] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing deep learning object detection methods. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing deep learning object detection methods.

[0126] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the deep learning object detection device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules among the image scanning module 71, block partitioning module 72, alternating cache module 73, object perception backbone network 74, fusion module 75, and post-processing module 76.

[0127] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0128] Figure 11 One possible implementation is shown. For example... Figure 11As shown, the two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of the image scanning module 71, the block partitioning module 72, and the alternating cache module 73. Meanwhile, the memory 106 in computing device 100B stores instructions for executing a deep learning object detection method.

[0129] Figure 11 The connection method between the computing device clusters shown can be such that, considering the deep learning target detection method provided in this application requires a large amount of data storage and computation, the functions implemented by the target perception backbone network 74, the fusion module 75 and the post-processing module 76 are delegated to the computing device 100B for execution.

[0130] It should be understood that Figure 11 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.

[0131] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 10 and Figure 11 The connection method of the computing device cluster is different in that the memory 106 of one or more computing devices 100 in the computing device cluster can store the same instructions for executing the deep learning object detection method.

[0132] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing deep learning object detection methods. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing deep learning object detection methods.

[0133] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing parts of the functions of the deep learning object detection system. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more devices in the deep learning object detection apparatus 70.

[0134] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a deep learning-based object detection method.

[0135] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a deep learning-based object detection method.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0137] Furthermore, various aspects or features of the embodiments of this application can be implemented as methods, apparatus, or articles of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used in this application encompasses a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). Additionally, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0138] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0139] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0143] The above are merely specific implementations of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A deep learning-based object detection method, characterized in that, The method, based on images acquired through raster scanning or video capture, includes: The entire scene information is scanned line by line in sequence to obtain full-frame target image data; the full-frame target image data includes static target image data and moving target image data at any location in the scene; The full-frame target image data is divided into N blocks in sequence to obtain N block data; wherein there is no overlapping area between each of the N blocks, and N is a natural number greater than 2; In the aforementioned order, adjacent blocks of data from the N blocks are alternately written into the first and second cache areas; simultaneously... The target perception backbone network sequentially and alternately reads data from two adjacent blocks in the first and second cache areas; The target perception backbone network extracts N block feature vectors based on the N block data; The feature vectors of the N blocks are fused sequentially to obtain the fused feature; Post-processing is performed based on the fused features to obtain the target's category and location information.

2. The method according to claim 1, characterized in that, The full-scene information includes foreground information and background information. The step of sequentially scanning the full-scene information line by line to obtain full-frame target image data includes: The foreground and background information are scanned line by line from top to bottom using a raster scanning mode to obtain the full-frame target image data.

3. The method according to claim 1, characterized in that, The step of dividing the full-frame target image data into N blocks according to the specified order to obtain N block data includes: The full-frame target image data is divided into N blocks at equal intervals from top to bottom along the scanning direction, with no overlapping areas between adjacent blocks. Each of the N blocks includes H rows of target image data.

4. The method according to any one of claims 1-3, characterized in that, The step of sequentially and alternately writing two adjacent blocks of data from the N blocks of data into the first cache and the second cache includes: Write two adjacent blocks of data from the N blocks of data into the first cache and the second cache respectively, in a top-down order. After the block data in the first cache and the second cache are read, the next two adjacent blocks of data are written into the first cache and the second cache respectively.

5. The method according to any one of claims 1-3, characterized in that, The target perception backbone network sequentially and alternately reads two adjacent blocks of data from the first and second buffers, including: After writing the i-th block of data into the first buffer, the target perception backbone reads the i-th block of data from the first buffer. After writing the (i+1)th block of data into the second buffer, the target perception backbone reads the (i+1)th block of data from the second buffer.

6. The method according to any one of claims 1-3, characterized in that, The target perception backbone network includes a first subnetwork, which extracts N block feature vectors based on the N block data, including: The first subnet performs downsampling processing on the N blocks of data to reduce the resolution of the N blocks and extract the feature vectors of the N blocks.

7. The method according to any one of claims 1-3, characterized in that, The step of fusing the feature vectors of the N blocks sequentially to obtain the fused feature includes: The feature vectors of the N blocks are fused in a top-down order to obtain the fused feature.

8. The method according to claim 6, characterized in that, The target perception backbone network includes a second subnetwork. The post-processing based on the fused features to obtain the target's category and location information includes: The second subnet performs convolution and pooling operations at the junction of the fused features to obtain optimized features; The optimized features are post-processed to obtain the target category and location information.

9. The method according to any one of claims 1-3, characterized in that, The target perception backbone network is one of the following network architectures: visual neural network architecture, audio / video neural network architecture, text recognition neural network architecture, and speech recognition neural network architecture.

10. The method according to any one of claims 1-3, characterized in that, The target includes one of the following: semantic-level target, pixel-level target.

11. The method according to any one of claims 1-3, characterized in that, The method is applicable to the following low-power end-side devices: low-power home monitoring, home robots, or IoT visual monitoring devices.

12. An apparatus for object detection using deep learning, for performing the method as described in any one of claims 1-11, characterized in that, The device includes at least: The image scanning module is used to scan the entire scene information line by line in sequence to obtain full-frame target image data; the full-frame target image data includes static target image data and moving target image data at any position in the scene; The block division module is used to divide the full-frame target image data into N blocks in sequence to obtain N block data; wherein there is no overlapping area between each of the N blocks, and N is a natural number greater than 2; An alternating cache module is used to sequentially and alternately write two adjacent blocks of data from the N blocks of data into the first cache area and the second cache area in the order stated above; The target-aware backbone network is used to sequentially and alternately read two adjacent blocks of data in the first and second buffers; and extract N block feature vectors based on the N blocks of data. The fusion module is used to sequentially fuse the feature vectors of the N blocks to obtain a fused feature; The post-processing module is used to perform post-processing based on the fused features to obtain the category and location information of the target.

13. A deep learning-based object detection system, characterized in that, The system includes the apparatus for target detection using deep learning, wherein the apparatus is configured to perform the method as described in any one of claims 1-11 to obtain the category and location information of the target.

14. A computer device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-11.

15. A computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-11.

Images