Convolutional neural network-based system and method for detecting target object
By leveraging the depth separability and residual feature map structure of a lightly weighted convolutional neural network, combined with a nonmaximum suppression algorithm, the problem of insufficient computing resources for edge devices is solved, achieving efficient multi-object detection.
Patent Information
- Application Number
- CN202410564879.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-09
- Publication Date
- 2025-11-11
AI Technical Summary
Existing object detection methods based on convolutional neural networks have excessive computational resource requirements for edge devices such as microcontrollers (MCUs), making it difficult to achieve efficient multi-object detection.
We employ a lightweight convolutional neural network, which uses a multi-scale convolutional structure that allows depth-separable convolution of the initial feature map and residual feature maps, combined with a non-maximum suppression algorithm, to reduce computation and storage requirements and improve detection efficiency.
Efficient multi-object detection was achieved on embedded devices, reducing computational resource consumption and improving detection speed and accuracy.
Smart Images
Figure CN120932057A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for detecting target objects based on a lightly weighted convolutional neural network. Background Technology
[0002] Methods based on Convolutional Neural Networks (CNNs) have been used for object detection, especially multi-object detection. YOLO (You Only Look Once), SSD (Single Shot multibox Detector), and RetinaNet are known CNN-based multi-object detection schemes. Most CNN-based schemes require significant computational power and storage, which is unacceptable for edge devices such as microcontrollers (MCUs). Most MCUs do not include accelerators for auxiliary computation. Summary of the Invention
[0003] This summary is provided to illustrate selected simplified portions of the concepts detailed in the following detailed description. This summary is not intended to identify key or essential features of the claims, nor is it intended to limit the scope of the claims.
[0004] According to one embodiment, a method for detecting a target object using a convolutional neural network includes:
[0005] Receive image frames;
[0006] Generate the first feature map of the image frame;
[0007] The residual feature map is generated from the first feature map using the following method:
[0008] A first-scale convolution and a subsequent depthwise separable convolution are applied to the first feature map to produce a second feature map; and
[0009] Add the first feature map and the second feature map to generate an additive feature map;
[0010] At least one extracted feature map of an image frame is generated from the residual feature map by applying a second convolution of a different scale than the first scale to the residual feature map, followed by another depthwise separable convolution; and
[0011] At least one extracted feature map is used to determine a bounding box, wherein the determined box is configured to surround the target object.
[0012] In one or more embodiments, generating a first feature map of an image frame includes applying a second-scale convolution to the image frame and then at least one depthwise separable convolution.
[0013] In one or more embodiments, the method further includes applying a depthwise separable convolution with a scale of 3×3 and a stride of 2 to the additive feature map, followed by a 1×1 convolution with a linear activation layer, to generate a residual feature map.
[0014] In one or more embodiments, applying additional convolution and subsequent depthwise separable convolution to the residual feature map includes:
[0015] The residual feature map is fed into the first and second convolutional branches as input.
[0016] The first branch feature map is generated by using the residual feature map from the first convolutional branch; and
[0017] The second branch feature map is generated by using the residual feature map from the second convolution branch.
[0018] In one or more embodiments, the first convolutional branch includes:
[0019] A convolutional layer with a scale of 3×3 and a stride of 2;
[0020] At least one convolutional module, including a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer with a stride of 1;
[0021] Depth-separable convolutional layers with a scale of 5×5;
[0022] Depth-separable convolutional layers with a scale of 3×3; and
[0023] A convolutional layer with a scale of 1×1 and a stride of 1.
[0024] In one or more embodiments, the first convolutional branch includes two convolutional modules.
[0025] In one or more embodiments, the second convolutional branch includes:
[0026] A convolutional layer with a scale of 3×3 and a stride of 1;
[0027] At least one convolutional module, including a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer with a stride of 1;
[0028] Depth-separable convolutional layers with a scale of 5×5;
[0029] Depth-separable convolutional layers with a scale of 3×3; and
[0030] A convolutional layer with a scale of 1×1 and a stride of 1.
[0031] In one or more embodiments, the second convolutional branch includes two convolutional modules.
[0032] In one or more embodiments, using at least one extracted feature map to determine a bounding box includes:
[0033] Candidate boxes are determined using extracted feature maps, where each candidate box corresponds to a confidence value higher than a detection threshold; and
[0034] The final bounding box is determined from the candidate bounding boxes through nonmaximum suppression.
[0035] In one or more embodiments, the extracted feature map includes a first channel for confidence values, a second channel for the predicted abscissa of the center position, a third channel for the predicted ordinate of the center position, a fourth channel for width prediction, and a fifth channel for height prediction; wherein determining the candidate bounding box includes:
[0036] The confidence value of the first channel of the extracted feature map is determined to be higher than the detection threshold;
[0037] The candidate box is determined to be located at the center of the corresponding x-coordinate of the second channel and the corresponding y-coordinate of the third channel;
[0038] The candidate box is determined to have the corresponding width of the fourth channel and the corresponding height of the fifth channel.
[0039] According to another embodiment, a system based on a convolutional neural network configured to detect a target object includes:
[0040] The preprocessing unit is configured to receive image frames and generate preprocessed image data using the received image frames.
[0041] The detection unit is configured to receive preprocessed image data and generate at least one extracted feature map of an image frame; and
[0042] The post-processing unit is configured to receive at least one extracted feature map and determine a bounding box using at least one extracted feature map, wherein the bounding box is configured to surround a target object; wherein
[0043] The detection unit includes:
[0044] The initial extraction module is configured to generate the first feature map of the image frame;
[0045] The residual module is configured to generate a residual feature map from the first feature map in the following way:
[0046] Applying a first-scale convolution and subsequent depthwise separable convolution to the first feature map to generate a second feature map; and adding the first and second feature maps to generate a summed feature map; and
[0047] The extraction output module is configured to apply a second convolution of a different scale than the first scale and a subsequent additional depthwise separable convolution to the residual feature map to generate an extracted feature map of the image frame.
[0048] In one or more embodiments, the initial extraction module is configured to apply a second-scale convolution to the image frame and then at least one depthwise separable convolution to generate a first feature map of the image frame.
[0049] In one or more embodiments, the residual module is further configured to apply a depthwise separable convolution with a scale of 3×3 and a stride of 2 to the additive feature map, followed by a 1×1 convolution with a linear activation layer, to generate a residual feature map.
[0050] In one or more embodiments,
[0051] The output extraction module includes a first convolutional branch and a second convolutional branch, and therein
[0052] The output extraction module is configured to apply additional convolutions and subsequent depthwise separable convolutions to the residual feature map after the residual module feeds the residual feature map as input into the first and second convolution branches.
[0053] The first convolutional branch is configured to generate a first branch feature map using the residual feature map; and
[0054] The second convolutional branch is configured to generate a second branch feature map using the residual feature map.
[0055] In one or more embodiments, the first convolutional branch includes:
[0056] A convolutional layer with a scale of 3×3 and a stride of 2;
[0057] At least one convolutional module, including a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer with a stride of 1;
[0058] Depth-separable convolutional layers with a scale of 5×5;
[0059] Depth-separable convolutional layers with a scale of 3×3; and
[0060] A convolutional layer with a scale of 1×1 and a stride of 1.
[0061] In one or more embodiments, the first convolutional branch includes two convolutional modules.
[0062] In one or more embodiments, the second convolutional branch includes:
[0063] A convolutional layer with a scale of 3×3 and a stride of 1;
[0064] At least one convolutional module, including a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer with a stride of 1;
[0065] Depth-separable convolutional layers with a scale of 5×5;
[0066] Depth-separable convolutional layers with a scale of 3×3; and
[0067] A convolutional layer with a scale of 1×1 and a stride of 1.
[0068] In one or more embodiments, the second convolutional branch includes two convolutional modules.
[0069] In one or more embodiments,
[0070] The post-processing unit includes a box extraction unit and a box output unit, and therein...
[0071] The bounding box extraction unit is configured to determine candidate bounding boxes using the extracted feature map, where each candidate bounding box corresponds to a confidence value higher than a detection threshold; and
[0072] The output box is configured to determine the final box from the candidate boxes through nonmaximum suppression.
[0073] In one or more embodiments, the extracted feature map includes a first channel for confidence values, a second channel for the predicted abscissa of the center position, a third channel for the predicted ordinate of the center position, a fourth channel for width prediction, and a fifth channel for height prediction; the bounding box extraction unit is configured as follows:
[0074] The confidence value of the first channel of the extracted feature map is determined to be higher than the detection threshold;
[0075] The candidate box is determined to be located at the center of the corresponding x-coordinate of the second channel and the corresponding y-coordinate of the third channel;
[0076] The candidate box is determined to have the corresponding width of the fourth channel and the corresponding height of the fifth channel. Attached Figure Description
[0077] To enable a more concrete understanding of the foregoing contents of this invention, further detailed description of the invention can be obtained by referring to embodiments, some of which are illustrated in the accompanying drawings. The accompanying drawings illustrate only typical embodiments of the invention, and since the invention can have other equally effective embodiments, the accompanying drawings should not be construed as limiting the scope of the invention. The drawings are drawn for ease of understanding rather than measurement of the invention. The benefits of the claimed inventive subject matter will be readily understood by those skilled in the art upon reading this description and in conjunction with the accompanying drawings. In the drawings, similar reference numerals are used to indicate similar elements, and:
[0078] Figure 1 This is a block diagram of a system for detecting objects according to one embodiment;
[0079] Figure 2 yes Figure 1 A block diagram of the detection unit and post-processing unit in the process;
[0080] Figure 3 yes Figure 1 Detailed block diagram of the detection unit and post-processing unit in the process;
[0081] Figure 4 show Figure 2 Operations of the residual module in the code;
[0082] Figure 5 show Figure 2 The operation of the output extraction module in the middle; and
[0083] Figure 6 exhibit Figure 1 Examples of feature maps and bounding boxes generated by the system. Detailed Implementation
[0084] Figure 1 This is a block diagram of a system for detecting objects according to one embodiment. "Detecting objects" here means identifying one or more instances of the occurrence, location, and size of objects of a predetermined kind or category in an image. Without limitation, this category of objects can be, for example, hands or faces. Generally, the invention relates to the detection of one or more instances of objects of the same category; however, the invention is not limited thereto and can be applied to the detection of one or more objects of one category as well as one or more objects of another or several other categories.
[0085] System 100 receives image data from an image capturing device, such as a camera, and uses the received image data to perform object detection. System 100 includes a preprocessing unit 102, a detection unit 104, and a post-processing unit 106. Preprocessing unit 102 receives image data, for example, frame-by-frame image data from a camera. The image data can be in various formats, such as YVYU format, in which the image is represented by Y, U, and V components, where Y is the luminance component and U and V are the chrominance components. For subsequent processing, preprocessing unit 102 converts the image data to RGB888 format, in which the image is represented by R, G, and B components, where R is the red component, G is the green component, and B is the blue component, and each of the R, G, and B components is described by 8 bits of data, collectively referred to as RGB888 format. Preprocessing unit 102 can also perform scaling operations on the received image data to match the input size required by subsequent detection unit 104. For example, the preprocessing unit 102 can scale the image frame to a predetermined size specified by the subsequent detection unit 104. The preprocessing unit 102 provides preprocessed image data to the detection unit 104.
[0086] The detection unit 104 performs object detection using preprocessed image data provided from the preprocessing unit 102. In this embodiment, the detection unit is based on a lightweight convolutional neural network (CNN). Lightweight means that the convolutional neural network used in the detection unit 104 includes a relatively small number of target weights, thus consuming fewer computational resources. Typically, in the system, weights are represented by 8-bit bytes. For example, the system for detecting target objects based on a lightweight CNN according to the present invention can have less than 2 20 A weight count of 1 MB is advantageous for embedded device implementations compared to typical CNN-based systems with hundreds of megabytes of weights. Typically, detection unit 104 performs operations on preprocessed image data to extract features from image frames and generate feature maps of those frames. Those skilled in the art of CNNs will understand that, here, "feature" may not, and generally does not, correspond to physically, visually identifiable concepts such as lines or edges. More precisely, "feature" is a mathematical construct in CNN processing. Based on the resulting feature maps, post-processing unit 106 performs detection of one or more objects of interest. Post-processing unit 106 detects each or any object of interest ("target object") in the feature maps from detection unit 104 using a fixed detection threshold and determines a bounding box to be used around the detected target object. In this context, "bounding box" generally refers to the outline of a shape, typically a rectangle. However, depending on the context, "bounding box" can also refer to a solid shape defined by and within an outline. More specifically, post-processing unit 106 determines the bounding box by determining its size and position based on the feature maps from detection unit 104. The defined bounding box is positioned and surrounds the target object in the image.
[0087] according to Figure 1 The system 100 further includes a display unit 108. The display unit 108 receives data of the determined frame, such as the center position (cx) of the frame corresponding to the detected object. i,j ,cy i,j ), and the width and height (w) of the box corresponding to the target object of interest. i,j ,h i,j The display unit 108 composites the frame with the image by placing the frame at a position corresponding to the target object and making the frame surround the target object, and displays the outline of the determined frame around the object detected in the image.
[0088] Figure 2This is a block diagram of a detection unit 104 according to one embodiment. As described above, the detection unit 104 is based on a lightly weighted convolutional neural network structure. The detection unit 104 includes an initial extraction module 202, a residual module 204, and an extraction output module 206. The initial extraction module 202 receives preprocessed image data from the preprocessing unit 102 and processes it to obtain initial features of the input image frame. Typically, the initial extraction module 202 is based on convolution kernels. The residual module 204 further processes the initial features of the input frame to narrow the features and provide weighted features of the input frame. "Weighted" means that the features are generated by the convolution of the residual module 204, where weights are applied in the convolution. Figure 2 As can be seen, the residual module 204 is largely convolutional and has a residual structure, which will be detailed below. The extraction output module 206 generates an extracted feature map of the input image frame based on the weighted features from the residual module 204. Figure 2 In the example shown, the output extraction module 206 includes a first branch and a second branch, which will be described in detail below.
[0089] Figure 3 yes Figure 1 and Figure 2 A detailed framework diagram of the detection unit 104 and post-processing unit 106 is shown below. The initial extraction module 202, also known as the initial layer for detecting the convolutional neural network structure, uses a 3×3 convolutional kernel 222 with a stride of 2 to process the preprocessed image data from the preprocessing unit 102. The 3×3 convolutional kernel 222 with a stride of 2 quickly narrows the feature map of the image frame. Compared to other configurations, such as smaller-sized or smaller-stride convolutional kernels, this is beneficial for reducing the number of computations required and reducing memory consumption, such as RAM consumption, in the early stages of processing. The initial extraction module 202 then includes a first depth-wise separable convolutional layer 224 and a second depth-wise separable convolutional layer 226, each with a stride of 1, to acquire the initial features of the input frame and produce a first feature map as output.
[0090] Now for reference Figure 4 It is a flowchart of the operation performed by the residual module 204 according to one embodiment. Figure 4 The process will refer to Figure 2 , Figure 3The input 402 to the residual module is typically the output of the second depthwise separable convolutional layer 226. An example of the residual module 204 is constructed using a residual structure comprising a first convolutional layer 242, a third depthwise separable convolutional layer 244, and an adder 246, followed by a fourth depthwise separable convolutional layer 248 and a second convolutional layer 250. According to an embodiment, the first convolutional layer 242 is a 1×1 convolutional kernel, and the third depthwise separable convolutional layer 244 is a separable convolutional layer. The first convolutional layer 242 and the third depthwise separable convolutional layer 244 together form a first convolutional branch, which filters features from a first feature map of the image frame provided by the initial extraction module 202 and provides a second feature map as input to the adder 246. The adder 246 receives the second feature map from the third depthwise separable convolutional layer 244 and directly receives the first feature map from the second depthwise separable convolutional layer 226 of the initial extraction module 202. In other words, adder 246 receives the output of a first branch comprising a first convolutional layer 242 and a third depthwise separable convolutional layer 244, as well as the input to that first convolutional branch, and adds the first feature map and the second feature map to produce a summed feature map. As described above, it is understood that the feature map is represented by an array of values derived from the convolution operation of the image frame. "Summing" the feature maps means producing a summed feature map with an array of values, where the values are the sum of the values at corresponding positions in the first and second feature maps. Connecting the input and output of the first branch using adder 246 helps to preserve useful information from the image frame that might be lost in the first convolutional branch.
[0091] Because of the connection of adder 246, the additive feature map produced by adder 246 maintains the same resolution as the feature map from the second depthwise separable convolutional layer 226 of the initial extraction module 202. The additive feature map output by adder 246 is further narrowed by the fourth depthwise separable convolutional layer 248 and the second convolutional layer 250. Figure 4The operation of the fourth depthwise separable convolutional layer 248 and the second convolutional layer 250 is illustrated. The fourth depthwise separable convolutional layer 248 performs a 3×3 depthwise convolution on the input summed feature map with a stride of 2. The second convolutional layer 250 further performs a 1×1 convolution on the output of the fourth depthwise separable convolutional layer 248. Those skilled in the art will understand that the present invention is not limited to any particular type of convolutional layer. Without limitation, the first convolutional layer 242 and the third depthwise separable convolutional layer 244, which can be used according to embodiments of the present invention, include a rectified linear unit (ReLU) as the activation layer after convolution. ReLU is a typical nonlinear function that simulates nonlinear behavior in nature, such as the generally dispersed and sparse information encoding of biological neurons. The fourth depthwise separable convolutional kernel 248 also includes ReLU as the activation function. An example of the second convolutional kernel 250 includes a linear activation function.
[0092] The residual feature map generated by the residual module 204 is fed into the extraction output module 206 as its input 501 to produce the final output feature map. In one example, the extraction output module 206 is constructed as a multi-scale structure, which includes at least one output branch, and as shown... Figure 2 Examples and references Figure 5 As shown, the first output branch 502 and the second output branch 504 are similar to each other, except that they have different strides. "Multi-scale" here means that the kernel and stride of the convolution have different sizes. An example of the output extraction module 206 may include more branches to facilitate the detection of objects of more sizes.
[0093] Taking the first output branch 502 as an example, it includes a 3×3 convolutional kernel 262 with a stride of 2, and a multi-scale kernel 264. A portion of the multi-scale kernel 264 includes a convolutional layer similar to the first convolutional branch of the residual module 204. The convolutional kernels of the other output branches of the extraction output module 206 can have different strides, for example... Figure 2 and Figure 5 The second branch 504 shown has a convolution kernel 266 with a stride of 1. It can be understood that a convolution kernel with a stride of 2 is beneficial for filtering features of larger objects, while a convolution kernel with a stride of 1 is beneficial for filtering features of smaller objects.
[0094] Figure 5 It shows Figure 2 , Figure 3 The flowchart shows the processing of the output extraction module 206. The output of the convolution kernel 262 is provided to module 268 of the multi-scale kernel 264. As described, module 268 is similar to... Figure 2 and Figure 3The first convolutional branch of the residual module 204 shown includes a convolutional kernel and a depthwise convolutional kernel. The multi-scale kernel 264 includes at least one module 268. An example of the multi-scale kernel 264 includes two sequentially connected modules 268 (in...). Figure 5 Each branch is shown as "module × 2"). The output of at least one module 268 is provided to subsequent layers, which sequentially include a 5×5 convolutional layer 506, a 3×3 convolutional layer 508, and a 1×1 convolutional layer 510 with a stride of 1. Similarly, the multi-scale kernel 270 of the second output branch 504 includes at least one module 512. An example of the multi-scale kernel 270 includes two sequentially connected modules 512 (in...). Figure 5 Each branch is shown as "Module × 2". The output of at least one module 512 is provided to subsequent layers, which sequentially include a 5×5 convolutional layer 514, a 3×3 convolutional layer 516, and a 1×1 convolutional layer 518 with a stride of 1. Multiple examples of branches that extract the output modules have different sizes or different scales, which is beneficial for more accurate object positioning.
[0095] The outputs of branches 502 and 504 of the output extraction module 206 are provided to the post-processing unit 106. (Reference) Figure 3 An example of post-processing unit 106 includes a box extraction unit 282 and a box output unit 284. Box extraction unit 282 receives feature maps from the extraction output module 206 of self-detection unit 104 and determines candidate boxes based on the feature maps obtained by convolving the original image frames. Box output unit 284 determines the final boxes from the candidate boxes and outputs the determined final boxes.
[0096] Figure 6 This illustrates an example of the process from the initial image to the final defined bounding box. Figure 1 Taking system 100 as an example, image frame 602 from preprocessing unit 102 includes target object 604, which is the object of interest of the system. Image frame 602 is processed by convolutional network 606. Figures 1 to 3 The detection unit 104 in the example can be a convolutional network 606. For example... Figure 6 As shown, each of the feature maps 608 and 610 output by the corresponding branches is represented by 5 channels: center positions cx 612 and cy 614, width w 616, height h 618, and confidence value 620. Because the convolutional kernels of each branch have different scales, the resolution of the feature maps obtained from each branch also differs. Figure 6 As shown, the feature resolution of branch 608 is twice that of branch 610.
[0097] Based on the feature maps 608 and 610 of each branch, the box extraction unit 282 determines candidate boxes 622 through fixed threshold detection. As an example, specifically, the "candidate box" is determined based on a confidence value 620 in the confidence value channel, and more specifically, based on whether the confidence value 620 is higher than the detection threshold. In some implementations, the detection threshold is fixed and preset. In other implementations, the threshold is set based on the application used by the detection system.
[0098] As described above, candidate boxes 622 have location coordinates and dimensions. When the confidence value 620 is higher than a threshold, the associated boxes (defined by the other four channels based on their center coordinates, height, and width) are determined and provided as "candidate boxes." Further, the box output unit 284 filters candidate boxes 622 by selecting a filter, such as Non-Maximum Suppression (NMS), and determines the final box 624 for the detected object. NMS is used to remove redundant boxes and retain the most relevant ones. Non-limitingly, in the illustrated embodiment, NMS is typically implemented iteratively by: selecting the box with the highest confidence score as the seed box; calculating the Intersection over Union (IoU) value between the seed box and other boxes; discarding boxes with IoU values lower than the NMS threshold and replacing the seed box with the box with the highest IoU value. The IoU is the ratio of the intersection area to the union area of the boxes. An example value for the NMS threshold is 0.5.
[0099] In one example, the detection unit 104 includes multiple depthwise separable convolutional kernels, such as a first depthwise separable convolutional layer 224, a second depthwise separable convolutional layer 226, a third depthwise separable convolutional layer 244, a fourth depthwise separable convolutional layer 248, and a multi-scale kernel 264, comprising 5×5 and 3×3 depthwise convolutions. This produces fewer weights compared to standard convolutions that typically use weight arrays, thus reducing the consumption of ROM (Read-Only Memory) or RAM (Random-Access Memory) in the detection unit 104. For reference, a conventional 8-input-16-output channel convolution has a weight size of 16×8×3×3=1152, while a depthwise convolution with the same number of input and output channels produces only 8×3×3+1×1×8×16=200 weights. Reducing ROM or RAM consumption is particularly beneficial for embedded devices, enabling efficient multi-object detection on embedded devices with lower computational resource consumption.
[0100] During the training of detection unit 104, the weights are governed by the loss function: Where SH and SW correspond to the height and width of the output feature map, Whether the represented object appears in the corresponding cell. This indicates that the object does not appear in the corresponding cell, (cx) i,j ,cy i,j (w) is the truth value of the center position of the object. i,j ,h i,j ) is the truth value for the width and height of the object, conf i,j The confidence value that represents the object's position in the corresponding cell. This represents the predicted value of the feature map in the network. Using this loss function, the accuracy of object localization is improved. This loss function can also provide classification of object regions and non-object regions.
[0101] As mentioned above, ReLU layers following convolution produce more representative characteristics. ReLU-type layers have an advantage over other activation layer types, such as PReLU layers, due to their lower 8-bit quantization loss. As is known, ReLU activation layers do not suffer quantization loss when the activation value is negative.
[0102] Thus, one aspect of the present invention includes receiving an image frame 602; generating a first feature map 608 of 202 image frames; generating a 204 residual feature map from the first feature map by: applying a convolution 242 of a first scale to the first feature map and a subsequent depthwise separable convolution 244 to generate a second feature map; adding the first feature map and the second feature map 246 to generate a summed feature map; generating at least one extracted feature map of 206 image frames from the residual feature map by applying an additional convolution 262 of a second scale different from the first scale to the residual feature map and a subsequent additional depthwise separable convolution 264; and determining 106 boxes using at least one extracted feature map, wherein the determined boxes 624 are configured to surround a target object 604.
[0103] Another aspect of the invention includes a system based on a convolutional neural network, configured to detect a target object. The system includes a preprocessing unit 102 configured to receive an image frame 602 and generate preprocessed image data using the received image frame; a detection unit 104 configured to receive the preprocessed image data and generate at least one extracted feature map of the image frame; and a post-processing unit 106 configured to receive at least one extracted feature map and determine a bounding box 624 using the at least one extracted feature map. The determined bounding box 624 is configured to surround the target object 604. The detection unit 104 includes: an initial providing module 202 configured to generate a first feature map of an image frame; a residual module 204 configured to generate a residual feature map from the first feature map by: applying a first-scale convolution 242 and a subsequent depthwise separable convolution 244 to the first feature map to generate a second feature map, and adding the first feature map and the second feature map 246 to generate a summed feature map; and an extraction output module 206 configured to apply a second-scale convolution 262 different from the first scale and a subsequent further depthwise separable convolution 264 to the residual feature map to generate an extracted feature map of the image frame.
[0104] It can now be understood that the system examples include ultra-lightweight neural networks with residual structures for multi-object detection, thereby reducing memory and computation usage. This makes them highly suitable for embedded device applications, such as microcontroller units (MCUs), and offers improved performance at 1-GHz. The Cortex-M7 microcontroller can detect up to 5 frames per second.
[0105] Implementations of various embodiments have been described herein with reference to specific examples shown. These examples were chosen to assist those skilled in the art in forming a clear understanding of and implementing the embodiments. However, the scope of systems, structures, and devices that can be constructed to include one or more embodiments, and the scope of methods implemented according to one or more embodiments, are not limited to the exemplary examples shown. Rather, those skilled in the art will understand based on this specification that many other configurations, structures, and methods can be implemented according to the various embodiments.
[0106] It should be understood that, with regard to the various positional indications used in the foregoing description of the invention, such as top, bottom, upper, and lower, these indications are given only with reference to the corresponding drawings, and may instead have other positional relationships when the orientation of the device changes during manufacturing or operation. As stated above, those positional relationships are described for clarity only and are not intended to be limiting.
[0107] The foregoing description in this specification refers to specific embodiments and accompanying drawings, but the invention should not be limited thereto; rather, it should be defined by the claims. The described drawings are exemplary and not restrictive. In the drawings, for illustrative purposes, the dimensions of elements may be enlarged and may not be drawn to a specific scale. This specification should also include variations in the tolerances and properties of the elements, operating methods, and other aspects. Various weakened embodiments of the invention should also be included.
[0108] The term "comprising" as used in this description and claims does not exclude other elements or steps. Unless specifically indicated, when using singular forms such as "a" or "an" to refer to a definite or indefinite element, the plural of that element should be included. Therefore, the term "comprising" should not be construed as limited to the items listed thereafter, nor should it be construed as excluding other elements or steps; the scope of the description "the device comprises items A and B" should not be limited to a device that includes only elements A and B. This description indicates that, for the purposes of this description, only elements A and B of the device are relevant. Although coupling generally includes inductive connections, and connection generally means a connection via, for example, wires, the terms "connection," "coupled," and "coupled" used herein all indicate an electrical link between coupled or connected elements and do not imply the absence of intermediate elements.
[0109] For those skilled in the art, various specific modifications can be made without departing from the scope of the claims of this invention.
Claims
1. A method for detecting target objects using a convolutional neural network, characterized in that, include: Receive image frames; Generate the first feature map of the image frame; The residual feature map is generated from the first feature map using the following method: A first-scale convolution and subsequent depthwise separable convolution are applied to the first feature map. To generate a second feature map; as well as Add the first feature map and the second feature map to generate an additive feature map; At least one extracted feature map of an image frame is generated from the residual feature map by applying an additional convolution of a second scale different from the first scale to the residual feature map, followed by an additional depthwise separable convolution. as well as At least one extracted feature map is used to determine a bounding box, wherein the determined box is configured to surround the target object.
2. The method according to claim 1, characterized in that, Applying additional convolutions to the residual feature map, followed by further depthwise separable convolutions, includes: The residual feature map is fed into the first and second convolutional branches as input; The first branch feature map is generated by using the residual feature map from the first convolutional branch; and The second branch feature map is generated by using the residual feature map from the second convolution branch.
3. The method according to claim 2, characterized in that, The first convolutional branch includes: A convolutional layer with a scale of 3×3 and a stride of 2; At least one convolutional module, including a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer with a stride of 1; Depth-separable convolutional layers with a scale of 5×5; Depth-separable convolutional layers with a scale of 3×3; and A convolutional layer with a scale of 1×1 and a stride of 1.
4. The method according to claim 1, characterized in that, Using at least one extracted feature map to determine the bounding box includes: Candidate boxes are determined using extracted feature maps, where each candidate box corresponds to a confidence value higher than a detection threshold; and The final bounding box is determined from the candidate bounding boxes through nonmaximum suppression.
5. The method according to claim 4, characterized in that, The extracted feature map includes the first channel of confidence value, the second channel of predicted horizontal coordinate of center position, the third channel of predicted vertical coordinate of center position, the fourth channel of predicted width, and the fifth channel of predicted height. The process of determining candidate boxes includes: The confidence value of the first channel of the extracted feature map is determined to be higher than the detection threshold; The candidate box is determined to be located at the center of the corresponding x-coordinate of the second channel and the corresponding y-coordinate of the third channel; The candidate box is determined to have the corresponding width of the fourth channel and the corresponding height of the fifth channel.
6. A system based on a convolutional neural network, configured to detect a target object, characterized in that, The system includes: The preprocessing unit is configured to receive image frames and generate preprocessed image data using the received image frames. The detection unit is configured to receive preprocessed image data and generate at least one extracted feature map of an image frame; and The post-processing unit is configured to receive at least one extracted feature map and determine a bounding box using at least one extracted feature map, wherein the bounding box is configured to surround a target object; wherein The detection unit includes: The initial extraction module is configured to generate the first feature map of the image frame; The residual module is configured to generate a residual feature map from the first feature map in the following way: Applying a first-scale convolution and subsequent depthwise separable convolution to the first feature map to generate a second feature map; and adding the first and second feature maps to generate a summed feature map; and The extraction output module is configured to apply a second convolution of a different scale than the first scale and a subsequent additional depthwise separable convolution to the residual feature map to generate an extracted feature map of the image frame.
7. The system according to claim 6, characterized in that: The residual module is further configured to apply a depthwise separable convolution with a scale of 3×3 and a stride of 2 to the additive feature map, followed by a 1×1 convolution with a linear activation layer, to generate the residual feature map.
8. The system according to claim 6, characterized in that: The output extraction module includes a first convolutional branch and a second convolutional branch, and therein The output extraction module is configured to apply additional convolutions and subsequent depthwise separable convolutions to the residual feature map after the residual module feeds the residual feature map as input into the first and second convolution branches. The first convolutional branch is configured to generate the first branch feature map using the residual feature map; as well as The second convolutional branch is configured to generate a second branch feature map using the residual feature map.
9. The system according to claim 8, characterized in that, The first convolutional branch includes: A convolutional layer with a scale of 3×3 and a stride of 2; At least one convolutional module, including a 1×1 convolutional layer and a 3×3 depthwise separable convolutional layer with a stride of 1; Depth-separable convolutional layers with a scale of 5×5; Depth-separable convolutional layers with a scale of 3×3; and A convolutional layer with a scale of 1×1 and a stride of 1.
10. The system according to claim 6, characterized in that: The post-processing unit includes a box extraction unit and a box output unit, and therein... The box extraction unit is configured to use the extracted feature map to determine candidate boxes, where each candidate box corresponds to a confidence value higher than the detection threshold; as well as The output box is configured to determine the final box from the candidate boxes through nonmaximum suppression.