Improved Densebox target detection method, device and storage medium

Abstract

The present invention discloses an improved Densebox target detection method, device and storage medium, aiming to solve the technical problems of large network parameter redundancy, high computational complexity and insufficient target detection accuracy of existing target detection algorithms. It includes: acquiring historical road image data and performing image preprocessing; performing adaptive perception of the central area based on the preprocessed image to obtain a training sample set of road images; training a pre-constructed CNN network using the training sample set to obtain a trained CNN network model; processing the road image to be detected using the trained CNN network model to obtain a road target detection result. The present invention optimizes the CNN network structure, gives full play to the performance of the network, and uses the adaptive perception central area to effectively balance the back propagation loss weighting of targets of various scales, thereby improving the target detection accuracy.

Description

Technical Field

[0001] The present invention relates to an improved Densebox target detection method, device and storage medium, and belongs to the technical field of ADAS. Background Art

[0002] With the development of urban transportation and the substantial increase in the number of cars, road traffic safety has become the focus of the whole society. At the same time, the safety of cars has also become a global social issue. Countries around the world are using various digital electronic technologies and methods to reduce traffic accidents and casualties as much as possible, while improving the active safety performance of cars. In terms of active safety technology for cars, target detection is the basis of ADAS technology and also a key technology. Through the camera, the data of the road ahead is obtained in real time, and the road conditions around the vehicle are monitored in real time. When danger comes, the driver can be reminded in time or the vehicle can be reasonably intervened directly to reduce the probability of vehicle collision accidents.

[0003] In order to ensure the performance of the existing forward target (vehicles, pedestrians, two-wheeled vehicles) detection algorithms, relatively deep networks are usually used, such as the Densebox network structure. The network parameter redundancy is large, resulting in a heavy load on the mobile chip and failing to meet the real-time effect. At the same time, the deeper the network, the more difficult it is for some small targets in the medium and long distance in the image to be well fed back to the network parameters through the loss function, resulting in a low recall rate of medium and long distance targets.

[0004] In the Densebox framework, a fixed-scale central area is defined for each bounding box. The category of the pixels in the central area must be consistent with the category of the bounding box. If any pixel in the central area is determined to be a certain category of target, the score of the bounding box is updated with the average of the scores of the upper left corner, lower right corner and center point, and the bounding box is saved. The fixed-scale central area will cause the weighted imbalance of targets of various scales to expand the loss function, so that small central areas have lower recall rates for small bounding boxes, while large areas have lower precision relative to large targets. Summary of the invention

[0005] In view of the problems in the prior art of Densebox target detection algorithm such as large network parameter redundancy, high computational complexity and insufficient target detection accuracy, the present invention proposes an improved Densebox target detection method, device and storage medium, which adaptively perceives the central area according to the annotation box in the image to adapt to targets of different scales, improve the recall rate and target detection accuracy of small targets at medium and long distances, and at the same time make modifications based on the original network to improve network utilization.

[0006] In order to solve the above technical problems, the present invention adopts the following technical means:

[0007] In a first aspect, the present invention proposes an improved Densebox target detection method, comprising the following steps:

[0008] Obtain historical road image data and perform image preprocessing;

[0009] Based on the preprocessed images, adaptive perception of the central area is performed to obtain a training sample set of road images;

[0010] Using the training sample set to train a pre-built CNN network to obtain a trained CNN network model, wherein the CNN network is built based on a feature pyramid network technology;

[0011] The trained CNN network model is used to process the road image to be detected to obtain the road target detection results.

[0012] In combination with the first aspect, further, the specific operations of the image preprocessing are as follows:

[0013] Acquire historical road image data, wherein the historical road image data includes a plurality of historical road images and historical target detection results corresponding to the historical road images;

[0014] Performing data annotation on the historical road image, determining the annotation frame of the historical road image, and obtaining the annotated image;

[0015] Perform data augmentation on the annotated image to obtain the preprocessed image.

[0016] In combination with the first aspect, further, the data augmentation processing includes image cropping and image flipping.

[0017] In combination with the first aspect, further, the annotated image has one or more annotated boxes.

[0018] In combination with the first aspect, further, the method for performing adaptive central area perception on each preprocessed image specifically includes the following steps:

[0019] The image coordinate system is established with the lower left corner of the preprocessed image i as the coordinate origin, where i = 1, 2, …, p, and p is the total number of preprocessed images;

[0020] Select the annotation box j in the preprocessed image i, and obtain the coordinates of the upper left corner of the annotation box j based on the image coordinate system (tl x , tl y ) and the lower right corner coordinates (br x ,br y ), where j = 1, 2, ..., q, q is the total number of annotation boxes in the preprocessed image i;

[0021] The coordinates of the center area j are adaptively perceived according to the coordinates of the annotation box j. The specific formula is as follows:

[0022]

[0023] Among them, (ctl x , ctl y ) represents the coordinate of the upper left corner of the center area j, (cbr x ,cbr y ) represents the coordinate of the lower right corner of the center area j. When the width of the annotation box j is greater than 100 pixels, n=5, otherwise n=3.

[0024] In combination with the first aspect, further, the specific operation of the CNN network training is as follows:

[0025] Step 301: Obtain a CNN network with cross-layer connections built based on feature pyramid network technology;

[0026] Step 302: input the training sample set into the input layer of the CNN network, use the CNN network to downsample each image training sample in the training sample set to obtain a downsampled image, and use the downsampled image as the input of the next layer of the CNN network;

[0027] Step 303: In the mth layer of the CNN network, up-sample the down-sampled image of the mth layer, and obtain a feature map of the up-sampled image, denoted as f1, wherein the CNN network has a total of M layers, m = 1, 2, ..., M;

[0028] Step 304, obtaining a feature map of the downsampled image of the m-1th layer in the CNN network, denoted as f2;

[0029] Step 305: Perform channel superposition processing on f1 and f2 to obtain a fused feature map, and use the fused feature map as the feature map of the downsampled image of the mth layer of the CNN network;

[0030] Step 306: repeat steps 303 to 305, so that the feature map of each layer in the CNN network incorporates features of different resolutions and different semantic strengths;

[0031] Step 307: Detect objects of different resolutions using the fused feature map, and output the initial object detection result;

[0032] Step 308: Iteratively optimize the CNN network based on the historical target detection results and the initial target detection results to obtain a trained CNN network model.

[0033] In a second aspect, the present invention proposes an improved Densebox target detection device, comprising:

[0034] An image processing module is used to obtain historical road image data and real-time road images to be detected, and to pre-process the historical road image data;

[0035] An adaptive perception module is used to perform adaptive perception of the central area of ​​the preprocessed image to obtain a training sample set;

[0036] The model training module is used to train the pre-built CNN network using the training sample set to obtain a trained CNN network model;

[0037] The target detection module is used to process the road image to be detected using the trained CNN network model to obtain the road target detection results.

[0038] In combination with the second aspect, further, the CNN network in the model training module is a CNN network with cross-layer connections constructed based on feature pyramid network technology.

[0039] In a third aspect, the present invention proposes an improved Densebox target detection device, comprising a processor and a storage medium;

[0040] The storage medium is used to store instructions;

[0041] The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

[0042] In a fourth aspect, the present invention proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0043] The following advantages can be obtained by using the above technical means:

[0044] The present invention proposes an improved Densebox target detection method, device and storage medium. The present invention integrates feature pyramid network technology (FPN) on the basis of the original Densebox network, optimizes the network structure, gives full play to the performance of the network, and greatly improves the performance of small target detection without substantially increasing the computational complexity of the original model. In addition, the present invention can define a relatively small central area when the scale of the annotation box is large, and define a relatively large central area when the scale of the annotation box is small through the adaptive perception central area of ​​the annotation box. Compared with the central area of ​​fixed scale, the present invention can effectively balance the back propagation loss weighting of targets of various scales, greatly improve the recall of small targets, and at the same time, the large-scale targets are almost unchanged, so that the system can detect farther and smaller targets, and ensure the stable operation of ADAS. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of the steps of an improved Densebox target detection method of the present invention.

[0046] Figure 2 Schematic diagram of a preprocessed image in an embodiment of the present invention.

[0047] Figure 3 Schematic diagram of defining the central area according to the marked box in an embodiment of the present invention; wherein (a) is a schematic diagram of defining the central area according to the marked box when n=3, and (b) is a schematic diagram of defining the central area according to the marked box when n=5.

[0048] Figure 4 Schematic diagram of the central area of ​​a conventional target detection algorithm.

[0049] Figure 5 Schematic diagram of the central area of ​​the method of the present invention.

[0050] Figure 6 Schematic diagram of the network structure of FPN in an embodiment of the present invention.

[0051] Figure 7 4 is a structural diagram of FPN in an embodiment of the present invention.

[0052] Figure 8 It is a structural schematic diagram of an improved Densebox target detection device of the present invention.

[0053] In the figure, 1 is the image processing module, 2 is the adaptive perception module, 3 is the model training module, and 4 is the target detection module. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:

[0055] The present invention proposes an improved Densebox target detection method, such as Figure 1 As shown, the specific steps include:

[0056] Obtain historical road image data and perform image preprocessing;

[0057] Based on the preprocessed images, adaptive perception of the central area is performed to obtain a training sample set of road images;

[0058] Using the training sample set to train a pre-built CNN network to obtain a trained CNN network model, wherein the CNN network is built based on a feature pyramid network technology;

[0059] The trained CNN network model is used to process the road image to be detected to obtain the road target detection results.

[0060] The specific operations of image preprocessing are as follows:

[0061] Step 101: Acquire historical road image data, where the historical road image data includes multiple historical road images and historical target detection results corresponding to the historical road images. The historical road images are generally acquired using vehicle-mounted cameras, and the historical target detection results may be accurate detection results processed manually or by software.

[0062] Step 102: annotate the historical road images, determine an annotation frame for each historical road image, and obtain an annotated image. The number of annotation frames is consistent with the number of targets in the historical road image. Generally, one target corresponds to one annotation frame, so there can be one or more annotation frames in the annotated image.

[0063] Step 103: perform data augmentation processing on the annotated image to obtain a pre-processed image. The data augmentation processing includes operations such as image cropping and image flipping. Figure 2 shown.

[0064] In order to detect targets, it is necessary to first extract the annotation boxes that may contain targets from the image and define a central area for each annotation box. The setting of the central area adds a function to observe the visual pattern of each candidate area, which can determine the correctness of each annotation box. By judging the category confidence of each pixel in the central area of ​​each annotation box and combining appropriate thresholds to determine the existence of the target, the network has the ability to perceive the internal information of the annotation box and can effectively remove the erroneous annotation boxes. However, the fixed-scale central area will affect the removal effect of the erroneous box. If the central area is too small, many small-scale erroneous annotation boxes cannot be removed, and if the central area is too large, many large-scale erroneous annotation boxes cannot be removed, which in turn affects the target detection effect.

[0065] In order to solve the above problems, the present invention adopts a method of adaptively sensing the central area, defining a relatively small central area when the scale of the annotation box is large, and defining a relatively large central area when the scale of the annotation box is small. For multiple pre-processed images, the method of adaptively sensing the central area of ​​each pre-processed image specifically includes the following steps:

[0066] Step 201: Establish an image coordinate system with the lower left corner of the preprocessed image i as the coordinate origin, where i=1, 2, ..., p, and p is the total number of preprocessed images.

[0067] Step 202: Select the annotation box j in the preprocessed image i, and obtain the coordinates of the upper left corner of the annotation box j based on the image coordinate system (tl x , tl y) and the lower right corner coordinates (br x ,br y ),like Figure 3 As shown, j = 1, 2, ..., q, q is the total number of annotation boxes in the preprocessed image i.

[0068] Step 203: In order to determine whether the annotation box j needs to be retained, the coordinates of the corresponding central area j are adaptively sensed according to the coordinates of the annotation box j. The specific formula is as follows:

[0069]

[0070] Among them, (ctl x , ctl y ) represents the coordinate of the upper left corner of the center area j, (cbr x ,cbr y ) represents the coordinate of the lower right corner of the center area j. When the width of the annotation box j is greater than 100 pixels, n = 5. Figure 3 As shown in (b) in , otherwise n = 3, such as Figure 3 As shown in (a) in .

[0071] The above operation is performed for each preprocessed image to obtain a training sample set, which will be used for subsequent neural network training.

[0072] Figure 4 , 5 They are the central area defined by the conventional target detection algorithm and the central area defined by the method of the present invention for the same road image. As can be seen from the figure, the central area of ​​the conventional target detection algorithm is obtained by simply reducing the annotation box by half, and the definition of the central area is not accurate enough. The central area of ​​the method of the present invention can define central areas of different scales for targets of different scales, so that the central area fits the size of the target to be detected. By defining the central area of ​​the annotation box by the method of the present invention, targets of various scales can be effectively adapted, the back propagation loss weighting of targets of various scales can be balanced, and the recall rate of small targets at medium and long distances can be improved while the recall of large targets can be improved.

[0073] After obtaining the training sample set, the training sample set is used as the input of the CNN network, and the historical target detection results are used as the output of the CNN network to train the CNN network. The specific operations are as follows:

[0074] Step 301: Obtain a CNN network with cross-layer connections built based on feature pyramid network technology.

[0075] The traditional Densebox network structure adopts an end-to-end training method, uses a deeper network structure, has more network parameters, and has a high computational complexity. At the same time, it fails to integrate feature maps of different scales well, resulting in laborious calculations on mobile devices and poor results for small targets. The present invention uses FPN to solve the multi-scale problem in target detection. By simply changing the network connection, a feature map with a resolution twice the size of the next resolution is introduced for each feature map of each resolution to perform an element-wise addition operation. The network structure and operation structure of FPN in the embodiment of the present invention are as follows: Figure 6 , 7 shown.

[0076] Step 302: Input the training sample set into the input layer of the CNN network, use the CNN network to downsample each image training sample in the training sample set to obtain a downsampled image, and use the downsampled image as the input of the next layer of the CNN network. For example, if an image training sample is an image with a resolution of 80×80, then in the CNN network, first downsample it to a downsampled image with a resolution of 40×40, and use the downsampled image with a resolution of 40×40 as an input signal to the next layer, and then in the next layer, the downsampled image with a resolution of 40×40 is downsampled to a downsampled image with a resolution of 20×20, and so on, until the final result is output.

[0077] Step 303: Assume that the CNN network has a total of M layers. In the mth layer of the CNN network, upsample the downsampled image of the mth layer to obtain a corresponding upsampled image, and obtain a feature map of the upsampled image, denoted as f1, where m=1, 2, ..., M.

[0078] Step 304: Obtain a feature map of the downsampled image of the m-1th layer in the CNN network, denoted as f2.

[0079] Step 305: Perform channel superposition processing on f1 and f2 to obtain a fused feature map, and use the fused feature map as the feature map of the downsampled image of the mth layer of the CNN network. For example, if the mth layer of the CNN network is a downsampled image with a resolution of 40×40, then f1 is obtained after upsampling the downsampled image, and then the feature map of the downsampled image of the previous layer with a resolution of 80×80 is obtained, that is, f2. Perform channel superposition processing on f1 and f2, and the fused feature map is the feature map of the downsampled image of the mth layer with a resolution of 40×40.

[0080] Step 306: Repeat steps 303 to 305, so that the feature map of each layer in the CNN network integrates features of different resolutions and different semantic strengths.

[0081] Step 307: Use the fused feature map to detect targets of different resolutions. Specifically, the category and precise position of each target in the image are parsed through the feature map, and the initial target detection result is output.

[0082] Step 308: Iteratively optimize the CNN network based on the historical target detection results and the initial target detection results to obtain a trained CNN network model. In the iterative optimization process, an existing optimization algorithm or optimization model can be used to calculate the error between the initial target detection result and the historical target detection result, and then the error is reduced by modifying the CNN network parameters to obtain the optimal CNN network parameters.

[0083] The CNN network connection of the present invention can make the feature maps used for prediction in each layer fuse features of different resolutions and different semantic strengths. The fused feature maps of different resolutions are used for target detection of corresponding resolution sizes, which greatly improves the performance of small object detection without substantially increasing the computational complexity of the original model.

[0084] After completing the model training, the image data of the current road is obtained in real time through the on-board equipment, and the road image to be detected is sent to the trained CNN network model to obtain the corresponding feature map. The category and precise position of each target in the image are parsed through the feature map to obtain the road target detection result.

[0085] The present invention also proposes an improved Densebox target detection device, such as Figure 8 As shown, it includes an image processing module 1, an adaptive perception module 2, a model training module 3 and a target detection module 4, wherein the image processing module is mainly used to obtain historical road image data and real-time road images to be detected by using a camera device, and preprocess the historical road image data, and the image preprocessing includes data annotation and data augmentation and other processing; the adaptive perception module is mainly used to perform central area adaptive perception on the preprocessed image to obtain a training sample set of the road image, wherein the operation of central area perception is consistent with the operation of the method of the present invention; the model training module is mainly used to train a pre-constructed CNN network using the training sample set to obtain a trained CNN network model, wherein the CNN network is a CNN network with cross-layer connections constructed based on feature pyramid network technology; the target detection module is mainly used to process the road image to be detected using the trained CNN network model to obtain a road target detection result.

[0086] The present invention also proposes an improved Densebox target detection device, comprising a processor and a storage medium, wherein the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the improved Densebox target detection method of the present invention.

[0087] The present invention also proposes a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the improved Densebox target detection method of the present invention are implemented.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An improved image target detection method based on Densebox, characterized in that: The steps include: Obtain historical road image data and perform image preprocessing; Based on the preprocessed images, adaptive perception of the central area is performed to obtain a training sample set of road images; Using the training sample set to train a pre-built CNN network to obtain a trained CNN network model, wherein the CNN network is built based on a feature pyramid network technology; Use the trained CNN network model to process the road image to be detected to obtain the road target detection result; The method for adaptively sensing the central area of ​​each preprocessed image specifically includes the following steps: The image coordinate system is established with the lower left corner of the preprocessed image i as the coordinate origin, where: , p is the total number of preprocessed images; Select the annotation box j in the preprocessed image i, and obtain the coordinates of the upper left corner of the annotation box j based on the image coordinate system ( ) and the lower right corner coordinates ( ),in, , q is the total number of annotation boxes in the preprocessed image i; The coordinates of the center area j are adaptively perceived according to the coordinates of the annotation box j. The specific formula is as follows: ； in,( ) represents the coordinate of the upper left corner of the center area j, ( ) represents the coordinate of the lower right corner of the center area j. When the width of the annotation box j is greater than 100 pixels, n=5, otherwise n=3.

2. The image target detection method according to claim 1, characterized in that: The specific operations of the image preprocessing are as follows: Acquire historical road image data, wherein the historical road image data includes a plurality of historical road images and historical target detection results corresponding to the historical road images; Performing data annotation on the historical road image, determining the annotation frame of the historical road image, and obtaining the annotated image; Perform data augmentation on the annotated image to obtain the preprocessed image.

3. The image target detection method according to claim 2, characterized in that: The data augmentation process includes image cropping and image flipping.

4. The image target detection method according to claim 2, characterized in that: The annotated image has one or more annotated boxes.

5. The image target detection method according to claim 1, characterized in that: The specific operation of the CNN network training is as follows: Step 301: Obtain a CNN network with cross-layer connections built based on feature pyramid network technology; Step 302: input the training sample set into the input layer of the CNN network, use the CNN network to downsample each image training sample in the training sample set to obtain a downsampled image, and use the downsampled image as the input of the next layer of the CNN network; Step 303: In the mth layer of the CNN network, upsample the downsampled image of the mth layer, and obtain a feature map of the upsampled image, denoted as f1, where the CNN network has a total of M layers, m=1,2,…,M; Step 304, obtaining a feature map of the downsampled image of the m-1th layer in the CNN network, denoted as f2; Step 305: Perform channel superposition processing on f1 and f2 to obtain a fused feature map, and use the fused feature map as the feature map of the downsampled image of the mth layer of the CNN network; Step 306: repeat steps 303 to 305, so that the feature map of each layer in the CNN network incorporates features of different resolutions and different semantic strengths; Step 307: Detect objects of different resolutions using the fused feature map, and output the initial object detection result; Step 308: Iteratively optimize the CNN network based on the historical target detection results and the initial target detection results to obtain a trained CNN network model.

6. An improved image object detection device based on Densebox, characterized in that: include: An image processing module is used to obtain historical road image data and real-time road images to be detected, and to pre-process the historical road image data; An adaptive perception module is used to perform adaptive perception of the central area of ​​the preprocessed image to obtain a training sample set; The model training module is used to train the pre-built CNN network using the training sample set to obtain a trained CNN network model; The target detection module is used to process the road image to be detected using the trained CNN network model to obtain the road target detection result; The method for adaptively sensing the central area of ​​each preprocessed image specifically includes the following steps: The image coordinate system is established with the lower left corner of the preprocessed image i as the coordinate origin, where: , p is the total number of preprocessed images; Select the annotation box j in the preprocessed image i, and obtain the coordinates of the upper left corner of the annotation box j based on the image coordinate system ( ) and the lower right corner coordinates ( ),in, , q is the total number of annotation boxes in the preprocessed image i; The coordinates of the center area j are adaptively perceived according to the coordinates of the annotation box j. The specific formula is as follows: ； in,( ) represents the coordinate of the upper left corner of the center area j, ( ) represents the coordinate of the lower right corner of the center area j. When the width of the annotation box j is greater than 100 pixels, n=5, otherwise n=3.

7. The image object detection device according to claim 6, characterized in that: The CNN network in the model training module is a CNN network with cross-layer connections built based on feature pyramid network technology.

8. An improved image object detection device based on Densebox, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images