A pedestrian detection method, device, and storage medium based on multiple models

Abstract

The present invention relates to the field of image recognition technology, and specifically provides a multi-model based pedestrian detection method, device and storage medium, the method comprising the following steps: acquiring pedestrian images in different scenes; performing pedestrian detection annotation on the pedestrian images based on full-image annotation, and performing pedestrian attribute annotation on the pedestrian images based on cutout annotation; acquiring a first detection result based on a first model and the pedestrian detection annotation, acquiring a second detection result based on a second model and the pedestrian attribute annotation, and acquiring a pedestrian classification result based on the first detection result, the second detection result and the third model. This application uses two models in two stages to perform precise positioning of pedestrian detection, and selects a lightweight network such as Vargnet as a training model, which is suitable for installation on a vehicle-mounted terminal GPU with small memory and complex environment, and can complete pedestrian detection well and quickly.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a pedestrian detection method, device and storage medium based on multiple models. Background Art

[0002] Pedestrian detection is an important branch of object detection in computer vision. It involves identifying specific types of pedestrians, typically on urban roads. Pedestrian detection is a benchmark problem in object detection and has great practical value in scenarios such as autonomous driving, intelligent transportation, passenger counting, and surveillance.

[0003] Chinese patent publication number CN113343785A discloses a pedestrian detection method based on lightweight YOLOv3. This invention discloses a pedestrian detection method based on lightweight YOLOv3, including: establishing a pedestrian dataset for perimeter intrusion protection applications; constructing a lightweight YOLOv3 pedestrian detection network; dividing the pedestrian detection training set, training the lightweight YOLOv3 pedestrian detection network to obtain a lightweight pedestrian detection model; dividing the pedestrian detection validation set, verifying the effectiveness of the trained lightweight pedestrian detection model; and deploying the lightweight YOLOv3 pedestrian detection model into an embedded front-end device. This method replaces the backbone network of the traditional YOLOv3 detection network with a high-precision lightweight backbone network, significantly reducing the forward computation load and the amount of parameter data required by the pedestrian detection network. This method significantly improves the detection speed while maintaining pedestrian detection accuracy. This method is suitable for embedded devices with low computing power and limited storage space. However, this solution lacks detailed pedestrian detection and attribute classification, and cannot fully reflect the attributes of pedestrians in actual applications. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-model based pedestrian detection method, device and storage medium to solve the problem that the existing detection methods are not detailed enough for pedestrian detection and attribute classification.

[0005] In order to achieve the above technical objectives, the technical solutions adopted by the present invention are as follows:

[0006] In a first aspect, the present application provides a pedestrian detection method based on multiple models, comprising the following steps:

[0007] S1, obtain pedestrian images in different scenes;

[0008] S2, performing pedestrian detection annotation on the pedestrian image based on the full-image annotation, and performing pedestrian attribute annotation on the pedestrian image based on the cutout annotation;

[0009] S3, obtaining a first detection result based on the first model and the pedestrian detection annotation, obtaining a second detection result based on the second model and the pedestrian attribute annotation, and obtaining a pedestrian classification result based on the first detection result, the second detection result and the third model.

[0010] In combination with the first aspect, in some optional implementations, the full-image annotation includes: annotating the pedestrian image once based on a rectangular frame, wherein the annotated categories include pedestrians and cyclists.

[0011] In combination with the first aspect, in some optional embodiments, the cutout annotation includes: based on the full-image annotation result, extracting and secondary annotation of the area where the pedestrians and cyclists are located on the pedestrian image, and the content of the secondary annotation is the attributes of the pedestrians and cyclists, wherein the attributes of the pedestrians and cyclists include head, age, direction and posture.

[0012] In combination with the first aspect, in some optional implementations, the first model, the second model, and the third model use a Vargnet model as a backbone.

[0013] In combination with the first aspect, in some optional implementations, the number of UnitA and UnitB related blocks and channels of the backbone in the first model is greater than the number of UnitA and UnitB related blocks and channels of the backbone in the second model.

[0014] In combination with the first aspect, in some optional implementations, the third model is generated by adding a classification head and a detection head on the basis of the second model.

[0015] In combination with the first aspect, in some optional implementations, the method further includes: calculating the error between the prediction of the first model and the actual sample based on a DIOU loss algorithm.

[0016] In a second aspect, the present application provides a multi-model-based pedestrian detection device, comprising:

[0017] An image acquisition unit, which acquires pedestrian images in different scenes;

[0018] a labeling unit, performing pedestrian detection labeling on the pedestrian image based on full-image labeling, and performing pedestrian attribute labeling on the pedestrian image based on cutout labeling;

[0019] A processing unit obtains a first detection result based on the first model and the pedestrian detection annotation, obtains a second detection result based on the second model and the pedestrian attribute annotation, and obtains a pedestrian classification result based on the first detection result, the second detection result and the third model.

[0020] In conjunction with the second aspect, in some optional embodiments, the apparatus further includes:

[0021] The calculation unit calculates the error between the prediction of the first model and the real sample based on the DIOU loss algorithm.

[0022] In a third aspect, the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is run on a computer, the above method can be executed.

[0023] The invention adopting the above technical solution has the following advantages:

[0024] 1. This application uses two models and two stages to accurately locate pedestrians and achieve accurate target detection in intelligent driving.

[0025] 2. Vargnet, a lightweight network, is chosen as the training model. It is suitable for installation on the GPU of an in-vehicle terminal with small memory and complex environment, and can complete pedestrian detection well and quickly.

[0026] 3. This application provides a model for classifying human attributes to assist in subsequent vehicle control. It refines pedestrian attributes in pedestrian detection methods to facilitate the intelligent selection of subsequent response measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention can be further illustrated by the non-limiting examples given in the accompanying drawings;

[0028] Figure 1 Schematic diagram of a multi-model-based pedestrian detection method in an embodiment of the present application;

[0029] Figure 2 Schematic diagram of UnitA and UnitB in the full-image pedestrian detection model in an embodiment of the present application;

[0030] Figure 3 This is a Backbone structure diagram of the full-image pedestrian detection model in the embodiment of this application;

[0031] Figure 4 This is a Backbone structure diagram in the pedestrian re-detection model in the embodiment of the present application;

[0032] Figure 5 Flowchart of a multi-model-based pedestrian detection method in an embodiment of the present application;

[0033] Figure 6 Schematic diagram of a multi-model-based pedestrian detection device in an embodiment of the present application.

[0034] The main component symbols are described as follows:

[0035] 10: Image acquisition unit; 20: Annotation unit; 30: Processing unit; 31: Calculation unit. DETAILED DESCRIPTION

[0036] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that in the drawings or descriptions, similar or identical parts are numbered the same. Implementations not shown or described in the drawings are forms known to those of ordinary skill in the art. In addition, directional terms mentioned in the embodiments, such as "upper," "lower," "top," "bottom," "left," "right," "front," and "back," are merely references to the directions in the drawings and are not intended to limit the scope of protection of the present invention.

[0037] like Figure 1 As shown, the embodiment of the present application provides a pedestrian detection method based on multiple models, including the following steps:

[0038] S1, obtain pedestrian images;

[0039] S2, performing pedestrian detection annotation on the pedestrian image based on the full-image annotation, and performing pedestrian attribute annotation on the pedestrian image based on the cutout annotation;

[0040] S3, obtaining a first detection result based on the first model and the pedestrian detection annotation, obtaining a second detection result based on the second model and the pedestrian attribute annotation, and obtaining a pedestrian classification result based on the first detection result, the second detection result and the third model.

[0041] In step S1, pedestrian images in different scenes are obtained. Figure 1 and attached Figure 5 , using the vehicle's forward-looking camera to collect data, collecting pedestrian data in different scenarios such as daytime, nighttime, and rainy days. Online pedestrian detection datasets cannot meet the requirements, so this application uses the vehicle's forward-looking camera to re-collect data and designs annotation documents according to the requirements. Preferably, other means can also be used to collect pedestrian data near the vehicle, such as infrared imagers, radars, etc.

[0042] In step S2, pedestrian detection annotation is performed on the pedestrian image based on full-image annotation, and pedestrian attribute annotation is performed on the pedestrian image based on cutout annotation. Full-image annotation: In the pedestrian detection and annotation process, this application directly uses a rectangular frame to perform an annotation on the collected pedestrian image. The annotated categories are

[0043] 1. Pedestrians

[0044] 2. Cyclist

[0045] Pedestrians and cyclists are post-processed differently in the intelligent driving system. This patent requires detecting them separately and excluding cyclists from pedestrians. Therefore, this patent marks pedestrians and cyclists separately, deducts pedestrians and cyclists through the full-image detection model, and sends them to the pedestrian re-detection model for secondary precise positioning and exclusion of cyclists from pedestrians.

[0046] Cutout annotation: After the detection and annotation are completed, this patent will extract the areas where pedestrians and cyclists are located based on the annotation results, perform secondary annotation, and mark the various attributes of the pedestrians.

[0047] Secondary labeling task:

[0048] 1. Head labeling

[0049] Category: Human head

[0050] 2. Age marking

[0051] Category: Elderly, Youth, Children

[0052] 3. Orientation marking

[0053] Category: front, back, left, right, left front, right front, left back, right back

[0054] 4. Human Pose Annotation

[0055] Category: walking, squatting, bending over, cycling

[0056] This application designs a document with multiple attribute annotations for pedestrians. Compared with other patents, it outputs more attributes of pedestrians to assist intelligent driving.

[0057] In step S3, a first detection result is obtained based on the first model and the pedestrian detection annotation, a second detection result is obtained based on the second model and the pedestrian attribute annotation, and a pedestrian classification result is obtained based on the first detection result, the second detection result, and the third model. Among them, the first detection result is the pedestrian detection result; the second detection result is the pedestrian re-detection result. First, the first model, the second model, and the third model are established. Secondly, the corresponding training is completed based on the established models. The first model is a full-image pedestrian detection model; the second model is a pedestrian re-detection model; and the third model is a pedestrian classification model. The design steps of each model are as follows:

[0058] Full-image pedestrian detection model: This patent uses a VAR-based model as its backbone. The VAR-based network aims to balance computational intensity and make the network computation graph easier to optimize. Therefore, a variable group convolution (Variable Group CNN) based on depthwise separable convolution is proposed. In variable group convolution, the number of input channels for each group is fixed, and the number of groups can be adjusted as a hyperparameter. This increases network capacity, allowing for a smaller channel number, which helps reduce the time consumption of off-chip communication.

[0059] Vargnet creates two of the most commonly used structures of its network based on variable group convolution and point convolution: UnitA and UnitB. The structure is as follows Figure 2 shown.

[0060] This patent uses vargnet as the backbone for 32x downsampling. The backbone is a network used to extract image features, and then the neck is upsampled, returning the downsampled feature map to 1 / 4 the size of the original image. The head network is connected to the head network after the upsampling output of each neck layer. The head network consists of a UnitA and a separable group convolution. The four head results output by the neck are combined to output the pedestrian detection position and category, that is, the first detection result. The backbone structure diagram is shown in the figure below. Figure 3 shown.

[0061] The full-image pedestrian detection model also includes an error algorithm to calculate the error between the model prediction and the real sample. Preferably, the error algorithm can be a DIOU loss algorithm, as follows:

[0062]

[0063] ground truth: Bgt=(xgt,ygt,wgt,hgt);

[0064] predicted box: B=(x,y,w,h);

[0065] x, y, w, and h are the upper left corner position and the width and height of the box respectively;

[0066]

[0067] b and b gt Represents B and B gt The center point of , ρ() represents the Euclidean distance, and c is the length of the diagonal of the smallest rectangular box surrounding the two boxes.

[0068] Pedestrian re-detection model: After the full-image pedestrian detection model, this patent will deduct the obtained pedestrian and cyclist area map (up to 6 areas), expand the area and resize it to 128*128 size, and send it to the pedestrian re-detection model. This size greatly reduces the computational complexity of the model. Figure 4 For engineering applications, this patent uses a small model based on vargnet as the backbone, reducing the number of blocks and channels related to UnitA and UnitB of the backbone, thereby speeding up the network.

[0069] The backbone of the pedestrian re-detection model is also divided into 6 stages for downsampling, followed by the neck and head to perform more accurate position detection on the subtracted area and compare it with the first detection result. At the same time, pedestrians and cyclists are classified as the second detection result.

[0070] Pedestrian Classification Model: The pedestrian classification results from the full-image pedestrian detection and pedestrian re-detection models are fed into the pedestrian classification model to identify attributes and obtain pedestrian classification results. The pedestrian classification model input is also a 128*128 area identified as a pedestrian. This model uses the same backbone as the pedestrian re-detection model. To meet multi-task requirements, this patent adds three classification heads and one detection head to the backbone.

[0071] The classification tasks are age classification, posture classification, and orientation classification. The three classification heads have the same structure and are not connected to the neck. They are directly connected to the head from the backbone and then connected to the cross-entropy classification loss.

[0072] The detection task is head detection, which outputs the position of the head. The detection head is connected to the output of the neck, which is consistent with the detection head structure of the pedestrian re-detection model.

[0073] Provide different inputs to each built model to complete the training of the corresponding project. The training of the full-image pedestrian detection model includes the following steps: 1. Crop the collected and labeled image data and resize it to 960*480 size; 2. Package the image and the corresponding label json file into a rec file according to different scenarios; 3. Set different sampling ratios for rec and input it into the model for training, and modify the learning rate, weight decay rate, positive and negative sample judgment threshold and other hyperparameters for iterative optimization training; The training steps of the pedestrian re-detection model are as follows: 1. Set the minimum frame pixel, deduct the area outside the annotation area where the annotated image pixel is larger than the minimum pixel, and resize it to 128*128 image size; 2. Modify the full-image label json file corresponding to the image into the corresponding deducted area label json file, and package it into a rec file according to different scenarios; 3. Set different sampling ratios for rec and input it into the model for training, and iteratively optimize the parameters according to the parameter adjustment method of the full-image pedestrian detection model. Pedestrian classification model training: 1. Set the minimum frame pixel, deduct the area outside the annotated area where the annotated image pixel is larger than the minimum pixel, and perform head detection and annotation tasks, age classification, orientation classification, and posture classification and annotation tasks. 2. Package different tasks separately, and each task has its own dataloader to ensure that the loss calculation of each task is not affected by the data of other tasks. 3. Send the separately packaged rec to the model for training. Because it is a multi-task model, the proportion of each task in the training is modified according to the evaluation results. At the same time, the proportion of each category in the branch is modified, and the relevant hyperparameters are modified to iteratively optimize the model training.

[0074] like Figure 6 The present application provides a multi-model-based pedestrian detection device, which includes at least one software function module stored in a storage module in the form of software or firmware, or in an operating system (OS) embedded in a control device. Each function module is configured to execute executable modules stored in the storage module, such as the software function modules and computer program modules included in the multi-model-based pedestrian detection device.

[0075] The device includes an image acquisition unit 10, a labeling unit 20, and a processing unit 30 that are communicatively connected in sequence. The functions of each unit may be as follows:

[0076] An image acquisition unit 10 acquires a pedestrian image;

[0077] A labeling unit 20 performs pedestrian detection labeling on the pedestrian image based on full-image labeling, and performs pedestrian attribute labeling on the pedestrian image based on cutout labeling;

[0078] The processing unit 30 obtains a first detection result based on the first model and the pedestrian detection annotation, obtains a second detection result based on the second model and the pedestrian attribute annotation, and completes pedestrian classification based on the third model and the first and second detection results.

[0079] The apparatus further includes a calculation unit 31 coupled to the processing unit 30. The calculation unit 31 can calculate the error between the prediction of the first model and the true sample based on the DIOU loss.

[0080] The labeling process, processing process, and correction process of each of the above units can be referred to the above multi-model-based pedestrian detection method, and will not be repeated here.

[0081] In this embodiment, the storage module may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc. In this embodiment, the storage module may be used to store preset parameters in each module (e.g., a correction algorithm in a computing unit), etc. Of course, the storage module may also be used to store programs, which the processing module executes upon receiving an execution instruction.

[0082] The present application also provides a computer-readable storage medium that stores a computer program, which, when executed on a computer, causes the computer to execute the multi-model-based pedestrian detection method described in the above embodiment.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented through hardware or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0084] In summary, the embodiments of the present application provide a multi-model-based pedestrian detection method, device and storage medium. The method includes the following steps: acquiring pedestrian images in different scenarios; performing pedestrian detection annotation on the pedestrian images based on full-image annotation, and performing pedestrian attribute annotation on the pedestrian images based on cutout annotation; acquiring a first detection result based on the first model and the pedestrian detection annotation, acquiring a second detection result based on the second model and the pedestrian attribute annotation, and acquiring a pedestrian classification result based on the first detection result, the second detection result and the third model. The present application adopts two models using two stages to perform precise positioning of pedestrian detection. Furthermore, a lightweight network such as Vargnet is selected as a training model, which is suitable for installation on a vehicle-mounted terminal GPU with small memory and complex environment, and can complete pedestrian detection well and quickly. The present application provides a model for classifying human attributes to assist the vehicle in subsequent manipulation.

[0085] In the embodiments provided in the present application, it should be understood that the disclosed devices, systems and methods can also be implemented in other ways. The device, system and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0086] The above describes in detail the multi-model-based pedestrian detection method, device, and storage medium provided by the present invention. The description of the specific embodiments is intended only to facilitate understanding of the method and core concepts of the present invention. It should be noted that those skilled in the art may make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A pedestrian detection method based on multiple models, characterized in that: The following steps are included: S1, obtain pedestrian images; S2, performing pedestrian detection annotation on the pedestrian image based on the full-image annotation, and performing pedestrian attribute annotation on the pedestrian image based on the cutout annotation; S3, obtaining a first detection result based on the first model and the pedestrian detection annotation, obtaining a second pedestrian detection result based on the second model and the pedestrian attribute annotation, and obtaining a pedestrian classification result based on the first detection result, the second pedestrian detection result, and the third model; The first model is a full-image pedestrian detection model. In the full-image pedestrian detection model, Vargnet is used as the backbone for 32x downsampling. The backbone is used to extract image features. The neck is used for upsampling, and the downsampled feature map is returned to 1 / 4 the size of the original image. The upsampling output of each neck layer is connected to the head network. The head network consists of a UnitA and a separable group convolution. The four head results output by the neck are combined to output the pedestrian detection position and category to obtain the first detection result. The second model is a pedestrian re-detection model. In the pedestrian re-detection model, a vargnet is used as a backbone for downsampling, followed by a neck and a head to detect the position of the subtracted area, and the position is compared with the first detection result, and pedestrians and cyclists are classified to obtain the second detection result; The third model is a pedestrian classification model. In the pedestrian classification model, Vargnet is used as the backbone, and three classification heads and a detection head are connected to the backbone. The three classification heads have the same structure and are not connected to the neck. The three classification heads are used for age classification, posture classification, and orientation classification respectively. The detection head is connected to the output of the neck.

2. The pedestrian detection method according to claim 1, characterized in that The full-image annotation includes: annotating the pedestrian image once based on a rectangular frame, wherein the annotated categories include pedestrians and cyclists.

3. The pedestrian detection method according to claim 2, characterized in that The cutout annotation includes: based on the full image annotation result, extracting and secondary annotation of the area where the pedestrians and cyclists are located on the pedestrian image, wherein the secondary annotation content is the attributes of the pedestrians and cyclists, wherein the attributes of the pedestrians and cyclists include head, age, direction and posture.

4. The pedestrian detection method according to claim 3, characterized in that: The number of blocks and channels related to UnitA and UnitB of the backbone in the first model is greater than the number of blocks and channels related to UnitA and UnitB of the backbone in the second model.

5. The pedestrian detection method according to claim 4, characterized in that: The third model is generated by adding a classification head and a detection head on the basis of the second model.

6. The pedestrian detection method according to claim 5, characterized in that: The method further includes calculating the error between the predicted value of the first model and the actual sample based on the DIOUloss algorithm.

7. A pedestrian detection device based on multiple models, characterized in that: include: An image acquisition unit (10) acquires a pedestrian image; A labeling unit (20) performs pedestrian detection labeling on the pedestrian image based on full-image labeling, and performs pedestrian attribute labeling on the pedestrian image based on cutout labeling; A processing unit (30) is configured to obtain a first detection result based on the first model and the pedestrian detection annotation, obtain a second detection result based on the second model and the pedestrian attribute annotation, and obtain a pedestrian classification result based on the first detection result, the second detection result, and the third model; The first model is a full-image pedestrian detection model. In the full-image pedestrian detection model, Vargnet is used as the backbone for 32x downsampling. The backbone is used to extract image features. The neck is used for upsampling, and the downsampled feature map is returned to 1 / 4 the size of the original image. The upsampling output of each neck layer is connected to the head network. The head network consists of a UnitA and a separable group convolution. The four head results output by the neck are combined to output the pedestrian detection position and category to obtain the first detection result. The second model is a pedestrian re-detection model. In the pedestrian re-detection model, a vargnet is used as a backbone for downsampling, followed by a neck and a head to detect the position of the subtracted area, and the position is compared with the first detection result, and pedestrians and cyclists are classified to obtain the second detection result; The third model is a pedestrian classification model. In the pedestrian classification model, Vargnet is used as the backbone, and three classification heads and a detection head are connected to the backbone. The three classification heads have the same structure and are not connected to the neck. The three classification heads are used for age classification, posture classification, and orientation classification respectively. The detection head is connected to the output of the neck.

8. The pedestrian detection device according to claim 7, characterized in that: Also includes: The calculation unit (31) calculates the error between the prediction of the first model and the real sample based on the DIOU loss algorithm.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is run on a computer, the method according to any one of claims 1 to 6 can be executed.

Images