Pedestrian data augmentation method and apparatus, and object detection method, electronic device and storage medium

Abstract

Disclosed in the present invention are a pedestrian data augmentation method and apparatus, and an object detection method, an electronic device and a storage medium. The pedestrian data augmentation method comprises: acquiring an autonomous driving scene picture, and a pedestrian database obtained by means of pedestrian data preprocessing, wherein the pedestrian database comprises a plurality of pieces of pedestrian data extracted from an image data set, and each piece of pedestrian data is at least provided with a pedestrian detection box; and selecting pedestrian data from the pedestrian database and splicing the pedestrian data into the autonomous driving scene picture to obtain a spliced picture, wherein the spliced picture has a corresponding pedestrian category and position information of a pedestrian in the autonomous driving scene picture.

Description

Pedestrian data augmentation method and device, target detection method, electronic device and storage medium TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of intelligent driving, in particular to a pedestrian data augmentation method and device, a target detection method, an electronic device and a storage medium. BACKGROUND

[0002] In related technologies, a target detection algorithm generally includes two stages. In the first stage, for a large amount of input training data, the target detection algorithm randomly samples a small batch of data each time, performs data augmentation on the data, calculates a loss function through a neural network, and propagates the loss function backward. After multiple iterations, a trained neural network is obtained. In the second stage, an image to be predicted is input into the neural network, and the corresponding pedestrian, vehicle and non-motor vehicle category and position information are output.

[0003] Existing data augmentation schemes use the following three methods: image processing on input training images, using the idea of reinforcement learning to generate image processing sequences and random parameters, and randomly erasing input images.

[0004] As shown in FIG. 1, the specific process of image processing on input training images is as follows: Step 1: random color enhancement; Step 2: filling the image on a random size mask; Step 3: random cropping of the mask; Step 4: enlarging / reducing the image to a specified size. This scheme has the advantage of large image transformation, which can effectively prevent neural network overfitting.

[0005] Another scheme uses the idea of reinforcement learning to generate image processing sequences and random parameters, and the specific process is as follows: Step 1: using a recurrent neural network to generate the type and sequence of image processing in data augmentation as a data augmentation scheme; Step 2: training the network and predicting using the data augmentation scheme generated in Step 1; Step 3: obtaining the recurrent neural network loss function and the gradient of the back propagation corresponding to this data augmentation scheme through the prediction result, and updating the parameters of the recurrent neural network; Step 4: executing Step 1, and continuing this cycle until the recurrent neural network converges. Step 5: using the data augmentation scheme generated by the converged recurrent neural network as the final data augmentation scheme. This scheme has the advantage of large search space, which can exhaustively search for the optimal data augmentation scheme, but has the disadvantage of high requirements for time resources and space resources, and is difficult to implement in resource-limited scenarios.

[0006] There is another kind of scheme, which randomly erases the input image, that is, randomly generates a small block in the image, and sets the pixels of the small block to a constant value. As shown in FIG. 2, the left is the original image, and the right is the output image after random erasing in two possible ways.

[0007] The inventors found in the process of implementing the present application that the main drawback of the above-mentioned data enhancement schemes in the related art is that no matter how the data enhancement scheme is designed, it is limited by the quantity and quality of the data set used for training.

[0008] SUMMARY

[0009] The embodiments of the present application provide a pedestrian data enhancement method and device, a target detection method, an electronic device, and a storage medium, which are used to at least solve one of the above technical problems.

[0010] In a first aspect, the embodiments of the present application provide a pedestrian data enhancement method, comprising: acquiring an autonomous driving scene picture and a pedestrian database obtained through pedestrian data preprocessing, wherein the pedestrian database comprises a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame; and selecting pedestrian data from the pedestrian database to splice into the autonomous driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian categories and position information of pedestrians in the autonomous driving scene picture.

[0011] In a second aspect, the embodiments of the present application provide a target detection method, comprising: inputting a to-be-detected picture into a trained target detection model; and acquiring a detection result output by the trained target detection model.

[0012] In a third aspect, the embodiments of the present application provide a pedestrian data enhancement device, comprising: an acquisition program module, configured to acquire an autonomous driving scene picture and a pedestrian database obtained through pedestrian data preprocessing, wherein the pedestrian database comprises a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame; and an enhancement program module, configured to select pedestrian data from the pedestrian database to splice into the autonomous driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian categories and position information of pedestrians in the autonomous driving scene picture.

[0013] In a fourth aspect, the embodiments of the present application provide an electronic device, comprising: at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any one of the above-mentioned pedestrian data enhancement methods and target detection methods.

[0014] In a fifth aspect, an embodiment of the present application provides a storage medium, wherein one or more programs including execution instructions are stored in the storage medium, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any one of the pedestrian data augmentation method and the target detection method described above.

[0015] In a sixth aspect, an embodiment of the present application further provides a computer program product, which comprises a computer program stored on a storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes any one of the pedestrian data augmentation method and the target detection method described above.

[0016] The method of the present application greatly increases the number of pedestrian labels by screening pedestrian data in an image data set, cutting out a large number of pedestrian images of pedestrian data, and randomly pasting them into existing automatic driving scene training data, increases the supervision information of the neural network, and effectively improves the detection effect of the human class. Further, the randomness of the existing labels is greatly increased, for example, in the original data set, the data of the type of vehicles occluded by pedestrians is less, and the random increase of pedestrians will lead to a great increase in the category of vehicles occluded by pedestrians, further improving the fitting ability of the neural network and improving the accuracy of pedestrian detection. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0018] FIG. 1 is a schematic diagram of image processing on an input training image in the related art;

[0019] FIG. 2 is a schematic diagram of random erasing on an input image in the related art;

[0020] FIG. 3 is a flowchart of a pedestrian data augmentation method provided by an embodiment of the present application;

[0021] FIG. 4 is a flowchart of a target detection method provided by an embodiment of the present application;

[0022] FIG. 5 is a Faster-RCNN model architecture diagram of one specific example of the pedestrian data augmentation method provided by an embodiment of the present application;

[0023] FIG. 6 is a whole algorithm flowchart of one specific example of the pedestrian data augmentation method provided by an embodiment of the present application;

[0024] FIG. 7 is a schematic diagram of an image after data enhancement of one specific example of the pedestrian data enhancement method provided by an embodiment of the present application;

[0025] FIG. 8 is a schematic diagram of an image after data enhancement of another specific example of the pedestrian data enhancement method provided by an embodiment of the present application;

[0026] FIG. 9 is a schematic diagram of the structure of a pedestrian data enhancement device provided by an embodiment of the present application;

[0027] FIG. 10 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0029] An autonomous vehicle / automatic driving vehicle mainly includes three modules of perception, decision and planning, the perception module is the first ring of automatic driving and is the link of vehicle and environment interaction. The overall performance of an automatic driving system is largely dependent on the performance of the perception system. The key visual perception algorithms in the perception module include two-stage detection, single-stage detection, Anchor-free detection and Transformer detection.

[0030] Two-stage detection refers to the way of realizing detection in two processes in order, one is to extract the object region, and the second is to classify and identify the region by convolutional neural network. Therefore, “two-stage” is also called target detection based on region proposal. Representative algorithms include R-CNN series (R-CNN, Fast R-CNN, Faster R-CNN, etc.). Among them, Faster R-CNN is an end-to-end detection network. The first stage uses a region generation network (RPN, Region Proposal Network) to generate candidate frames based on feature maps, and uses region of interest pooling (ROIPooling) to align the size of the candidate features; the second stage uses a fully connected layer for detailed classification and regression. Here, the idea of Anchor is proposed to reduce the difficulty of operation and improve the speed. Each position of the feature map will generate Anchors of different sizes and aspect ratios, which are used as references for object frame regression. The introduction of Anchor makes the regression task only need to deal with relatively small changes, so the network learning will be easier.

[0031] Compared with two-stage algorithms, single-stage algorithms only need to extract features once to realize target detection, and the speed algorithm is faster, and the general accuracy is slightly lower. The pioneering work of this kind of algorithm is YOLO (You Only Look Once) algorithm, which is a real-time target detection algorithm. It realizes the prediction of target region and the identification of target category through a single neural network, so as to convert the target detection task into a regression problem. The main advantages of YOLO algorithm include fast speed, simple pipeline, low background false detection rate, strong universality, and high detection rate for non-natural image objects.

[0032] Anchor-free detection generally represents objects as some key points, and convolutional neural network is used to regress the positions of these key points. The key points can be the center point, corner point or representative point of the object frame. CenterNet algorithm can convert the target detection problem into a center point prediction problem, that is, use the center point of the target to represent the target, and obtain the rectangular frame of the target by predicting the offset of the target center point and the height and width. Heat map represents classification information, and each class will generate a separate heat map. For each heat map, when a coordinate contains the center point of the target, a key point will be generated at the target, and a Gaussian circle is used to represent the entire key point.

[0033] No matter single-stage or two-stage object detection, no matter with or without Anchor, attention mechanism is not well utilized. In view of this situation, DETR (DEtection TRansformer) introduces attention mechanism into the field of object detection by using Transformer. DETR is an algorithm that applies Transformer to the field of object detection, which adopts the classical Encoder-Decoder structure. The backbone network of DETR is a convolutional network, while the Encoder and Decoder are based on the structure of Transformer. The output layer of the algorithm is a multilayer perceptron (MLP). DETR uses a loss function based on bipartite graph matching, which is based on the matching between ground truth (true value, real effective value) and predicted bounding box (bounding box). Finally, the performance of DETR is on par with Faster-RCNN.

[0034] The present inventors found in their research that in the prior art, because they only make bottom-level transformation on images, do not introduce any additional semantic information, and cannot increase any additional label information, they cannot improve the detection performance of a certain specific class, for example, in the automatic driving scene, the existing data enhancement scheme cannot solve the problem of lack of pedestrian data. In addition, the existing data enhancement scheme does not introduce any additional information to help improve the information capacity of the image, so it is difficult to further improve the performance of the trained neural network, and there are many image information and data sets on the network, so the existing data enhancement scheme is limited by the quantity and quality of the data set used for training.

[0035] Please refer to FIG. 3, which shows a flowchart of a pedestrian data enhancement method according to an embodiment of the present application. The pedestrian data enhancement method can exist in the computer in the form of an algorithm, and in the subsequent embodiments, it can be referred to as a pedestrian data enhancement device, such as a program code or a software, etc., which is not limited in the present application.

[0036] As shown in FIG. 3, in step 301, an automatic driving scene picture and a pedestrian database obtained by pedestrian data preprocessing are acquired, wherein the pedestrian database includes a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame;

[0037] In step 302, the pedestrian data is selected from the pedestrian database and spliced into the automatic driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian class and position information of the pedestrian in the automatic driving scene picture.

[0038] In this embodiment, for step 301, the pedestrian data augmentation device obtains an autonomous driving scene picture and a pedestrian database obtained by preprocessing pedestrian data, wherein the pedestrian database includes a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame. The autonomous driving scene picture has a corresponding semantic label, and the pedestrian data extracted from the image data set also has a corresponding semantic label. For example, the original autonomous driving scene picture used for training is denoted as X, wherein X is a matrix with a shape of HxWxC, H, W and C are the length, width and height of the image; the semantic label corresponding to the autonomous driving scene picture is denoted as Y, which indicates the categories and positions of pedestrians, vehicles and non-motor vehicles in the image, wherein Y is a matrix with a shape of Nx5, N is the number of corresponding targets, and the five elements of each row are represented as (x0, y0, x1, y1, c), which respectively correspond to the column number of the left upper corner of each target, the row number of the left upper corner, the column number of the right lower corner, the row number of the right lower corner and the corresponding category, when c = 1, the corresponding category is pedestrian; when c = 2, the corresponding category is vehicle; when c = 3, the corresponding category is non-motor vehicle. The pedestrian data extracted from the image data set also has a corresponding semantic label V, which is a matrix with a shape of Mx3, M is the number of extracted pedestrian data, and the three elements of each row are represented as (w i , h i , 1), w i represents the width of the pedestrian data, hi represents the height of the pedestrian data, and 1 represents that the extracted data is of the pedestrian category.

[0039] Then, for step 302, the pedestrian data augmentation device selects pedestrian data from the pedestrian database and splices it into the autonomous driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian categories and position information of the pedestrians in the autonomous driving scene picture. The selection method can be a random algorithm. In some specific examples, it is randomly selected whether to enhance the current autonomous driving scene picture, and the pedestrian data spliced into the autonomous driving scene picture and the corresponding number are randomly selected. In some specific examples, a predetermined number N of pedestrian data can be directly selected from the pedestrian database and spliced into the autonomous driving scene picture to obtain a spliced picture for training of the target detection model. Further, the position of each pedestrian data in the autonomous driving scene picture can also be determined by random or preset method, and then the pedestrian data is pasted to the corresponding position to replace the original pixel value, to obtain data-augmented image data and semantic label data, and finally the data-augmented image data and semantic label data are output to replace the original picture.

[0040] The method of the embodiment obtains an automatic driving scene picture and a pedestrian database obtained through preprocessing of pedestrian data, then selects pedestrian data from the pedestrian database to splice into the automatic driving scene picture to obtain a spliced picture, and updates a semantic label of the spliced picture, so that the obtained training data introduces more semantic information, and the accuracy and recall rate of the trained model for pedestrian detection can be improved.

[0041] In some optional embodiments, the preprocessing of the pedestrian data comprises: obtaining an image data set, screening pedestrian data related to a pedestrian category from each picture of the image data set, and processing the screened pedestrian data to form a pedestrian database.

[0042] In the embodiment, the obtained image data set has data of multiple categories. In a specific example, the image data set can be a public data set such as a Cityscapes data set, a waymo data set, a nuscenes data set, and a kitti data set, or a self-acquired data set or a purchased non-public data set, as long as the data set contains pedestrian detection boxes, and the present application is not limited herein. Taking the Cityscapes data set as an example, the Cityscapes data set is a large-scale data set specially designed for urban scene semantic segmentation tasks, which contains high-resolution street view images and accurate manual pixel-level labels. The data set is divided into a training set, a validation set, and a test set, and contains 30 different categories covering various elements in the urban environment, such as roads, pedestrians, vehicles, and buildings.

[0043] Then, pedestrian data related to a pedestrian category is screened from each picture of the image data set. Each data in the image data set has a corresponding category label, and the category label is screened to obtain data with a pedestrian category label. For example, in some data sets, a pedestrian is defined as data with a person category label in the image data set, and the image data set also includes data of other categories, which is not described herein. In some image data sets, another category, rider, also includes pedestrian information, but since the pedestrian in the rider category is similar in shape to the non-motor vehicle category in the automatic driving scene task, in order to avoid the network from being unable to distinguish pedestrians and non-motor vehicles, the pedestrian data of the rider category in the image data set is ignored, and only the person category is selected.

[0044] Finally, the screened pedestrian data is processed to form a pedestrian database. The step of forming the pedestrian database can be performed only once, and the pedestrian database can be directly used subsequently. Thus, the amount of pedestrian data can be increased.

[0045] In some optional embodiments, the processing of the screened pedestrian data to form a pedestrian database comprises: first, the pedestrian data enhancement device determines whether the screened pedestrian data meets a preset condition; then, if the screened pedestrian data meets the preset condition and the data in the image data set has a corresponding mask, pixels in the image of the pedestrian data that do not belong to the pedestrian foreground are set to 0 as background according to the mask corresponding to the pedestrian data; and finally, the processed image is saved to form a pedestrian database.

[0046] In the embodiment, the preset condition can include clarity, size, presence or absence of occlusion, etc., and is mainly used to eliminate pedestrian data with poor quality. For example, the detection box corresponding to each pedestrian data needs to meet that the length and width are both greater than a preset pixel, so that pedestrian data with serious occlusion or blurred and unclear distance can be removed, and pedestrian data with poor quality can be avoided.

[0047] Then, after the screened pedestrian data meets the preset condition, it is further determined whether the pedestrian data has a corresponding mask. In some specific examples, each image in some image data sets contains a detection box label of a pedestrian and a corresponding mask. The detection box can frame the circumscribed rectangle of the pedestrian, and the mask can set the position occupied by the pedestrian to 1 and the background position not occupied by the pedestrian to 0. Then, pixels in the image of the pedestrian data that do not belong to the pedestrian foreground are set to 0 as background according to the mask corresponding to the pedestrian data. Specifically, in image matting, a simple solution is to predict the transparency-alpha channel of the pixel to accurately remove the background, considering the saliency image with two main components of foreground and background. Due to the ambiguity of detecting the foreground region, for example, whether an object of a person belongs to the part of the human body foreground, many studies use a ternary graph to define the foreground, background and unknown or transition region, but creating a ternary graph in a video consumes a lot of resources. Compared with the ternary graph, the alternative binary mask is easier to obtain by drawing or ready-made segmentation model, and provides greater flexibility; wherein the alpha value refers to the transparency of color, which determines whether we can see the object behind an object. In computer graphics, the alpha value is usually represented by a floating-point number of 0 to 1, 0 means completely transparent, and 1 means completely opaque.

[0048] In some optional embodiments, determining whether the screened pedestrian data meets the preset condition further includes: if the screened pedestrian data meets the preset condition and the data in the image data set does not have a corresponding mask, cutting out the pedestrian image from the image of the pedestrian data, for example, using a lasso tool, a frame tool, an eraser tool, etc. to directly select, a quick mask, a pen sketch path and then a selection area, an extraction filter, an external filter extraction, a channel, a calculation, an application image method, etc. to separate the pedestrian image from the image of the pedestrian data to become a separate layer, which is not limited in the present application. Finally, save the processed image to form a pedestrian database, so as to realize accurate background removal and obtain a pedestrian image.

[0049] Further, the pedestrian data can also be segmented by semantics to identify the pedestrian area and the background, and then only the pedestrian area is obtained for splicing. Among them, semantic segmentation is an important branch of image processing and machine vision, and its goal is to accurately understand the image scene and content. Semantic segmentation is a classification at the pixel level, and pixels belonging to the same class are classified into a class, so semantic segmentation is to understand the image from the pixel level. For example, the pixels belonging to a person are classified into a class, the pixels belonging to a motorcycle are classified into a class, and the background pixels are classified into a class. Through semantic segmentation, each pixel in the image can be labeled with a class label, for example, different colors can be used to distinguish the labels.

[0050] In some optional embodiments, if the screened pedestrian data meets the preset condition and the data in the image data set does not have a corresponding mask, the cut-out pedestrian data can also be directly retained as a pedestrian image to form a pedestrian database.

[0051] In some optional embodiments, selecting pedestrian data from the pedestrian database to splice into the automatic driving scene picture includes: obtaining an automatic driving scene picture for training; then for each picture, confirming whether to splice pedestrian data for enhancement, wherein each pedestrian data has corresponding size information and category information. If it is confirmed to enhance, the position information and category information of the pedestrian data are added in the target data; finally, output the picture after splicing the pedestrian data for enhancement and the target data corresponding to the spliced picture.

[0052] In the embodiment, the pedestrian data augmentation device acquires an automatic driving scene picture for training, wherein the automatic driving scene picture has corresponding target data, and the target data includes position information and category information of at least one target on the picture. In a specific example, the automatic driving scene picture can be recorded as X, wherein X is, for example, a matrix with a shape of HxWxC, H, W and C are the length, width and height of the image; the target data can be recorded as Y, and the corresponding semantic label is used to indicate the category and position of the pedestrians, vehicles and non-motor vehicles in the image, wherein Y is, for example, a matrix with a shape of Nx5.

[0053] Subsequently, for each picture, it can be determined whether to splice pedestrian data for augmentation by a preset rule or a random manner. For example, a random algorithm can be used to determine whether to augment the current picture, and a random algorithm can also be used to determine the pedestrian data required for augmentation, the position of each pedestrian data, the scaling factor of each pedestrian data, and the like. Subsequent details are not described herein. If a random algorithm is not used, the augmentation, the pedestrian data required for augmentation, the position of each pedestrian data, the scaling factor of each pedestrian data, and the like can be directly preset, which is not limited herein. For example, all pictures can be augmented, and a preset number of pedestrian data, a preset scaling factor and a preset position can be used, and details are not described herein.

[0054] If it is determined to augment, the position information and category information of the pedestrian data are added to the target data. In a specific example, a position of a randomly generated pedestrian data in the automatic driving scene picture can be generated, and then the pedestrian data is pasted to the corresponding position to replace the original pixel value, to obtain an augmented picture after splicing pedestrian data and target data corresponding to the spliced picture. When the randomly generated pedestrian data is pasted to the corresponding position in the automatic driving scene picture, the entire picture can be pasted, or only the lower half of the picture or the upper half of the picture can be pasted, which is not limited herein. Finally, the augmented picture after splicing pedestrian data and the target data corresponding to the spliced picture are output. Thus, the combination of image splicing and target data can be realized, and the pedestrian data has more rich semantic information.

[0055] In some optional embodiments, the determining whether to splice pedestrian data for augmentation for each picture includes: determining whether to augment data for each picture by a random algorithm; then if it is determined to augment data, a preset number of pedestrian data is selected from a pedestrian database; and finally, the preset number of pedestrian data is traversed, and each pedestrian data is spliced on the picture.

[0056] In the embodiment, a 0-1 random number can be generated by a random probability distribution. If the random number is greater than 0.5, random pedestrian data augmentation is performed, otherwise the original image and the corresponding label are directly output. The random probability distribution can be a uniform distribution, a normal distribution according to the center point of the image, or other various distributions, which are not limited in the present application. If random pedestrian data augmentation is performed, a preset number of pedestrian data are selected from the pedestrian database. The preset number can be pre-set or generated by a random algorithm, which is not limited in the present application. Then the positions of each pedestrian data in the autonomous driving scene picture are generated in turn, and then the pedestrian data are pasted to the corresponding positions to replace the original pixel values. After multiple executions, the spliced pedestrian data augmented picture and the target data corresponding to the spliced picture are obtained, and finally the spliced pedestrian data augmented picture and the target data corresponding to the spliced picture are output. The pedestrian image in each pedestrian data can be a matrix of length x width x height.

[0057] The method of the embodiment can improve the fitting ability of the neural network by confirming whether to perform data augmentation by a random algorithm.

[0058] In some optional embodiments, the selecting pedestrian data from the pedestrian database and splicing the pedestrian data into the autonomous driving scene picture comprises: the pedestrian data augmentation device scales each selected pedestrian data according to a preset scaling factor, and splices the scaled pedestrian data on the autonomous driving scene picture. The preset scaling factor can be a fixed value set in advance or a randomly generated scaling factor, which is not limited in the present application. In a specific example, a scaling factor uniformly distributed from 0.3 times to 1 times can be randomly generated. For example, the generated scaling factor is 0.5 times, and then the pedestrian data is reduced by 0.5 times. The scaling factor can be 0.5 times to 1 times, or 0.3 times to 10 times, which is not limited in the present application. In physics, near large and far small generally refers to the relationship between the size, distance, force, etc. of an object. In mechanics, far small refers to a large distance between two objects, and near large refers to a small distance. In optics, far small means that a distant object appears small, and near large means that a close object appears large. According to the principle of near large and far small, the spliced picture can be made more consistent with the actual situation. For example, the scaling factor can be determined according to the size of the pedestrian image and the determined splicing position, so that the spliced image can appear more consistent with the real scene (such as consistent with the near large and far small rule), and thus the recognition accuracy of the target detection model trained using the spliced picture can be improved.

[0059] The method of the embodiment provides more data augmentation means by scaling each pedestrian data using a scaling factor, and thus the accuracy of pedestrian detection can be improved.

[0060] In some optional embodiments, the enhanced data is used for training of a neural network-based target detection model. For example, the target detection model can be based on Faster R-CNN, YOLO, CenterNet, and DETR.

[0061] Referring to FIG. 4, a flowchart of a target detection method according to an embodiment of the present application is shown. The target detection method can exist in the computer in the form of an algorithm, and in subsequent embodiments, it can be referred to as a target detection device, such as a program code or a software, etc., which is not limited in the present application.

[0062] As shown in FIG. 4, in step 401, an image to be detected is input into a target detection model; the target detection model is trained using the spliced image;

[0063] In step 402, a detection result output by the trained target detection model is obtained.

[0064] In the present embodiment, for step 401, the target detection device inputs the image to be detected into the target detection model trained using the spliced image in the above embodiment for target detection; and then, for step 402, the target detection device obtains the detection result output by the trained target detection model.

[0065] The method of the present embodiment performs target detection by inputting the image to be detected into the target detection model trained using the spliced image in the above embodiment, and since the target detection model is trained using the spliced image, more pedestrian data and corresponding semantic information are introduced, thereby improving the accuracy and recall rate of the target detection model for pedestrian detection.

[0066] Referring to FIG. 5, a Faster-RCNN model architecture diagram of one specific example of a pedestrian data enhancement method according to an embodiment of the present application is shown.

[0067] As shown in FIG. 5, Faster-RCNN can be divided into four main parts: Conv layers, RPN (Region Proposal Networks), Roi Pooling and Classification. Among them, Conv layers is a CNN network target detection method, Faster RCNN first uses a set of basic conv+ReLU+Pooling layers (convolution layer+ReLU layer+pooling layer) to extract the feature maps of the image. The feature maps are shared for subsequent RPN layers and fully connected layers. The RPN network is used to generate region proposals, and this layer judges whether the anchors belong to foreground or background through softmax, and then uses bounding box regression to correct the anchors to obtain accurate proposals. The Roi Pooling layer collects the input feature maps and proposals, extracts the proposal feature maps after integrating the information, and sends them to the subsequent fully connected layer to determine the target category. Classification uses proposal feature maps to calculate the category of the proposal, and again uses bounding box regression to obtain the final accurate position of the detection box.

[0068] Further referring to FIG. 6, it shows the overall algorithm flowchart of one specific example of the pedestrian data enhancement method provided by an embodiment of the present application.

[0069] As shown in FIG. 6, first, the pedestrian data is preprocessed, which includes the following steps:

[0070] Step 1: Download the public dataset Cityscape from the open source website. This dataset contains a large amount of road data, including various countries' downtown and suburban areas, and has a large number of pedestrians. Each image in the dataset contains the detection box label and the corresponding mask of the pedestrians. The detection box frames the bounding rectangle of the pedestrians, and the mask sets the positions of all pedestrians to 1 and all backgrounds to 0.

[0071] Step 2: Traverse the Cityscape dataset, and for each picture, do the following: select all the labels corresponding to pedestrians, including bounding boxes and masks; filter each pedestrian label, where a pedestrian is defined as the label named "person" in Cityscape, and another class "rider" in Cityscape also includes pedestrian information, but since the shape of the "rider" class is similar to the non-motor vehicle class in the autonomous driving scene task, in order to avoid the network from being unable to distinguish pedestrians and non-motor vehicles, this place ignores the pedestrians in the "rider" class in Cityscape and only selects the "person" class; in order to avoid introducing low-quality pedestrian labels, such as severely occluded pedestrians or distant and unclear pedestrians, each pedestrian also needs to meet the condition that the length and width of the corresponding bounding box are both greater than 35 pixels; cut out the image in the bounding box corresponding to the filtered pedestrian, and set the pixels in the image that do not belong to the pedestrian foreground to 0 as the background according to the mask; save the cut-out corresponding pedestrian image and form an image database, as shown in Figure 6 by the dashed line as a sample of part of the pedestrian image. The pedestrian data preprocessing step only needs to be performed once to obtain the pedestrian database, and does not need to be re-executed when training the neural network each time. Specifically, cutting out the image in the bounding box corresponding to the filtered pedestrian can include: first input a normal open source data picture and give the image coordinates of the upper left and lower right corners of the corresponding pedestrian label box (generally, the default label box is a rectangle, so only the coordinates of the upper left corner and the lower right corner need to be given). The pedestrian in the label box coordinates is extracted into a new picture and the new picture is saved. Because the cut-out pedestrian image is a rectangle, the image will contain background information, so the background pixels that do not belong to the pedestrian need to be set to 0 using the mask.

[0072] After obtaining the pedestrian database, in the data augmentation stage during the training of Faster-RCNN, data augmentation is performed according to the following steps:

[0073] Step 1: define that the input is an original autonomous driving scene picture used for training, denoted as X, which is a matrix with a shape of HxWxC, where H, W, and C are the length, width, and height of the image, respectively; and the corresponding semantic label indicates the class and position of pedestrians, vehicles, and non-motor vehicles in the image, denoted as Y, which is a matrix with a shape of Nx5, N is the number of corresponding targets, and the 5 elements in each row are represented as (x0, y0, x1, y1, c), which correspond to the left upper corner corresponding column number, the left upper corner corresponding row number, the right lower corner corresponding column number, the right lower corner corresponding row number, and the corresponding class, respectively. When c = 1, the corresponding class is pedestrian; when c = 2, the corresponding class is vehicle; when c = 3, the corresponding class is non-motor vehicle.

[0074] Step 2: Randomly generate a 0-1 uniformly distributed random number, if greater than 0.5, then perform random pedestrian data enhancement, otherwise directly output the original image and corresponding label.

[0075] Step 3: If random pedestrian data enhancement is performed, randomly generate an integer n∈{1, 2, 3, …, 6} with equal probability, defined as the number of pedestrian data spliced on this picture, and randomly select n pedestrian images from the pedestrian database, define the length and width of the i-th pedestrian as h i , w i , and the corresponding image as x i ’ is a matrix with shape h i ×w i ×c.

[0076] Step 4: Traverse n pedestrians, and for the i-th pedestrian, perform the following steps:

[0077] First, randomly generate a scaling factor s∈[0.3, 1] uniformly distributed from 0.3 to 1, scale the pedestrian image to shape sh i ×sw i ×c, and the scaled image is denoted as x i ’.

[0078] Generate t,1 defined as the corresponding top-left corner position of the pedestrian data map in the input training data, t,1 is uniformly distributed, denoted as t~U(0, H-sh i ),1~(0,W-swi).

[0079] Paste all non-zero (background) pixel values in x i ’ to the input image X in the corresponding position with column number t to t+sh i and row number 1 to 1+sw i , replacing the original pixel values in the corresponding position.

[0080] Add a row to the input label Y, and the corresponding 5 element values are [l, t, l+swi, t+shi, 1], execute n times to get the data enhanced image data X’ and label data Y’.

[0081] Each execution will generate new annotations corresponding to the original annotations. The original n pedestrian images have n annotations, and after execution, there will also be n new annotations.

[0082] Step 5: Output X’ and Y’, replace the original image.

[0083] In one specific example, firstly, we have an original picture in a training set, which has only one car (corresponding to the category 3), and the corresponding label is [100, 100, 200, 300, 3], which corresponds to the upper left corner coordinates (10, 100), the lower right corner coordinates (200, 300), and the category is 3. Secondly, suppose we randomly generate n = 2 pedestrians, and then select the data of two pedestrians, the first pedestrian has a length l = 10 and a width w = 20, and the second pedestrian has a length l = 20 and a width w = 40. Then, we randomly scale the length and width of the pedestrians, and then randomly generate a position and paste the image at the position, and the corresponding label is [50, 100, 60, 120, 1; 300, 200, 322, 244, 1]. Finally, we concatenate the pedestrian label and the original label to obtain the new label [100, 100, 200, 300, 3; 50, 100, 60, 120, 1; 300, 200, 322, 244, 1].

[0084] Further, the way of increasing n pedestrian data is equivalent to executing the above process n times, which is not described herein.

[0085] As shown in FIGS. 7 and 8, they are schematic diagrams of images after data augmentation according to the embodiments of the present application, wherein the white boxes are the detection boxes of all targets.

[0086] Please refer to FIG. 9, which shows a structural schematic diagram of a pedestrian data augmentation device according to an embodiment of the present application.

[0087] As shown in FIG. 9, the pedestrian data augmentation device 900 includes an acquisition program module 910 and an augmentation program module 920.

[0088] The acquisition program module 910 is configured to acquire an autonomous driving scene picture and a pedestrian database obtained by pre-processing pedestrian data, wherein the pedestrian database includes a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection box. The augmentation program module 920 is configured to select pedestrian data from the pedestrian database and splice it into the autonomous driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian categories and position information of the pedestrians in the autonomous driving scene picture.

[0089] It should be understood that the modules described in FIG. 9 correspond to the steps in the method described with reference to FIG. 3. Therefore, the operations and features described above for the method and the corresponding technical effects also apply to the modules in FIG. 9, which will not be described herein.

[0090] It is worth noting that the modules in the embodiments of the present application are not used to limit the solutions of the present application. For example, the obtaining program module can be described as obtaining an autonomous driving scene picture and a pedestrian database obtained by preprocessing pedestrian data, wherein the pedestrian database includes a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame. In addition, the related function modules can also be implemented by a hardware processor, for example, the obtaining program module can also be implemented by a processor, and details are not described here.

[0091] In some other embodiments, the present application also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are used to execute the pedestrian data enhancement method and the target detection method in any of the above method embodiments.

[0092] As an embodiment, the non-volatile computer storage medium of the present application stores computer executable instructions, and the computer executable instructions are configured to:

[0093] obtain an autonomous driving scene picture and a pedestrian database obtained by preprocessing pedestrian data, wherein the pedestrian database includes a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame;

[0094] select pedestrian data from the pedestrian database and splice the pedestrian data into the autonomous driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian categories and position information of pedestrians in the autonomous driving scene picture.

[0095] As another embodiment, the non-volatile computer storage medium of the present application stores computer executable instructions, and the computer executable instructions are configured to:

[0096] input a to-be-detected picture into a target detection model, wherein the target detection model is trained using the spliced picture;

[0097] obtain a detection result output by the trained target detection model.

[0098] The non-volatile computer readable storage medium can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created according to the use of the pedestrian data augmentation device and the target detection device. In addition, the non-volatile computer readable storage medium can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the non-volatile computer readable storage medium can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the pedestrian data augmentation device or the target detection device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0099] The embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer readable storage medium, and the computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above pedestrian data augmentation method and target detection method.

[0100] FIG. 10 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. As shown in FIG. 10, the device includes one or more processors 1010 and a memory 1020, and FIG. 10 takes one processor 1010 as an example. The device of the pedestrian data augmentation method or the target detection method can also include an input device 1030 and an output device 1040. The processor 1010, the memory 1020, the input device 1030, and the output device 1040 can be connected through a bus or other means, and FIG. 10 takes the connection through the bus as an example. The memory 1020 is the non-volatile computer readable storage medium described above. The processor 1010 performs various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1020, that is, implements the pedestrian data augmentation method or the target detection method of the method embodiments described above. The input device 1030 can receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the pedestrian data augmentation device or the target detection device. The output device 1040 can include a display device such as a display screen.

[0101] The above product can perform the method provided by the embodiments of the present application, has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the present embodiments can be referred to the method provided by the embodiments of the present application.

[0102] As an implementation form, the electronic device is applied to a pedestrian data augmentation device and a target detection device, and includes at least one processor, and a memory connected with the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0103] obtain an automatic driving scene picture and a pedestrian database obtained through pedestrian data preprocessing, wherein the pedestrian database includes a plurality of pedestrian data extracted from an image data set, and each pedestrian data has at least a pedestrian detection frame;

[0104] select pedestrian data from the pedestrian database and splice the pedestrian data into the automatic driving scene picture to obtain a spliced picture, wherein the spliced picture has corresponding pedestrian categories and position information of pedestrians in the automatic driving scene picture.

[0105] As another implementation form, the electronic device is applied to a pedestrian data augmentation device and a target detection device, and includes at least one processor, and a memory connected with the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0106] input a to-be-detected picture into a target detection model, wherein the target detection model is trained using the spliced picture;

[0107] obtain a detection result output by the trained target detection model.

[0108] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0109] (1) Mobile communication device: the feature of this type of device is to have mobile communication function and to mainly provide voice and data communication. This type of terminal includes smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0110] (2) Ultra-mobile personal computer device: this type of device belongs to the category of personal computers and has computing and processing functions, and generally has the feature of mobile Internet. This type of terminal includes PDA, MID and UMPC devices, such as iPad.

[0111] (3) Portable entertainment device: this type of device can display and play multimedia content. This type of device includes audio and video players (such as iPod), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0112] (4) Server: a device providing computing services, the configuration of the server includes a processor, a hard disk, a memory, a system bus, etc., the server is similar to a general computer architecture, but since high reliable services need to be provided, therefore, higher requirements are required in processing capability, stability, reliability, security, scalability, manageability, etc.

[0113] (5) Other electronic devices with data interaction function.

[0114] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0115] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of each embodiment or some parts of the embodiment.

[0116] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A pedestrian data augmentation method, comprising: The system acquires images of autonomous driving scenarios and a pedestrian database obtained through pedestrian data preprocessing. The pedestrian database includes multiple pedestrian data extracted from an image dataset, and each pedestrian data has at least a pedestrian detection box. Pedestrian data is selected from the pedestrian database and stitched into the autonomous driving scene image to obtain a stitched image, wherein the stitched image has a corresponding pedestrian category and the pedestrian's location information in the autonomous driving scene image.

2. The method according to claim 1, characterized in that, The pedestrian data preprocessing includes: Obtain an image dataset, wherein the image dataset contains data of multiple categories; Filter pedestrian data related to pedestrian categories from the image dataset; The selected pedestrian data is processed to form a pedestrian database.

3. The method according to claim 1, characterized in that, The process of processing the selected pedestrian data to form a pedestrian database includes: Determine whether the filtered pedestrian data meets the preset conditions; If the selected pedestrian data meets the preset conditions and the data in the image dataset has a corresponding mask, then according to the mask corresponding to the pedestrian data, the pixels in the image of the pedestrian data that do not belong to the pedestrian foreground are set to 0 as the background. The processed images are saved to form a pedestrian database.

4. The method according to claim 3, characterized in that, Also includes: If the selected pedestrian data meets the preset conditions and the data in the image dataset does not have a corresponding mask, then the pedestrian image is extracted from the image of the pedestrian data; The extracted pedestrian image is used as the processed image.

5. The method according to claim 1, characterized in that, The step of selecting pedestrian data from the pedestrian database and stitching it into the autonomous driving scene image includes: Acquire autonomous driving scene images for training, wherein the images have corresponding target data, and the target data includes the position information and category information of at least one target on the image; For each image, determine whether to stitch in pedestrian data for enhancement, where each pedestrian data has corresponding size and category information; If enhancement is confirmed, the location and category information of the pedestrian data will be added to the target data. Output the image after stitching together the pedestrian data for augmentation, and the target data corresponding to the stitched image.

6. The method according to claim 5, characterized in that, The step of determining whether to stitch pedestrian data together for enhancement for each image includes: For each image, a random algorithm is used to determine whether data augmentation should be performed. If data augmentation is confirmed, a preset number of pedestrian data points are selected from the pedestrian database; The preset number of pedestrian data points are traversed, and the data points of each pedestrian are stitched together on the image.

7. The method according to any one of claims 1-6, characterized in that, The step of selecting pedestrian data from the pedestrian database and stitching it into the autonomous driving scene image includes: The selected pedestrian data is scaled according to a preset scaling factor, and the scaled pedestrian data is then stitched onto the autonomous driving scene image.

8. The method according to any one of claims 1-7, characterized in that, The enhanced data is used to train a neural network-based object detection model.

9. A target detection method, comprising: The image to be detected is input into the target detection model trained by the stitched image according to any one of claims 1-8; Obtain the detection results output by the trained target detection model.

10. A pedestrian data augmentation device, comprising: The acquisition module is used to acquire autonomous driving scene images and a pedestrian database obtained after pedestrian data preprocessing. The pedestrian database includes multiple pedestrian data extracted from the image dataset, and each pedestrian data has at least a pedestrian detection box. An enhancement module is used to select pedestrian data from the pedestrian database and stitch it into the autonomous driving scene image to obtain a stitched image, wherein the stitched image has a corresponding pedestrian category and the pedestrian's location information in the autonomous driving scene image.

11. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 9.

12. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images