Pedestrian detection method and device, equipment, storage medium and product

Abstract

The invention provides a pedestrian detection method and device, equipment, a storage medium and a product. The method comprises the steps that a preset neural network model used for pedestrian detection and a to-be-detected image are acquired, the preset neural network model comprises a backbone network, a neck network and a head network, and the backbone network comprises variable kernel convolution; inputting the to-be-detected image into the backbone network, and performing feature extraction on the to-be-detected image through the variable kernel convolution and other modules to obtain image feature information; and performing pedestrian detection on the to-be-detected image based on the image feature information through a neck network and a head network in the preset neural network model to obtain a pedestrian detection result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a pedestrian detection method, device, equipment, storage medium and product. Background Art

[0002] With the development of science and technology, autonomous driving technology has gradually begun to be put into practical application. In autonomous driving technology, the response speed and accuracy of pedestrian detection are crucial, which is directly related to road safety and the reliability and safety of the corresponding autonomous driving technology, and directly determines whether the autonomous driving technology can be used. Therefore, pedestrian detection is a key part of the autonomous driving system.

[0003] In existing autonomous driving technologies, pedestrian detection is usually performed through image recognition. In the process of image recognition, convolutional neural networks are generally used, and the convolution kernels in the convolution operations in convolutional neural networks in the prior art are usually fixed. However, as pedestrians are targets with non-fixed shapes and move irregularly, the current convolution kernels can only capture local information and cannot obtain information in a larger range or at different locations. This limits the adaptability of convolutional neural networks to different possible shapes of pedestrians, resulting in low accuracy in pedestrian detection. Summary of the invention

[0004] The embodiments of the present application provide a pedestrian detection method, device, equipment, storage medium and product, which can improve the accuracy of pedestrian detection.

[0005] In a first aspect, the present application provides a pedestrian detection method, comprising:

[0006] Acquire a preset neural network model for pedestrian detection and an image to be detected, wherein the preset neural network model includes a backbone network, a neck network, and a head network, and the backbone network includes a variable kernel convolution;

[0007] Inputting the image to be detected into the backbone network, extracting features of the image to be detected through the backbone network to obtain feature information of the image, wherein variable kernel convolution is used in the backbone network;

[0008] Pedestrian detection is performed on the image to be detected based on the image feature information through the neck network and the head network in the preset neural network model to obtain a pedestrian detection result.

[0009] In some possible implementations, performing pedestrian detection on the image to be detected based on the image feature information using the neck network and the head network in the preset neural network model to obtain a pedestrian detection result includes:

[0010] Performing partial convolution and feature fusion on the image feature information through the neck network to obtain image feature information after feature fusion;

[0011] The head network performs pedestrian detection on the image to be detected based on the image feature information after the feature fusion to obtain a pedestrian detection result.

[0012] In some possible implementations, performing partial convolution and feature fusion on the image feature information through the neck network to obtain the image feature information after feature fusion includes:

[0013] Determining target data points in the image feature information according to a preset mask;

[0014] Performing point-by-point convolution on the target data point through the neck network to obtain a first convolution result;

[0015] Partial convolution and feature fusion are performed on the first convolution result to obtain image feature information after feature fusion.

[0016] In some possible implementations, before obtaining the preset neural network model for pedestrian detection and the image to be detected, the method further includes:

[0017] Acquire a training sample set; the training sample set includes a plurality of training samples, each of the training samples includes a sample image and a labeling file corresponding to the sample image, and the labeling file includes a pedestrian identification box label in the sample image;

[0018] For each training sample, perform the following steps:

[0019] Inputting the sample image into a preset neural network model to determine a prediction detection result for predicting a pedestrian position in the sample image, wherein the prediction detection result includes a pedestrian prediction identification box;

[0020] Determining the loss function value of the preset neural network model according to the pedestrian prediction identification frame and the corresponding pedestrian identification frame label of each training sample;

[0021] When the loss function value of the preset neural network model does not satisfy the preset training stop condition, the model parameters of the preset neural network model are adjusted, and the sample image is input into the preset neural network model to return to the step of inputting the sample image into the preset neural network model to determine the prediction detection result of the pedestrian position in the sample image;

[0022] When the loss function value of the preset neural network model meets the preset training stop condition, the training is stopped to obtain the trained preset neural network model.

[0023] In some possible implementations, determining the loss function value of the preset neural network model according to the pedestrian prediction identification box and the corresponding pedestrian identification box label of each training sample includes:

[0024] Calculating the area of ​​an overlapping region between the pedestrian prediction identification frame and the pedestrian identification frame label according to the positional relationship between the pedestrian prediction identification frame and the pedestrian identification frame label to obtain a first area;

[0025] Subtract the area of ​​the overlapping area from the sum of the areas of the pedestrian prediction identification frame and the pedestrian identification frame label to obtain a second area;

[0026] Calculating a ratio of the first area to the second area to obtain a first intersection-and-union ratio;

[0027] Determining position offset information between the pedestrian prediction identification frame and the pedestrian identification frame label based on the position of the pedestrian prediction identification frame and the position of the pedestrian identification frame label;

[0028] Determine a DIou loss value according to the first intersection-over-union ratio and the position offset information;

[0029] Determining an adjustment range corresponding to the first intersection-and-combination ratio according to a preset attention factor;

[0030] The loss function value of the preset neural network model is determined based on the first intersection-and-union ratio, the DIou loss value and the adjustment amplitude.

[0031] In some possible implementations, the preset attention factor includes a first attention factor and a second attention factor, the first attention factor is smaller than the second attention factor, and determining the adjustment range corresponding to the first intersection-over-union ratio according to the preset attention factor includes:

[0032] When the first intersection-and-union ratio is less than the first focus factor, determining the adjustment amplitude corresponding to the intersection-and-union ratio as a first preset value;

[0033] When the first intersection-and-union ratio is greater than the second focus factor, the adjustment amplitude corresponding to the intersection-and-union ratio is determined to be a second preset value, and the first preset value is less than the second preset value;

[0034] When the first IoU is greater than the first focus factor and less than the second focus factor, an adjustment amplitude corresponding to the first IoU is determined according to a difference between the first focus factor and the second focus factor and the first IoU.

[0035] In some possible implementations, determining the loss value according to the first intersection-over-union ratio and the position offset information includes:

[0036] The loss value is determined by the following formula:

[0037]

[0038] Among them, L DIoU represents the loss value, IoU represents the first intersection-over-union ratio, d is the distance between the reference point of the predicted identification box and the reference point of the identification box label, and c is the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the real box. Indicates position offset information.

[0039] In some possible implementations, obtaining a training sample set includes:

[0040] Obtaining an initial sample image and a labeling file corresponding to the initial sample image;

[0041] Performing data expansion on the initial sample image to obtain a plurality of derivative sample images corresponding to the initial sample image;

[0042] Constructing samples based on the initial sample image and the corresponding annotation file, and the derived image and the corresponding annotation file respectively;

[0043] A training sample set is constructed according to the training samples.

[0044] In some possible implementations, the data expansion of the initial sample image includes at least one of flipping the initial sample image, performing color space transformation on the initial sample image, and adding noise to the initial sample image.

[0045] In some possible implementations, after obtaining the preset neural network model for pedestrian detection and the image to be detected, the method further includes:

[0046] De-noising the image to be detected and enhancing the contrast of the image to be detected;

[0047] Calculating the ratio of the aspect ratio of the image to be detected to the target aspect ratio to obtain a scaling ratio;

[0048] Scaling the image to be detected according to the scaling ratio;

[0049] The step of inputting the image to be detected into the preset neural network model, extracting features of the image to be detected through the backbone network, and obtaining feature information of the image, wherein variable kernel convolution is used in the backbone network, comprises:

[0050] The scaled image to be detected is input into the preset neural network model, and the feature extraction of the image to be detected is performed through the backbone network to obtain feature information of the image, wherein variable kernel convolution is used in the backbone network.

[0051] In a second aspect, the present application provides a pedestrian detection device, the device comprising:

[0052] An acquisition module, used to acquire a preset neural network model for pedestrian detection and an image to be detected, wherein the preset neural network model includes a backbone network, a neck network and a head network, and the backbone network includes a variable kernel convolution;

[0053] An input module, used for inputting the image to be detected into the backbone network, performing feature extraction on the image to be detected through the backbone network to obtain feature information of the image, wherein variable kernel convolution is used in the backbone network;

[0054] The detection module is used to perform pedestrian detection on the image to be detected based on the image feature information through the neck network and the head network in the preset neural network model to obtain a pedestrian detection result.

[0055] In a third aspect, the present application provides a pedestrian detection device, comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the pedestrian detection method as described above.

[0056] In a fourth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions, when executed by a processor, implement the pedestrian detection method as described above.

[0057] In a fifth aspect, the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the pedestrian detection method described above.

[0058] The pedestrian detection method, device, equipment, storage medium and product provided in the embodiments of the present application perform feature extraction through variable kernel convolution, which can adaptively adjust the size and shape of the convolution kernel. This adaptability enables the preset neural network model to more accurately capture different parts of the target even if the target presents different sizes or directions in the image, thereby being able to better adapt to the changing characteristics of non-rigid targets such as pedestrians. The preset neural network model can more effectively extract features, thereby improving the performance of pedestrian detection and improving the accuracy of pedestrian detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present application can be better understood from the following description of the specific embodiments of the present application in conjunction with the accompanying drawings, in which:

[0060] Other features, objects and advantages of the present application will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings, in which the same or similar reference numerals represent the same or similar features.

[0061] Figure 1 is a flow chart of a pedestrian detection method provided by an embodiment of the present application;

[0062] Figure 2 is a flowchart of a preset neural network model training process provided by an embodiment of the present application;

[0063] Figure 3 It is a hierarchical schematic diagram of a preset neural network model provided by an embodiment of the present application;

[0064] Figure 4 is a structural schematic diagram of a pedestrian detection device provided by an embodiment of the present application;

[0065] Figure 5 It is a schematic diagram of the hardware structure of the pedestrian detection device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0068] In order to solve the problems in the prior art, the embodiments of the present application provide a pedestrian detection method, device, equipment, storage medium and product. The pedestrian detection method provided by the embodiments of the present application is first introduced below.

[0069] Figure 1 FIG. 1 is a flow chart of a pedestrian detection method provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps S101 to S103.

[0070] Step S101: Obtain a preset neural network model for pedestrian detection and an image to be detected, wherein the preset neural network model includes a backbone network, a neck network and a head network, and the backbone network includes a variable kernel convolution.

[0071] In this embodiment, the preset neural network model consists of three parts: a backbone network, a neck network, and a head network. Among them, the backbone network is the basic part of the entire neural network model and is responsible for extracting image features. In the backbone network, the input data is convolved by variable kernel convolution. The neck network is located between the backbone network and the head network and is used to further process and integrate the features extracted from the backbone network. The head network predicts the location and type of the target based on the processed feature map.

[0072] The images to be detected may be original images input to a preset neural network model, and these images may contain images of pedestrians to be detected.

[0073] In the specific implementation, a pre-trained pedestrian detection model is loaded. This model is trained through a deep learning framework, including a backbone network, a neck network, and a head network. And the image to be detected is loaded from a preset storage device.

[0074] Step S102: input the above-mentioned image to be detected into the above-mentioned backbone network, and perform feature extraction on the above-mentioned image to be detected through the above-mentioned variable kernel convolution to obtain image feature information.

[0075] In the specific implementation, the preprocessed image is input into the backbone network part of the preset neural network model. The backbone network is a convolutional neural network, which contains variable kernel convolutions, which can be dynamically adjusted according to the content of the input image. The image is processed in turn through the convolution layer and the pooling layer, and the multi-level feature representation is extracted to obtain the image feature information.

[0076] As another example, the loaded image to be detected may be preprocessed, such as performing a normalization operation to scale the pixel values ​​to between 0 and 1 or performing mean normalization according to the requirements of the pre-trained model to ensure that the image meets the input requirements of the model.

[0077] As another example, at the output of the backbone network, we get image feature tensors after multiple layers of convolution and pooling operations. These feature tensors capture the semantic information and visual features of the image, providing basic data for subsequent pedestrian detection.

[0078] Step S103 performs pedestrian detection on the image to be detected based on the image feature information using the neck network and the head network in the preset neural network model to obtain a pedestrian detection result.

[0079] In a specific implementation, the image feature information obtained in step S102 is used as input and passed to the neck network and the head network in the preset neural network model, which perform convolution operations on the image features to generate a feature map. By decoding and analyzing the feature map, the position and bounding box of the pedestrian in the image are determined to obtain the pedestrian detection result.

[0080] The pedestrian detection method provided in the embodiment of the present application performs feature extraction through variable kernel convolution, which can adaptively adjust the size and shape of the convolution kernel. This adaptability enables the preset neural network model to more accurately capture different parts of the target even if the target presents different sizes or directions in the image, thereby being able to better adapt to the changing characteristics of non-rigid targets such as pedestrians. The preset neural network model can more effectively extract features, thereby improving the performance of pedestrian detection and improving the accuracy of pedestrian detection.

[0081] In order to speed up the processing speed of pedestrian detection, in some implementations, the above S103, obtaining the pedestrian detection result, includes steps A1 to B1:

[0082] Step A1: Perform partial convolution and feature fusion on the above image feature information through the above neck network to obtain image feature information after feature fusion.

[0083] In the specific implementation, in step S102, we have obtained the feature information of the image from the backbone network. This information usually exists in the form of a multi-dimensional tensor, representing the feature representation of the image at different levels of abstraction. In the processing of the neck network, a partial convolution operation is included. Partial convolution means that in the convolution operation, only a part of the input is operated, usually through a mask or a region of interest. This can effectively focus on the important parts of the image to improve detection accuracy and efficiency. The neck network fuses the feature information of different layers and levels of abstraction to obtain the image feature information after feature fusion.

[0084] As another example, feature fusion can be achieved by stacking feature maps, correlating channels, etc., to produce richer and more effective feature representations.

[0085] Step B1: Perform pedestrian detection on the image to be detected through the head network based on the image feature information after the feature fusion to obtain a pedestrian detection result.

[0086] In a specific implementation, after obtaining the image feature information after processing and feature fusion by the neck network, the head network may output one or more bounding boxes according to its specific design, each of which represents the position and size of a detected pedestrian in the image, thereby obtaining the pedestrian detection result.

[0087] In the implementation method provided in the embodiment of the present application, the neck network is used to perform partial convolution on the image feature information to obtain the image feature information after feature fusion. Then, the head network is used to perform pedestrian detection on the image to be detected based on the image feature information after feature fusion to obtain the pedestrian detection result. Since partial convolution is used in the neck network, the processing speed of pedestrian detection is accelerated.

[0088] In order to obtain more accurate image feature information, in some embodiments, the above A1 includes steps A2 to C2:

[0089] Step A2: Determine the target data point in the above image feature information according to the preset mask.

[0090] The preset mask can be a specified binary information used to mark the target area we are interested in. This may be generated by the segmentation network or set according to a specific task.

[0091] In a specific implementation, for each input image, by applying a preset mask, the location of the target data point can be determined, and the target data point in the above image feature information can be determined. The target data point is usually the pixel position marked as the target category in the mask. According to the preset mask, the feature representation of the target data point can be extracted.

[0092] As another example, the target data point can be a subset of the original image features or features recalculated for the target region. These representations will be used for the point-by-point convolution operation in the following step B2.

[0093] Step B2: Perform point-by-point convolution on the target data points through the neck network to obtain a first convolution result.

[0094] In a specific implementation, the target data point features outputted in step A2 are fed into the neck network. The neck network performs point-by-point convolution operations on the target data points. After point-by-point convolution of each target data point, a new feature representation will be obtained, i.e., the first convolution result.

[0095] Step C2: Perform partial convolution on the first convolution result to obtain image feature information after feature fusion.

[0096] In a specific implementation, partial convolution is performed on the first convolution result. After the partial convolution process, the output obtained is the image feature information after feature fusion.

[0097] In the implementation method provided by the embodiment of the present application, after determining the target data point in the image feature information according to the preset mask, the neck network performs point-by-point convolution on the target data point, and finally obtains the first convolution result. The first convolution result is then partially convolved to obtain the image feature information after feature fusion. Through point-by-point convolution, more accurate image feature information can be obtained.

[0098] In order to obtain a preset neural network model that can accurately detect pedestrians after training, in some embodiments, reference Figure 2 , before the above S101, the above method further includes steps S001 to S006:

[0099] Step S001: obtaining a training sample set; the training sample set includes a plurality of training samples, each of which includes a sample image and a labeling file corresponding to the sample image, wherein the labeling file includes a pedestrian identification box label in the sample image.

[0100] In the specific implementation, various sample images are collected in advance, and annotations are performed on the basis of the sample images to mark the positions of each pedestrian in the sample images and obtain an annotation file. The sample images and the corresponding annotation files are combined into training samples. Then, multiple training samples are combined into a training sample set. When needed, the training sample set is loaded and obtained.

[0101] For each training sample, the following steps S002 to S006 are performed respectively:

[0102] Step S002: Input the sample image into a preset neural network model to determine the result of predicting the position of the pedestrian in the sample image, wherein the prediction detection result includes a pedestrian prediction identification box.

[0103] In a specific implementation, a sample image in a training sample is input into a preset neural network model, and the neural network model generates a predicted target frame and target type, and obtains a predicted detection result of the target in the sample image.

[0104] Step S003: Determine the loss function value of the preset neural network model according to the pedestrian prediction identification box and the corresponding pedestrian identification box label of each of the training samples.

[0105] In a specific implementation, based on a pre-selected loss function, the loss function value of the above-mentioned preset neural network model is calculated according to the results predicted by the neural network model and the actual annotations in the annotation file.

[0106] Step S004: Determine whether the loss function value of the preset neural network model meets the preset training stop condition.

[0107] In the specific implementation, it is determined whether the current loss function value meets the preset training stop condition. This condition can be other evaluation indicators such as the loss function reaching a sufficiently small threshold, or stopping training after a certain number of iterations. If yes, execute S006, if not, execute S005.

[0108] Step S005: When the loss function value of the above-mentioned preset neural network model does not meet the preset training stop condition, adjust the model parameters of the above-mentioned preset neural network model, return to the step of inputting the above-mentioned sample image into the preset neural network model, and determine the prediction detection result of predicting the position of the pedestrian in the above-mentioned sample image.

[0109] In a specific implementation, when the loss function value of the preset neural network model does not meet the preset training stop condition, the parameters of the neural network model are adjusted through the back propagation algorithm to reduce the value of the loss function. After updating the parameters, the step of inputting the sample image into the preset neural network model to determine the predicted detection result of predicting the position of the pedestrian in the sample image is returned. A new round of training is carried out again to continuously optimize the prediction ability of the model.

[0110] Step S006: When the loss function value of the above-mentioned preset neural network model meets the preset training stop condition, stop training to obtain the trained preset neural network model.

[0111] In the specific implementation, if the loss function value meets the preset training stop condition, the training process is stopped and the current preset neural network model is used as the trained preset neural network model. The model has undergone multiple rounds of iterative training and can accurately predict the position of pedestrians.

[0112] The above implementation method provided in the embodiment of the present application obtains a training sample set and uses the training sample set to train the preset neural network model. During the training process, the loss function value of the above preset neural network model is determined. When the loss function value of the above preset neural network model does not meet the preset training stop condition, the model parameters of the above preset neural network model are adjusted, so as to continuously iterate and optimize the preset neural network model, thereby obtaining a trained preset neural network model that can accurately detect pedestrians.

[0113] In order to obtain an accurate loss function value, in some embodiments, the above S003 includes steps A3 to G3:

[0114] Step A3: According to the positional relationship between the pedestrian prediction identification frame and the pedestrian identification frame label, the area of ​​the overlapping region between the pedestrian prediction identification frame and the pedestrian identification frame label is calculated to obtain a first area.

[0115] In a specific implementation, the position of the pedestrian prediction identification box and the position of the pedestrian identification box label are obtained, and these position information are usually expressed as coordinates or rectangular bounding boxes. Using the coordinate information between the prediction identification box and the label identification box, the position and size of their overlapping area are calculated. This is usually achieved by calculating the intersection area of ​​the two rectangular boxes. The area of ​​the intersection area is calculated according to the position and size of the intersection area to obtain the first area.

[0116] Step B3: subtract the area of ​​the overlapping area from the sum of the areas of the pedestrian prediction identification box and the pedestrian identification box label to obtain a second area.

[0117] In a specific implementation, the area information of the pedestrian prediction identification frame and the pedestrian label identification frame is calculated, and the area of ​​the pedestrian prediction identification frame and the area of ​​the pedestrian label identification frame are added together to obtain their total area. Then, the area of ​​the overlapping area calculated in step A3 is subtracted from the total area, and the obtained area is the second area.

[0118] Step C3: Calculate the ratio of the first area to the second area to obtain a first intersection-and-union ratio.

[0119] In a specific implementation, the first area calculated in step A3 is divided by the second area calculated in step B3 to obtain a first intersection-over-union ratio. The intersection-over-union ratio is used to evaluate the degree of overlap between the predicted box and the true label box.

[0120] Step D3: Based on the position of the pedestrian prediction identification frame and the position of the pedestrian identification frame label, determine the position offset information between the pedestrian prediction identification frame and the pedestrian identification frame label.

[0121] In a specific implementation, the offset between the center of the prediction box and the center of the real label box, or the distance between the prediction box and the label box boundary, is calculated to determine the position offset information between the pedestrian prediction identification box and the pedestrian identification box label. This information describes the position offset of the prediction box relative to the real label box.

[0122] Step E3: Determine the loss value according to the first intersection-over-union ratio and the position offset information.

[0123] In a specific implementation, the loss value may be calculated according to the first intersection-over-union ratio and the position offset information according to a preset rule or formula.

[0124] Step F3: determining the adjustment range corresponding to the first intersection-over-union ratio according to a preset focus factor.

[0125] In the specific implementation, a focus factor is pre-set to adjust the impact of the first intersection-and-union ratio on the final result. This focus factor can be set according to the specific application scenario or task requirements. The adjustment range corresponding to the first intersection-and-union ratio is calculated based on this focus factor, and the size of the adjustment range directly affects the weight of the first intersection-and-union ratio in the loss function.

[0126] Step G3: Determine the loss function value of the preset neural network model based on the first intersection-over-union ratio, the loss value and the adjustment amplitude.

[0127] In the specific implementation, the loss function value of the neural network model is calculated according to the preset loss function formula, combined with the first intersection-over-union ratio, loss value and adjustment amplitude. This value is used to measure the degree of match between the prediction box and the label box, so as to optimize the model parameters during the training process.

[0128] The above implementation method provided in the embodiment of the present application first determines the first intersection-and-union ratio based on the positional relationship between the position of the predicted identification box and the position of the identification box label. Then, the position offset information and relative contour information of the identification box are used to determine the loss value in combination with the first intersection-and-union ratio. The adjustment amplitude corresponding to the above first intersection-and-union ratio is also determined according to the preset attention factor, and finally the loss function value of the above preset neural network model is determined based on the above first intersection-and-union ratio, the loss value and the above adjustment amplitude. Thereby, an accurate loss function value is obtained.

[0129] In order to obtain an adjustment range that accurately corresponds to the first intersection-over-combination ratio, in some implementations, the preset attention factor includes a first attention factor and a second attention factor, the first attention factor is smaller than the second attention factor, and the above F3 includes steps A4 to C4:

[0130] Step A4: when the first intersection-and-union ratio is smaller than the first focus factor, the adjustment range corresponding to the intersection-and-union ratio is determined as a first preset value.

[0131] In a specific implementation, after the first intersection-over-union ratio is calculated, if the first intersection-over-union ratio is smaller than the first focus factor, the adjustment amplitude is determined to be a first preset value.

[0132] Step B4: when the first intersection-and-union ratio is greater than the second focus factor, the adjustment amplitude corresponding to the intersection-and-union ratio is determined as a second preset value, and the first preset value is less than the second preset value.

[0133] In a specific implementation, if the first intersection-and-combination ratio is greater than the second focus factor, a corresponding adjustment is performed, and the adjustment amplitude corresponding to the above intersection-and-combination ratio is determined as the second preset value. Otherwise, proceed to the next step. The first preset value is less than the second preset value.

[0134] Step C4: when the first intersection-and-intersection ratio is greater than the first concern factor and less than the second concern factor, determining an adjustment range corresponding to the first intersection-and-intersection ratio according to a difference between the first concern factor and the second concern factor and the first intersection-and-intersection ratio.

[0135] In a specific implementation, if the first intersection-and-union ratio is between the first focus factor and the second focus factor, the difference between the first focus factor and the second focus factor is calculated, and the adjustment range corresponding to the intersection-and-union ratio is determined according to the difference and the specific value of the first intersection-and-union ratio.

[0136] In the above implementation mode provided in the embodiment of the present application, the preset attention factor is divided into a first attention factor and a second attention factor, and the first intersection-and-union ratio is compared with the first attention factor and the second attention factor. According to the size relationship between the first intersection-and-union ratio and the first attention factor and the second attention factor, the adjustment amplitude corresponding to the first intersection-and-union ratio is determined accordingly, so as to obtain the adjustment amplitude that accurately corresponds to the first intersection-and-union ratio.

[0137] In order to obtain an accurate loss value, in some implementations, the above E3, determining the loss value, includes:

[0138] The loss value is determined by the following formula:

[0139]

[0140] Among them, L DIoU represents the loss value, IoU represents the first intersection-over-union ratio, d represents the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the true box, and c represents the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the true box. Indicates position offset information.

[0141] In the above implementation provided by the embodiment of the present application, the position offset information is calculated using the distance between the reference point of the predicted identification box and the reference point of the identification box label and the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the real box, and the loss value is finally calculated based on the above data. An accurate loss value can be obtained.

[0142] For the purpose of model training, in some implementations, obtaining a training sample set includes:

[0143] Obtain an initial sample image and a labeling file corresponding to the initial sample image.

[0144] In a specific implementation, an initial sample image is read from a specified path or database, and the path or database location of the corresponding annotation file is determined according to the file name or identifier of the initial sample image, and the annotation file is loaded. The annotation file is a file containing target category and location information.

[0145] Data expansion is performed on the initial sample image to obtain a plurality of derivative sample images corresponding to the initial sample image.

[0146] In the specific implementation, appropriate data augmentation methods are selected according to specific tasks and application scenarios, such as rotation, scaling, translation, mirroring, etc. It is also possible to combine the data augmentation functions provided by the preset image processing library or deep learning framework. The selected data augmentation method is applied to each initial sample image to generate multiple derivative sample images.

[0147] Samples are constructed based on the initial sample image and the corresponding annotation file, and the derived image and the corresponding annotation file.

[0148] In the specific implementation, each initial sample image and its corresponding annotation file are used as part of the training sample. At the same time, each derived sample image and its corresponding original annotation file are used as additional training samples. In this way, multiple training samples are constructed.

[0149] According to the above training samples, a training sample set is constructed.

[0150] In the above implementation provided by the embodiment of the present application, after obtaining the initial sample image and its corresponding annotation file, data expansion is performed on the initial sample image, and then the initial sample image and the corresponding annotation file, as well as the derived image and the corresponding annotation file are respectively combined to obtain training samples and form a training sample set. In this way, more training samples are obtained to improve the effect of model training.

[0151] The data expansion of the initial sample image includes at least one of flipping the initial sample image, transforming the color space of the initial sample image, and adding noise to the initial sample image.

[0152] After obtaining the preset neural network model for pedestrian detection and the image to be detected, the method further includes:

[0153] The image to be detected is subjected to noise reduction and the contrast of the image to be detected is enhanced.

[0154] In the specific implementation, common image processing techniques such as Gaussian filtering, median filtering or wavelet transform are used to reduce noise on the image to be detected. Then, histogram equalization, contrast stretching or adaptive contrast enhancement are used to enhance the contrast of the image to be detected.

[0155] The ratio of the aspect ratio of the image to be detected to the target aspect ratio is calculated to obtain the scaling ratio.

[0156] In a specific implementation, the aspect ratio that the image to be detected should achieve is determined according to a predefined target, and the corresponding zoom ratio of the image is calculated according to the current size of the image to be detected and the aspect ratio of the target.

[0157] The image to be detected is scaled according to the above scaling ratio.

[0158] In a specific implementation, the image to be detected is scaled according to the scaling ratio calculated previously.

[0159] The above-mentioned image to be detected is input into the above-mentioned preset neural network model, and features of the above-mentioned image to be detected are extracted through the above-mentioned variable kernel convolution and other modules to obtain image feature information, including:

[0160] The image to be detected after the scaling is input into the preset neural network model, and features of the image to be detected are extracted through the variable kernel convolution and other modules to obtain image feature information.

[0161] In one embodiment of the present application, a pedestrian detection image to be identified is first obtained, and then image preprocessing is performed to reduce noise, enhance contrast, and scale the input image to a size of 640*640. The processed image is input into a pre-built improved single-shot recognition (You Only Look Once, YOLO), specifically, a YOLOv8 pedestrian detection algorithm model is used to obtain pedestrians in the image to be identified.

[0162] In the pedestrian detection algorithm model, the lightweight architecture variable kernel convolution (AKConv) is used to replace the convolution operation of the backbone network in the YOLOv8 model to enhance the effective feature extraction of the changing target. The lightweight processing can be omitted in the first convolution kernel to ensure that the improvement of the entire backbone network will not lose accuracy, thereby achieving the purpose of lightweight structure. The C2f module (C2f_PConv) improved by partial convolution is used to replace the channel module (Cross-stage Partial Bottleneck with two convolutions, C2F) structure of the neck part of the YOLOv8 model, achieving lightweight while improving network performance.

[0163] The above pedestrian detection algorithm model uses the linear interval mapping intersection-over-union (Focaler-DIou) loss during training. Specifically, in the process of calculating the Focaler-DIou loss, first calculate:

[0164]

[0165] Among them, B represents the predicted annotation box, B gt represents the true annotation box, and IoU is the first intersection over union ratio. At the same time, calculate:

[0166]

[0167] Among them, c is the distance between the diagonal of the smallest circumscribed rectangle of the predicted box and the real box, and d is the distance between the center points of the predicted box and the real box. Then calculate the loss value L DIoU :

[0168] L DIoU =1-DIOU

[0169] After the loss value is obtained, the original IoU loss is reconstructed through linear interval mapping to achieve the purpose of focusing on difficult and easy samples. In the specific calculation process, first calculate:

[0170]

[0171] Among them, [d,u]∈[0,1], by adjusting the values ​​of d and u, IoU can focus more on different regression samples.

[0172] The final Focaler-DIou loss is:

[0173] L Focaler-DIoU =L DIoU +IoU-IoU Focaler

[0174] This loss function is used to train the model.

[0175] In addition, partial convolution (PConv) can be used to improve the C2f module, while reducing redundancy and memory access, and extracting spatial features more efficiently. In order to adapt to YOLOv8, a convolution layer with a convolution kernel size of 1 is added to the original PConv structure. The modified PConv is used to replace the second convolution in the backbone network structure in C2f, thereby obtaining the improved C2f_PConv module, which can better integrate feature information of various scales in the neck network in the pedestrian detection model and is lightweight.

[0176] YOLO usually adopts a Backbone-Neck-Head network structure. The backbone network is mainly responsible for extracting high-level features from the input image, usually including multiple convolutional layers and pooling layers, to build a deep feature extractor; the neck network is usually used to further integrate and adjust the features extracted by the backbone network, which is conducive to the fusion of features at different levels and thus improve the network's perception of the target; the head network usually includes a bounding box regression layer and a classification layer to perform the final regression prediction, and then obtain the final pedestrian detection result.

[0177] In the field of autonomous driving, the accuracy and performance of pedestrian detection results directly affect the reliability and safety of the system. For example, pedestrian detection. Pedestrians are non-rigid bodies and can be understood as changing targets. Traditional convolution operations are limited to local windows and cannot capture information at other locations, and their sampling shapes are fixed. Moreover, the size of the convolution kernel is fixed to k×k, which is a fixed square shape, and the number of parameters increases quadratically with the size. Generally, fixed convolution kernels cannot adapt well to changing targets. To address this problem, an adjustable kernel convolution (AKConv) can be used. AKConv can be used as a plug-and-play convolution to improve the performance of the network. The main idea of ​​a pedestrian detection method in the field of autonomous driving based on the improved YOLOv8 is to replace part of the convolution operations in the YOLOv8 model with the lightweight architecture AKConv, and replace part of the C2f in the model with the improved C2f_PConv, so as to achieve lightweight while improving network performance, so that the final pedestrian detection effect and robust performance have been greatly improved.

[0178] In the specific method, the lightweight architecture AKConv is used to replace some convolution operations in the backbone network of the YOLOv8 model, and the first convolution kernel is not replaced to enhance the effective feature extraction of the changing target. The first Conv is not lightweighted to ensure that the improvement of the entire backbone network will not lose accuracy, thereby achieving the purpose of lightweight structure. The partial convolution PConv is used to improve the C2f module, while reducing redundancy and memory access, and extracting spatial features more effectively. In order to adapt to YOLOv8, a convolution layer with a convolution kernel size of 1 is added to the original PConv structure. The modified PConv is used to replace the second convolution in the backbone network structure in C2f, thereby obtaining the improved C2f_PConv module, which can better integrate the feature information of various scales in the neck network in the pedestrian detection model and is lightweight. Finally, the improved C2f_PConv is used to replace the C2f structure of the neck network part of the YOLOv8 model, achieving lightweight while improving network performance.

[0179] For details, please refer to Figure 3The input of the model is a 640*640 color image, which is first processed by the 301 backbone network. The convolution in the 301 backbone network is a variable kernel convolution. After the convolution, it passes through the C2f module and the SPPF module. The C2f module contains skip layer connections and additional split operations. Then it is handed over to the 302 neck network to fuse the results of different layers, including the partial channel (C2f_PConv) module obtained by using partial convolution PConv to improve C2f. Finally, the decoupling and head in the 303 head network are used to separate the regression branch from the prediction branch, and the detection result of the model is output. The detection result includes the category and location information of the target.

[0180] The lightweight architecture AKConv is used to replace the CONv operation in the backbone network part of the model, and the improved C2f_PConv is used to replace the C2f in the neck network part of the model. The two are combined to take into account the performance advantages of YOLO itself and the advantages of AKConv and C2f_PConv to complete feature extraction and fusion, making the model more lightweight and improving the final pedestrian detection effect and performance.

[0181] Based on the pedestrian detection method provided in the above embodiment, the present application also provides a specific implementation of a pedestrian detection device. Please refer to the following embodiment.

[0182] See first Figure 4 The pedestrian detection device 400 provided in the embodiment of the present application includes the following modules:

[0183] The acquisition module 401 is used to acquire a preset neural network model for pedestrian detection and an image to be detected, wherein the preset neural network model includes a backbone network, a neck network and a head network, and the backbone network includes a variable kernel convolution.

[0184] The input module 402 is used to input the above-mentioned image to be detected into the above-mentioned backbone network, and extract features of the above-mentioned image to be detected through the above-mentioned variable kernel convolution and other modules to obtain image feature information.

[0185] The detection module 403 is used to perform pedestrian detection on the above-mentioned image to be detected based on the above-mentioned image feature information through the neck network and the head network in the above-mentioned preset neural network model to obtain a pedestrian detection result.

[0186] The pedestrian detection device provided in the embodiment of the present application performs feature extraction through modules such as variable kernel convolution. The variable kernel convolution can adaptively adjust the size and shape of the convolution kernel. This adaptability enables the preset neural network model to more accurately capture different parts of the target even if the target presents different sizes or directions in the image, thereby being able to better adapt to the changing characteristics of non-rigid targets such as pedestrians. The preset neural network model can more effectively extract features, thereby improving the performance of pedestrian detection and improving the accuracy of pedestrian detection.

[0187] As an implementation of the present application, the detection module 403 may include:

[0188] The convolution unit is used to perform partial convolution and feature fusion on the above-mentioned image feature information through the above-mentioned neck network to obtain the image feature information after feature fusion.

[0189] The detection unit is used to perform pedestrian detection on the above-mentioned image to be detected based on the image feature information after the above-mentioned feature fusion through the above-mentioned head network to obtain a pedestrian detection result.

[0190] In the implementation method provided in the embodiment of the present application, the neck network is used to perform partial convolution on the image feature information to obtain the image feature information after feature fusion. Then, the head network is used to perform pedestrian detection on the image to be detected based on the image feature information after feature fusion to obtain the pedestrian detection result. Since partial convolution is used in the neck network, the processing speed of pedestrian detection is accelerated.

[0191] As an implementation of the present application, the convolution unit may include:

[0192] The determination subunit is used to determine the target data point in the above image feature information according to a preset mask.

[0193] The convolution subunit is used to perform point-by-point convolution on the target data point through the neck network to obtain a first convolution result.

[0194] The convolution subunit is further used to perform partial convolution on the first convolution result to obtain image feature information after feature fusion.

[0195] In the implementation method provided by the embodiment of the present application, after determining the target data point in the image feature information according to the preset mask, the neck network performs point-by-point convolution on the target data point, and finally obtains the first convolution result. The first convolution result is then partially convolved to obtain the image feature information after feature fusion. Through point-by-point convolution, more accurate image feature information can be obtained.

[0196] As an implementation of the present application, the pedestrian detection device 400 may further include:

[0197] The acquisition module is used to acquire a training sample set; the training sample set includes multiple training samples, each of which includes a sample image and a labeling file corresponding to the sample image, and the labeling file includes a pedestrian identification box label in the sample image.

[0198] The determination module is used to input the above sample image into a preset neural network model to determine the prediction detection result of the pedestrian position in the above sample image, and the above prediction detection result includes a pedestrian prediction identification box.

[0199] The determination module is also used to determine the loss function value of the above-mentioned preset neural network model based on the above-mentioned pedestrian prediction identification box and the corresponding pedestrian identification box label of each of the above-mentioned training samples.

[0200] The adjustment module is also used to adjust the model parameters of the above-mentioned preset neural network model when the loss function value of the above-mentioned preset neural network model does not meet the preset training stop conditions, return to the step of inputting the above-mentioned sample image into the preset neural network model, and determine the prediction detection result of predicting the pedestrian position in the above-mentioned sample image.

[0201] The stopping module is used to stop training when the loss function value of the above-mentioned preset neural network model meets the preset training stopping condition to obtain the trained preset neural network model.

[0202] The above implementation method provided in the embodiment of the present application obtains a training sample set and uses the training sample set to train the preset neural network model. During the training process, the loss function value of the above preset neural network model is determined. When the loss function value of the above preset neural network model does not meet the preset training stop condition, the model parameters of the above preset neural network model are adjusted, so as to continuously iterate and optimize the preset neural network model, thereby obtaining a trained preset neural network model that can accurately detect pedestrians.

[0203] As an implementation of the present application, the determination module may include:

[0204] The calculation unit is used to calculate the area of ​​the overlapping area between the pedestrian prediction identification frame and the pedestrian identification frame label according to the positional relationship between the pedestrian prediction identification frame and the pedestrian identification frame label to obtain a first area.

[0205] The subtraction unit is used to subtract the area of ​​the overlapping area from the sum of the areas of the pedestrian prediction identification frame and the pedestrian identification frame label to obtain a second area.

[0206] The calculation unit is further used to calculate the ratio of the first area to the second area to obtain a first intersection-and-union ratio.

[0207] The determination unit is used to determine the position offset information between the pedestrian prediction identification frame and the pedestrian identification frame label based on the position of the pedestrian prediction identification frame and the position of the pedestrian identification frame label.

[0208] The determination unit is further used to determine the loss value according to the first intersection-over-union ratio and the position offset information.

[0209] The determination unit is further used to determine the adjustment range corresponding to the first intersection-and-combination ratio according to a preset focus factor.

[0210] The determination unit is also used to determine the loss function value of the preset neural network model based on the first intersection-and-union ratio, the loss value and the adjustment amplitude.

[0211] The above implementation method provided in the embodiment of the present application first determines the first intersection-and-union ratio based on the positional relationship between the position of the predicted identification box and the position of the identification box label. Then, the position offset information and relative contour information of the identification box are used to determine the loss value in combination with the first intersection-and-union ratio. The adjustment amplitude corresponding to the above first intersection-and-union ratio is also determined according to the preset attention factor, and finally the loss function value of the above preset neural network model is determined based on the above first intersection-and-union ratio, the loss value and the above adjustment amplitude. Thereby, an accurate loss function value is obtained.

[0212] As an implementation of the present application, the determining unit may include:

[0213] The determination subunit is used to determine the adjustment amplitude corresponding to the above-mentioned intersection-and-union ratio as a first preset value when the above-mentioned first intersection-and-union ratio is smaller than the above-mentioned first focus factor.

[0214] The determination subunit is further configured to determine the adjustment amplitude corresponding to the above-mentioned intersection-and-union ratio as a second preset value when the above-mentioned first intersection-and-union ratio is greater than the above-mentioned second focus factor, and the above-mentioned first preset value is less than the second preset value.

[0215] The determination subunit is further used to determine the adjustment amplitude corresponding to the first intersection-and-union ratio according to the difference between the first focus factor and the second focus factor and the first intersection-and-union ratio when the first intersection-and-union ratio is greater than the first focus factor and less than the second focus factor.

[0216] In the above implementation mode provided in the embodiment of the present application, the preset attention factor is divided into a first attention factor and a second attention factor, and the first intersection-and-union ratio is compared with the first attention factor and the second attention factor. According to the size relationship between the first intersection-and-union ratio and the first attention factor and the second attention factor, the adjustment amplitude corresponding to the first intersection-and-union ratio is determined accordingly, so as to obtain the adjustment amplitude that accurately corresponds to the first intersection-and-union ratio.

[0217] As an implementation of the present application, the determining unit may include:

[0218] Determine the subunits for determining the loss value using the following formula:

[0219] The loss value is determined by the following formula:

[0220]

[0221] Among them, L DIoU represents the DIoU loss value, IoU represents the first intersection-over-union ratio, d is the distance between the reference point of the predicted identification box and the reference point of the identification box label, and c is the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the true box. Indicates position offset information.

[0222] In the above implementation provided by the embodiment of the present application, the position offset information is calculated using the distance between the reference point of the predicted identification box and the reference point of the identification box label and the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the real box, and the loss value is finally calculated based on the above data. An accurate loss value can be obtained.

[0223] Each module in the pedestrian detection device provided in the embodiment of the present application can implement each step in the above-mentioned pedestrian detection method and achieve the corresponding effect. For the sake of concise description, it will not be repeated here.

[0224] Figure 5 A schematic diagram of the structure of the pedestrian detection hardware provided in an embodiment of the present application is shown.

[0225] The pedestrian detection device may include a processor 501 and a memory 502 storing computer program instructions.

[0226] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0227] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0228] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the pedestrian detection method according to any one embodiment of the present disclosure.

[0229] The processor 501 implements any one of the pedestrian detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 502 .

[0230] In one example, the pedestrian detection device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0231] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0232] Bus 510 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0233] In addition, in combination with the pedestrian detection method in the above embodiments, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any pedestrian detection method in the above embodiments is implemented.

[0234] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements any one of the pedestrian detection methods in the above embodiments.

[0235] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0236] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0237] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0238] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0239] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A pedestrian detection method, characterized in that: include: Acquire a preset neural network model for pedestrian detection and an image to be detected, wherein the preset neural network model includes a backbone network, a neck network, and a head network, and the backbone network includes a variable kernel convolution; Inputting the image to be detected into the backbone network, extracting features of the image to be detected through the backbone network to obtain image feature information, wherein variable kernel convolution is used in the backbone network; Pedestrian detection is performed on the image to be detected based on the image feature information through the neck network and the head network in the preset neural network model to obtain a pedestrian detection result.

2. The pedestrian detection method according to claim 1, characterized in that: The performing pedestrian detection on the image to be detected based on the image feature information by using the neck network and the head network in the preset neural network model to obtain a pedestrian detection result includes: Performing partial convolution and feature fusion on the image feature information through the neck network to obtain image feature information after feature fusion; The head network performs pedestrian detection on the image to be detected based on the image feature information after the feature fusion to obtain a pedestrian detection result.

3. The pedestrian detection method according to claim 2, characterized in that: The performing partial convolution and feature fusion on the image feature information through the neck network to obtain the image feature information after feature fusion includes: Determining target data points in the image feature information according to a preset mask; Performing point-by-point convolution on the target data point through the neck network to obtain a first convolution result; Partial convolution and feature fusion are performed on the first convolution result to obtain image feature information after feature fusion.

4. The pedestrian detection method according to claim 1, characterized in that: Before obtaining the preset neural network model for pedestrian detection and the image to be detected, the method further includes: Acquire a training sample set; the training sample set includes a plurality of training samples, each of the training samples includes a sample image and a labeling file corresponding to the sample image, and the labeling file includes a pedestrian identification box label in the sample image; For each training sample, perform the following steps: Inputting the sample image into a preset neural network model to determine a prediction detection result of predicting a pedestrian position in the sample image, wherein the prediction detection result includes a pedestrian prediction identification box; Determining the loss function value of the preset neural network model according to the pedestrian prediction identification frame and the corresponding pedestrian identification frame label of each training sample; When the loss function value of the preset neural network model does not satisfy the preset training stop condition, the model parameters of the preset neural network model are adjusted, and the sample image is input into the preset neural network model to return to the step of determining the prediction detection result of predicting the position of the pedestrian in the sample image; When the loss function value of the preset neural network model meets the preset training stop condition, the training is stopped to obtain the trained preset neural network model.

5. The pedestrian detection method according to claim 4, characterized in that: The determining the loss function value of the preset neural network model according to the pedestrian prediction identification frame and the corresponding pedestrian identification frame label of each training sample includes: Calculating the area of ​​an overlapping region between the pedestrian prediction identification frame and the pedestrian identification frame label according to the positional relationship between the pedestrian prediction identification frame and the pedestrian identification frame label to obtain a first area; Subtract the area of ​​the overlapping area from the sum of the areas of the pedestrian prediction identification frame and the pedestrian identification frame label to obtain a second area; Calculating a ratio of the first area to the second area to obtain a first intersection-and-union ratio; Determining position offset information between the pedestrian prediction identification frame and the pedestrian identification frame label based on the position of the pedestrian prediction identification frame and the position of the pedestrian identification frame label; Determine a DIou loss value according to the first intersection-over-union ratio and the position offset information; Determining an adjustment range corresponding to the first intersection-and-combination ratio according to a preset attention factor; The loss function value of the preset neural network model is determined based on the first intersection-and-union ratio, the DIou loss value and the adjustment amplitude.

6. The pedestrian detection method according to claim 5, characterized in that: The preset attention factor includes a first attention factor and a second attention factor, the first attention factor is smaller than the second attention factor, and the determining the adjustment range corresponding to the first intersection-over-combination ratio according to the preset attention factor includes: When the first intersection-and-union ratio is less than the first focus factor, determining the adjustment amplitude corresponding to the intersection-and-union ratio as a first preset value; When the first intersection-and-union ratio is greater than the second focus factor, the adjustment amplitude corresponding to the intersection-and-union ratio is determined to be a second preset value, and the first preset value is less than the second preset value; When the first IoU is greater than the first focus factor and less than the second focus factor, an adjustment amplitude corresponding to the first IoU is determined according to a difference between the first focus factor and the second focus factor and the first IoU.

7. The pedestrian detection method according to claim 5, characterized in that: The determining the loss value according to the first intersection-over-union ratio and the position offset information includes: The loss value is determined by the following formula: Among them, L DIoU represents the loss value, IoU represents the first intersection-over-union ratio, d is the distance between the reference point of the predicted identification box and the reference point of the identification box label, and c is the distance between the predicted box and the diagonal of the minimum circumscribed rectangle of the real box. Indicates position offset information.

8. A pedestrian detection device, characterized in that: The device comprises: An acquisition module, used to acquire a preset neural network model for pedestrian detection and an image to be detected, wherein the preset neural network model includes a backbone network, a neck network and a head network, and the backbone network includes a variable kernel convolution; An input module, used for inputting the image to be detected into the backbone network, performing feature extraction on the image to be detected through the backbone network to obtain image feature information, wherein variable kernel convolution is used in the backbone network; The detection module is used to perform pedestrian detection on the image to be detected based on the image feature information through the neck network and the head network in the preset neural network model to obtain a pedestrian detection result.

9. A pedestrian detection device, characterized in that: The device comprises: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the pedestrian detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the pedestrian detection method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the pedestrian detection method as described in any one of claims 1 to 7.

Images