Pedestrian detection method, device, computer readable storage medium and chip

Abstract

The application provides a pedestrian detection method and device, a computer readable storage medium and a chip. It relates to the field of artificial intelligence, in particular to the field of computer vision. The method comprises the following steps: performing feature extraction on an image to obtain a basic feature map of the image; determining a candidate box of a possible pedestrian area according to the basic feature map; processing the basic feature map of the image to obtain an object visibility map, in which the response to the visible part of the pedestrian is greater than the response to the occluded part of the pedestrian and the background part; next, performing weighted summation processing on the basic feature map and the object visibility map to obtain an enhanced feature map of the image; finally, determining a bounding box of the pedestrian in the image and the confidence of the bounding box of the pedestrian in the image according to the candidate box of the image and the enhanced feature map of the image. The application can improve the accuracy of pedestrian detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision in the field of artificial intelligence, and more particularly, to a pedestrian detection method and device, a computer readable storage medium and a chip. BACKGROUND

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military fields. It is a science about how to use cameras and computers to obtain the data and information of the object being photographed. In a figurative sense, it is to install eyes (cameras) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Because perception can be regarded as extracting information from sensory signals, computer vision can also be regarded as a science that studies how artificial systems "perceive" from images or multidimensional data. In general, computer vision is to use various imaging systems to replace visual organs to obtain input information, and then use computers to replace the brain to process and interpret the input information. The ultimate goal of computer vision is to enable computers to observe and understand the world through vision like humans, with the ability to adapt to the environment autonomously.

[0003] In the field of computer vision, pedestrian detection is an important research direction, and pedestrian detection has important applications in many fields and scenarios. For example, in advanced driving assistant systems (ADAS) and autonomous driving systems (ADS), dynamic obstacles such as pedestrians on the road are detected and avoided; in a safe city and video monitoring, pedestrians are detected to find out criminal suspects or track missing persons; in a smart home system, the movement and obstacle avoidance of robots are realized by detecting pedestrians.

[0004] The traditional scheme performs poorly in pedestrian detection when the pedestrian is occluded, mainly in the form of missed detection and false detection of the occluded pedestrian. SUMMARY

[0005] The present application provides a pedestrian detection method, device, computer readable storage medium and chip to improve the accuracy of pedestrian detection, especially for occluded pedestrians.

[0006] In a first aspect, a pedestrian detection method is provided. The method comprises: obtaining an image; performing feature extraction on the image to obtain a base feature map of the image; determining a candidate box in which a pedestrian may exist in the image according to the base feature map; processing the base feature map of the image to obtain an object visibility map of the image; performing fusion processing on the base feature map of the image and the object visibility map of the image to obtain an enhanced feature map of the image; and determining a bounding box in which a pedestrian exists in the image and a confidence level of the bounding box in which the pedestrian exists in the image according to the candidate box and the enhanced feature map of the image.

[0007] The image can be an image containing a pedestrian.

[0008] Optionally, the image is obtained by photographing through a camera.

[0009] For example, for a mobile phone, the image can be obtained by photographing.

[0010] Optionally, the image is obtained from a memory.

[0011] For example, for a mobile phone, the image can be selected from an album.

[0012] In the process of obtaining the base feature map of the image, the base feature map can be obtained by performing convolution operation on the image, or by performing further processing (such as summation, weighting processing, connection, etc.) on the result of the convolution operation on the image.

[0013] The candidate box is a bounding box of a region in which a pedestrian may exist in the image. The region in which the candidate box is located can be a region enclosed by the candidate box (a region inside the candidate box), and the region in which the candidate box is located is the region in which the pedestrian may exist in the image.

[0014] The object visibility map of the image has different response degrees to different objects. In the object visibility map of the image, the response degree to the visible part of the pedestrian is greater than the response degree to the invisible part of the pedestrian. That is, in the object visibility map of the image, the feature of the visible part of the pedestrian is more prominent than the invisible part of the pedestrian. Specifically, in the object visibility map of the image, the pixel value of the visible part of the pedestrian is greater than the pixel value of the invisible part of the pedestrian.

[0015] The object visibility map of the image can more prominently reflect the feature of the visible part of the pedestrian.

[0016] In addition, in the object visibility map of the image, the visible part of the pedestrian is the part in which the image of the pedestrian can be seen, and the invisible part of the pedestrian is the part in which the image of the pedestrian cannot be seen.

[0017] The region feature of the candidate box can be the part of the region in the enhanced feature map corresponding to the candidate box. When determining the region feature of the candidate box, the position of the candidate box in the enhanced feature map can be determined first, and then the region feature of the candidate box can be determined as the feature of the region surrounded by the candidate box in the enhanced feature map.

[0018] The region feature of the candidate box can be the part of the region in the enhanced feature map corresponding to the candidate box. When determining the region feature of the candidate box, the position of the candidate box in the enhanced feature map can be determined first, and then the region feature of the candidate box can be determined as the feature of the region surrounded by the candidate box in the enhanced feature map.

[0019] Further, the feature of the region surrounded by the candidate box in the enhanced feature map can be sampled (which can be up-sampling or down-sampling) to obtain the region feature of the candidate box.

[0020] The bounding box of the pedestrian in the image and the confidence of the bounding box of the pedestrian in the image can be the detection result of the pedestrian detection on the image, which can be referred to as the pedestrian detection result of the image.

[0021] In the present application, the object visibility map of the image can more prominently reflect the features of the visible part of the pedestrian, and the enhanced feature map obtained by fusing the base feature map of the image and the object visibility map of the image prominently reflects the features of the visible part of the pedestrian, which can improve the accuracy of subsequent pedestrian detection based on the enhanced feature map. Moreover, the accuracy of the pedestrian detection of the present application is significantly improved for the case of pedestrian occlusion (more serious).

[0022] Further, in the present application, the object visibility map is combined to improve the accuracy of pedestrian detection without increasing the annotation amount of training data in the training process. The present application only needs to generate the object visibility map in the processing process and consider the object visibility map in the subsequent processing. Compared with the scheme of improving the accuracy of pedestrian detection by increasing the annotation amount of data, the present application can save the annotation amount of data and reduce the complexity of training.

[0023] Optionally, in the object visibility map of the image, the invisible part of the pedestrian includes the occluded part of the pedestrian.

[0024] When the pedestrian invisible part includes the pedestrian occluded part, the pedestrian visible part and the pedestrian occluded part can be distinguished in the object visibility map, and since the pixel value of the pedestrian visible part is greater than the pixel value of the pedestrian occluded part, the characteristics of the pedestrian visible part can be highlighted in the object visibility, the characteristics of the pedestrian occluded part can be weakened, the influence of the pedestrian occluded part on the detection result in the subsequent pedestrian detection process can be reduced, and the characteristics of the pedestrian visible part can be highlighted when subsequent pedestrian detection is performed, thereby improving the pedestrian detection effect.

[0025] Optionally, in the object visibility map of the image, the pedestrian invisible part includes a background part of the image.

[0026] The background part of the image can refer to other parts of the image except the pedestrian, or the background part of the image can also refer to other parts of the image except the pedestrian and main objects (for example, a car).

[0027] When the pedestrian invisible part includes the background part of the image, the pedestrian and the background part can be distinguished in the object visibility map, the characteristics of the pedestrian visible part can be highlighted, the characteristics of the background part can be weakened, the influence of the background part on the detection result in the subsequent pedestrian detection process can be reduced, and the characteristics of the pedestrian visible part can be highlighted when subsequent pedestrian detection is performed, thereby improving the pedestrian detection effect.

[0028] With reference to the first aspect, in some implementations of the first aspect, the processing of the base feature map of the image to obtain the object visibility map of the image includes: performing convolution processing on the base feature map of the image by using a first convolutional network to obtain a plurality of first semantic feature maps; and performing weighted sum processing on the plurality of first semantic feature maps to obtain the object visibility map of the image; and the determining of the bounding box in which the pedestrian exists in the image and the confidence of the bounding box in which the pedestrian exists in the image according to the features corresponding to the candidate box includes: performing convolution processing on the features corresponding to the candidate box by using a second convolutional network to obtain a plurality of second semantic feature maps; performing processing on the plurality of second semantic feature maps by using a regressor to determine the position of the bounding box; and performing processing on the plurality of second semantic feature maps by using a classifier to obtain the confidence of the bounding box in which the pedestrian exists in the image.

[0029] The plurality of first semantic feature maps are a plurality of feature maps of different semantics extracted from the base feature map full map. Specifically, in the plurality of first semantic feature maps, any two first semantic feature maps correspond to different semantics.

[0030] The plurality of second semantic feature maps respectively represent a plurality of different semantic feature maps extracted from the features corresponding to the candidate box, and the convolution parameters of the second convolutional network are the same as the convolution parameters of the first convolutional network. Specifically, in the plurality of second semantic feature maps, any two second semantic feature maps correspond to different semantics.

[0031] The same convolution parameters of the first convolutional network and the second convolutional network can mean that the convolution kernel parameters of the first convolutional network and the second convolutional network are the same. Further, the same convolution parameters of the first convolutional network and the second convolutional network can also mean that the network architecture and the convolution kernel parameters of the first convolutional network and the second convolutional network are completely the same, and the same image semantic features can be extracted when the first convolutional network and the second convolutional network are used to extract features from the same image.

[0032] The weighting coefficients in the weighted sum processing of the plurality of first semantic features are weight coefficients used to determine the pedestrian score in the classifier.

[0033] In this application, since the convolution parameters of the first convolutional network and the second convolutional network are the same, and the weighting coefficients in the weighted sum processing of the plurality of first semantic features are weight coefficients used to determine the pedestrian score in the classifier, the object visibility map that can highlight the visible part of the pedestrian can be obtained through the processing of the plurality of first semantic features, which facilitates more accurate pedestrian detection according to the object visibility map.

[0034] In combination with the first aspect, in some implementations of the first aspect, the features corresponding to the candidate box further include region features of a contour region of the candidate box, and the contour region of the candidate box is a region formed between the candidate box and a reduced candidate box obtained by reducing the candidate box according to a first preset ratio.

[0035] The reduction of the candidate box according to the first preset ratio can specifically mean that the width and the height of the candidate box are reduced according to a certain ratio, and the ratio of the reduction of the width and the height of the candidate box can be the same or different.

[0036] Optionally, the reduction of the candidate box according to the first preset ratio includes that the width of the candidate box is reduced according to a first reduction ratio, and the height of the candidate box is reduced according to a second reduction ratio.

[0037] The first preset ratio includes the first reduction ratio and the second reduction ratio, and the first reduction ratio and the second reduction ratio can be the same or different.

[0038] The first and second reduction ratios can be set empirically. For example, appropriate values can be set for the first and second reduction ratios so that the contour area of the candidate box after the width and height of the candidate box are reduced according to the first and second reduction ratios can better extract the contour features of the pedestrian.

[0039] The first and second reduction ratios can be set to values that can better extract the contour of the pedestrian.

[0040] The first reduction ratio can be 1 / 1.1, and the second reduction ratio can be 1 / 1.8.

[0041] The area features of the contour area of the candidate box generally include the contour features of the pedestrian, and the contour features of the pedestrian also play an important role in pedestrian detection.

[0042] In this application, when the features corresponding to the candidate box also include the area features of the contour area of the candidate box, the contour features of the pedestrian can also be considered in pedestrian detection, which facilitates subsequent comprehensive pedestrian contour features to better perform pedestrian detection.

[0043] In combination with the first aspect, in some implementations of the first aspect, the method further includes: setting the values of the features of the area features of the candidate box located in the reduced candidate box to zero to obtain the area features of the contour area of the candidate box.

[0044] In this application, when obtaining the area features of the contour area of the candidate box, the area features of the contour area of the candidate box can be quickly and conveniently obtained by directly setting the values of the features of the area features of the candidate box located in the reduced candidate box to zero.

[0045] It should be understood that in this application, other ways can also be used to obtain the area features of the contour area of the candidate box, for example, the area features of the contour area of the candidate box can be directly scanned from the enhanced feature map according to the area position of the contour area of the candidate box.

[0046] In combination with the first aspect, in some implementations of the first aspect, the features corresponding to the candidate box further include the area features of the background area of the candidate box, and the background area of the candidate box is the area formed between the candidate box and an enlarged candidate box obtained by enlarging the candidate box according to a second preset ratio.

[0047] The enlargement of the candidate box according to the second preset ratio can specifically mean that the width and height of the candidate box are enlarged according to a certain ratio, and the ratio when the width and height of the candidate box are enlarged can be the same or different.

[0048] Optionally, the candidate box is enlarged according to a second preset scale, including: the width of the candidate box is enlarged according to a first enlargement scale, and the height of the candidate box is enlarged according to a second enlargement scale.

[0049] The second preset scale includes the first enlargement scale and the second enlargement scale, and the first enlargement scale and the second enlargement scale can be the same or different.

[0050] The first enlargement scale and the second enlargement scale can be set according to experience. For example, appropriate numerical values can be set for the first enlargement scale and the second enlargement scale, so that the background area of the candidate box obtained by enlarging the width and height of the candidate box according to the first enlargement scale and the second enlargement scale can better extract the background features around the pedestrian.

[0051] The first enlargement scale can be 1.1, and the second enlargement scale can be 1.8.

[0052] The region features of the background area of the candidate box generally reflect the features of the background area in which the pedestrian in the image is located. The features of the background area can be combined with the features of the pedestrian for pedestrian detection.

[0053] In the present application, when the features corresponding to the candidate box further include the region features of the background area of the candidate box, the region features of the background area are also considered when performing pedestrian detection, which facilitates subsequent comprehensive background region features for better pedestrian detection.

[0054] In combination with the first aspect, in some implementations of the first aspect, the method further includes: obtaining region features of a first region, the region features of the first region being region features in the object visibility map located in the region of the expanded candidate box; and setting to zero the features in the region features of the first region located in the candidate box to obtain the region features of the background area of the candidate box.

[0055] In the present application, when obtaining the region features of the background area of the candidate box, the region features of the candidate box in the enhanced feature map are obtained, and then the features in the region features located in the candidate box are set to zero, so that the region features of the contour region of the candidate box can be quickly and conveniently obtained.

[0056] Optionally, the method is performed by a neural network (model).

[0057] Specifically, in the above method, after the image is obtained, the image can be processed by using the neural network (model), and finally the bounding box in which the pedestrian exists in the image and the confidence of the bounding box in which the pedestrian exists in the image are determined according to the features corresponding to the candidate box.

[0058] In a second aspect, a method for training a neural network is provided. The method comprises: obtaining training data, the training data comprising training images and a pedestrian annotation result of the training images; performing the following processing on the training images by the neural network: performing convolution processing on the training images to obtain a base feature map of the training images; determining a candidate box in which a pedestrian may exist in the training images according to the base feature map; performing processing on the base feature map of the training images to obtain an object visibility map of the training images; performing fusion processing on the base feature map of the training images and the object visibility map of the training images to obtain an enhanced feature map of the training images; determining a feature corresponding to the candidate box according to the candidate box and the enhanced feature map of the training images; determining a pedestrian detection result of the training images according to the feature corresponding to the candidate box; and determining a loss value of the neural network according to the pedestrian detection result of the training images and the pedestrian annotation result of the training images, and then adjusting the neural network through back propagation according to the loss value.

[0059] The candidate box is a surrounding box of a region in which a pedestrian may exist in the training image. The region in which the candidate box is located can be a region surrounded by the candidate box (a region inside the candidate box), and the region in which the candidate box is located is the region in which the pedestrian may exist in the training image.

[0060] In addition, in the object visibility map of the training image, the visible part of the pedestrian is a part in which the image of the pedestrian can be seen, and the invisible part of the pedestrian is a part in which the image of the pedestrian cannot be seen.

[0061] The feature corresponding to the candidate box can comprise a region feature of the candidate box, the region feature of the candidate box being a feature of a region in the enhanced feature map located in the candidate box.

[0062] The pedestrian detection result of the training image can comprise a surrounding box of the pedestrian existing in the training image and a confidence of the surrounding box of the pedestrian existing in the image.

[0063] The pedestrian detection annotation result of the training image comprises a surrounding box of the pedestrian existing in the training image.

[0064] The pedestrian detection annotation result of the training image can be pre-labeled (specifically, labeled by a human).

[0065] In addition, in the training process, the training images used are generally multiple.

[0066] In the process of training the neural network, a set of initial model parameters can be set for the neural network, and then the model parameters of the neural network are gradually adjusted according to the difference between the pedestrian detection annotation result of the training image and the pedestrian detection result of the training image, until the difference between the pedestrian detection result of the training image and the pedestrian detection annotation result of the training image is within a certain preset range, or when the number of training reaches a preset number, the model parameters of the neural network at this time are determined as the final parameters of the neural network model, so that the training of the neural network is completed.

[0067] It should be understood that the neural network trained by the method of the second aspect described above can be used to execute the method of the first aspect of the present application.

[0068] It should be understood that in the present application, the base feature map, the candidate box, the object visibility map, the enhanced feature map, the bounding box of the image in which the pedestrian exists, and the confidence of the bounding box of the image in which the pedestrian exists appearing in the description or explanation of the pedestrian detection method of the embodiments of the present application are all referred to the acquired image, while in the neural network training method of the embodiments of the present application, the base feature map, the candidate box, the object visibility map, the enhanced feature map, the bounding box of the image in which the pedestrian exists, and the confidence of the bounding box of the image in which the pedestrian exists are all referred to the training image.

[0069] In a third aspect, a pedestrian detection apparatus is provided, which comprises modules for executing the method of the first aspect described above.

[0070] In a fourth aspect, a neural network training apparatus is provided, which comprises modules for executing the method of the second aspect described above.

[0071] In a fifth aspect, a pedestrian detection apparatus is provided, which comprises a memory for storing a program, and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the first aspect described above.

[0072] In a sixth aspect, a neural network training apparatus is provided, which comprises a memory for storing a program, and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the second aspect described above.

[0073] In a seventh aspect, an electronic device is provided, which comprises the pedestrian detection apparatus of the third aspect or the fifth aspect described above.

[0074] In an eighth aspect, an electronic device is provided, which comprises the neural network training apparatus of the fourth aspect or the sixth aspect described above.

[0075] The electronic device can be specifically a mobile terminal (for example, a smart phone), a tablet computer, a notebook computer, an augmented reality / virtual reality device, a vehicle terminal device, and the like.

[0076] In a ninth aspect, a computer-readable storage medium is provided, which stores program codes including instructions for performing the steps in the method of the first aspect or the second aspect.

[0077] In a tenth aspect, a computer program product including instructions is provided, which, when executed on a computer, causes the computer to perform the method of the first aspect or the second aspect.

[0078] In an eleventh aspect, a chip is provided, which includes a processor and a data interface, the processor reads instructions stored on a memory through the data interface, and performs the method of the first aspect or the second aspect.

[0079] Optionally, as an implementation manner, the chip can further include a memory, the memory stores instructions, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to perform the method of the first aspect.

[0080] The chip can be specifically a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0081] It should be understood that, in the present application, the method of the first aspect can be specifically a method in the first aspect and any one of the implementation manners of the first aspect. The method of the second aspect can be specifically a method in the second aspect and any one of the implementation manners of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0082] Figure 1 FIG. 1 is a schematic diagram of a process of pedestrian detection in an auxiliary / automatic driving system provided by an embodiment of the present application;

[0083] Figure 2 FIG. 2 is a schematic diagram of a process of pedestrian detection in a safe city / video monitoring system provided by an embodiment of the present application;

[0084] Figure 3 FIG. 3 is a structural schematic diagram of a system architecture provided by an embodiment of the present application;

[0085] Figure 4 FIG. 4 is a schematic diagram of pedestrian detection using a convolutional neural network model provided by an embodiment of the present application;

[0086] Figure 5A chip hardware structure schematic diagram provided by an embodiment of the present application;

[0087] Figure 6 A schematic block diagram of a pedestrian detection device according to an embodiment of the present application;

[0088] Figure 7 A schematic block diagram of a pedestrian detection device according to an embodiment of the present application;

[0089] Figure 8 A schematic diagram of pedestrian detection using a pedestrian detection device;

[0090] Figure 9 A schematic diagram of pedestrian detection using a pedestrian detection device;

[0091] Figure 10 A schematic flow chart of a pedestrian detection method according to an embodiment of the present application;

[0092] Figure 11 A schematic diagram of convolution processing of an image to obtain a basic feature map;

[0093] Figure 12 A schematic diagram of a process of generating a basic feature map of an image;

[0094] Figure 13 A schematic diagram of a process of determining an image candidate frame by an RPN module;

[0095] Figure 14 A schematic diagram of an object visibility map;

[0096] Figure 15 A schematic diagram of an object visibility map;

[0097] Figure 16 A schematic flow chart of a pedestrian detection method according to an embodiment of the present application;

[0098] Figure 17 A schematic diagram of a process of obtaining an object visibility map from a basic feature map by a self-activation module;

[0099] Figure 18 A schematic diagram of a process of weighted sum of an object visibility map and a basic feature map;

[0100] Figure 19 A schematic diagram of a process of obtaining a pedestrian detection result from a candidate frame and an enhanced feature map;

[0101] Figure 20 A schematic diagram of a process of obtaining a pedestrian detection result from an object visibility map, a candidate frame and an enhanced feature map;

[0102] Figure 21 A schematic diagram of determining a contour region of a candidate frame;

[0103] Figure 22 is a schematic diagram of determining the background region of the candidate box;

[0104] Figure 23 is a schematic diagram of a process of obtaining a pedestrian detection result according to the object visibility map, the candidate box, and the enhanced feature map;

[0105] Figure 24 is a schematic diagram of processing the features corresponding to the candidate box by the RCNN module;

[0106] Figure 25 is a schematic block diagram of the pedestrian detection device according to an embodiment of the present application;

[0107] Figure 26 is a schematic diagram of the hardware structure of the pedestrian detection device according to an embodiment of the present application;

[0108] Figure 27 is a schematic diagram of the hardware structure of the neural network training device according to an embodiment of the present application. DETAILED DESCRIPTION

[0109] The scheme of the present application can be applied in the field of pedestrian recognition (which can also be referred to as pedestrian detection) required in assisted driving, autonomous driving, safe city, intelligent terminal, etc. The following briefly introduces two commonly used application scenarios.

[0110] Application scenario one: assisted / autonomous driving system

[0111] In the advanced driving assistant system (ADAS) and autonomous driving system (ADS), it is necessary to detect and avoid dynamic obstacles such as pedestrians on the road surface, especially to avoid colliding with pedestrians. In the traffic road, dense pedestrians often appear, and the occlusion between pedestrians or between pedestrians and other objects is serious, which poses a serious threat to driving safety. Therefore, it is of great significance to accurately detect pedestrians in a serious occlusion scenario for safe driving.

[0112] Specifically, as shown in Figure 1 , the road picture image obtained by the assisted / autonomous driving system is subjected to pedestrian detection to obtain a pedestrian detection result, and then the assisted / autonomous driving system can control the vehicle according to the pedestrian detection result.

[0113] Among them, the pedestrian detection can be performed by the pedestrian detection method according to an embodiment of the present application. According to the pedestrian detection, it can be determined whether there is a pedestrian in the road picture and the position of the pedestrian, which is convenient for the assisted / autonomous driving system to control the vehicle according to the recognition result.

[0114] Application scenario two: Safe City / video monitoring system

[0115] In the Safe City system and the video monitoring system, by performing real-time pedestrian detection, marking the pedestrian detection result, and putting the pedestrian detection result into the analysis unit of the system, the criminal suspects and missing persons can be found, and then the sky net tracking is realized.

[0116] Specifically, as shown in Figure 2 , the road picture image obtained by the Safe City / video monitoring system is subjected to pedestrian detection to obtain a pedestrian detection result, and then specific personnel can be identified and tracked according to the pedestrian detection result.

[0117] Among them, the pedestrian detection can be performed by the pedestrian detection method of the embodiment of the present application. According to the pedestrian detection, it can be determined whether there is a pedestrian in the monitoring picture and the position of the pedestrian. When there is a pedestrian in the monitoring picture, it can be identified whether the pedestrian is specific personnel (missing persons, criminal suspects, etc.). When the specific personnel is identified, the sky eye system (which can be regarded as part of the Safe City / video monitoring system) can be started to track the specific personnel.

[0118] The pedestrian detection method of the embodiment of the present application can be performed by a neural network (model). In order to better understand the pedestrian detection method of the embodiment of the present application, the related terms and concepts of the neural network are introduced first.

[0119] (1) Neural network

[0120] The neural network can be composed of neural units. The neural unit can be an operation unit with x s and intercept 1 as input. The output of the operation unit can be as shown in formula (1):

[0121]

[0122] Among them, s = 1, 2, … n, n is a natural number greater than 1, W s is the weight of x swhere b is the bias of the neuron. f is the activation function of the neuron, which is used to perform a non-linear transformation on the features in the neural network, so as to convert the input signal in the neuron into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neurons, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neurons.

[0123] (2) Deep neural network

[0124] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0125] Although the DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is expressed as the following linear relationship expression: wherein, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Due to the large number of layers in the DNN, the number of coefficients W and offset vectors is also relatively large. These parameters in the DNN are defined as follows: taking the coefficient W as an example, assuming that in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0126] In summary, the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as

[0127] It is noted that the input layer is without W parameters. In deep neural networks, more hidden layers allow the network to better capture the complexity of real-world situations. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity" to perform more complex learning tasks. Training a deep neural network is the process of learning the weight matrices, and the ultimate goal is to obtain the trained weight matrices of all layers of the deep neural network (weight matrices formed by vectors W of many layers).

[0128] (3) Convolutional Neural Network

[0129] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor composed of a convolutional layer and a subsampling layer, which can be regarded as a filter. The convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can be connected only to part of the adjacent layer neurons. In a convolutional layer, there are usually several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights are the convolution kernel. The shared weights can be understood as the way of extracting image information regardless of the position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning in the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between layers of the convolutional neural network, and at the same time, to reduce the risk of overfitting.

[0130] (4) Residual Network

[0131] A residual network is a kind of deep convolutional network proposed in 2015. Compared with traditional convolutional neural networks, residual networks are easier to optimize and can improve accuracy by increasing the depth. The core of the residual network is to solve the side effects (degradation problem) brought by increasing the depth, so that the network performance can be improved by simply increasing the network depth. The residual network generally includes many sub-modules with the same structure, and a number representing the number of repeated sub-modules is usually used to connect the residual network, such as ResNet50, which means that there are 50 sub-modules in the residual network.

[0132] (6) Classifier

[0133] Many neural network structures end with a classifier that classifies the objects in the image. The classifier is generally composed of a fully connected layer and a softmax function (which can be called a normalized exponential function) and can output the probability of different categories according to the input.

[0134] (7) Loss function

[0135] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is actually intended to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the target value that is actually intended to be predicted (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict a lower value, and the adjustment is continuously made until the deep neural network can predict the target value that is actually intended to be predicted or a value very close to the target value that is actually intended to be predicted. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss.

[0136] (8) Back propagation algorithm

[0137] The neural network can use the back propagation (BP) algorithm to correct the values of the parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output produces an error loss, and the error loss information is propagated backward to update the parameters in the initial neural network model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0138] The above briefly introduces some basic contents of the neural network. The following describes some specific neural networks that can be used in image data processing.

[0139] The following describes the system architecture of the embodiment of the present application. Figure 3 The system architecture of the embodiment of the present application is described in detail.

[0140] Figure 3 is a schematic diagram of the system architecture of the embodiment of the present application. As shown in Figure 3As shown, the system architecture 100 includes an execution device 110, a training device 120, a database 130, a client device 140, a data storage system 150, and a data collection system 160.

[0141] In addition, the execution device 110 includes a computing module 111, an I / O interface 112, a pre-processing module 113, and a pre-processing module 114. The target model / rule 101 can be included in the computing module 111, and the pre-processing modules 113 and 114 are optional.

[0142] The data collection device 160 is configured to collect training data. For the pedestrian detection method of the embodiments of the present application, the training data can include training images (including pedestrians in the training images) and a label file, wherein the label file gives the coordinates of the bounding box of the pedestrians in the training images. After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0143] The training device 120 trains the target model / rule 101 based on the training data, and the training device 120 performs object detection on the input training images, compares the output pedestrian detection results (bounding box of pedestrians in the images and confidence of the bounding box of pedestrians in the images) with the labeled results, until the difference between the object pedestrian detection results output by the training device 120 and the pre-labeled results is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0144] The above-mentioned target model / rule 101 can be used to implement the pedestrian detection method of the embodiments of the present application, that is, the pedestrian detection result of the to-be-processed image (after related pre-processing) can be obtained by inputting the to-be-processed image into the target model / rule 101. The target model / rule 101 in the embodiments of the present application can be a neural network. It should be noted that in actual application, the training data maintained in the database 130 can not all be collected from the data collection device 160, but can also be received from other devices. In addition, it should be noted that the training device 120 can not completely train the target model / rule 101 based on the training data maintained in the database 130, but can also obtain training data from the cloud or other places for model training, and the above description should not be regarded as a limitation of the embodiments of the present application.

[0145] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as the execution device 110, the client device 140, and the data storage system 150. Figure 3The illustrated execution device 110 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) terminal, a vehicle-mounted terminal, etc., and can also be a server or a cloud, etc. In Figure 3 The execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. A user can input data to the I / O interface 112 through a client device 140, and the input data in the embodiments of the present application can include an image to be processed input by the client device. The client device 140 here can be a terminal device.

[0146] The pre-processing modules 113 and 114 are used for pre-processing the input data (such as the image to be processed) received by the I / O interface 112. In the embodiments of the present application, there can be no pre-processing modules 113 and 114 or only one pre-processing module. When there are no pre-processing modules 113 and 114, the input data can be directly processed by the computing module 111.

[0147] During the pre-processing of the input data by the execution device 110 or during the processing of the computing module 111 of the execution device 110, the execution device 110 can call data, codes, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0148] Finally, the I / O interface 112 presents the processing result, such as the pedestrian detection result calculated by the target model / rule 101, to the client device 140, thereby providing the user.

[0149] Specifically, the pedestrian detection result processed by the target model / rule 101 in the computing module 111 can be sent to the I / O interface after processing by the pre-processing module 113 (and the pre-processing module 114 can also be added) and then sent to the client device 140 by the I / O interface for display.

[0150] It should be understood that when there are no pre-processing modules 113 and 114 in the above system architecture 100, the computing module 111 can also transmit the processed pedestrian detection result to the I / O interface, and then the I / O interface sends the processing result to the client device 140 for display.

[0151] It is worth mentioning that the training device 120 can generate corresponding target model / rule 101 based on different training data for different targets or different tasks, and the corresponding target model / rule 101 can be used to achieve the above target or complete the above task, thereby providing the required result for the user.

[0152] In Figure 3 the case shown in the figure, the user can manually give the input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send the input data to the I / O interface 112, and if the user's authorization is required to automatically send the input data to the client device 140, the user can set the corresponding permission in the client device 140. The user can view the result output by the execution device 110 in the client device 140, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 140 can also act as a data acquisition end to collect the input data input into the I / O interface 112 and the output result output from the I / O interface 112 as new sample data and store it into the database 130. Of course, the collection can also be performed without the client device 140, and the I / O interface 112 directly stores the input data input into the I / O interface 112 and the output result output from the I / O interface 112 as new sample data into the database 130.

[0153] It is worth noting that Figure 1 The system architecture shown in the figure is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1 , the data storage system 150 is an external memory relative to the execution device 110, and in other cases, the data storage system 150 can also be placed in the execution device 110.

[0154] As Figure 3 shown, the target model / rule 101 obtained by training the training device 120 can be a neural network in the embodiments of the present application, and specifically, the neural network provided by the embodiments of the present application can be a CNN and a deep convolutional neural network (DCNN) and the like.

[0155] Since the CNN is a very common neural network, the following will be described in combination with Figure 4The structure of the CNN is described in detail. As described above in the introduction of the basic concepts, the convolutional neural network is a deep neural network with a convolutional structure, and is a deep learning architecture that learns at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, the CNN is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to an image input therein.

[0156] As shown in Figure 4 , the convolutional neural network (CNN) 200 can include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230. The details of these layers are described below.

[0157] Convolutional layer / pooling layer 220:

[0158] Convolutional layer:

[0159] As shown in Figure 4 , the convolutional layer / pooling layer 220 can include layers such as examples 221-226, for example: in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0160] The internal working principle of a convolutional layer will be described below using convolutional layer 221 as an example.

[0161] The convolution layer 221 can include a plurality of convolution operators, also known as kernels, which function as filters in image processing to extract specific information from an input image matrix. The convolution operator can be essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is usually processed on the input image along the horizontal direction, one pixel after another (or two pixels after two pixels, depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as that of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix will produce a single-depth convolution output, but most cases do not use a single weight matrix, but apply multiple weight matrices of the same size (rows x columns), i.e., multiple homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as determined by the "multiple" described above. Different weight matrices can be used to extract different features in the image, such as a weight matrix for extracting image edge information, another weight matrix for extracting specific colors of the image, and another weight matrix for blurring unwanted noise in the image, etc. The multiple weight matrices are of the same size (rows x columns), and the sizes of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same. The multiple convolution feature maps of the same size are combined to form the output of the convolution operation.

[0162] The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight matrices formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0163] When the convolutional neural network 200 has multiple convolution layers, the initial convolution layer (e.g., 221) often extracts more general features, which can also be referred to as low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolution layers (e.g., 226) become more and more complex, such as high-level semantic features, etc. The higher the semantic features, the more suitable they are for the problem to be solved.

[0164] Pooling layer:

[0165] Since it is often necessary to reduce the number of training parameters, a pooling layer is often periodically introduced after the convolution layer, such as in the AlexNet, VGG, and ResNet networks. Figure 4Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0166] Fully connected layer 230:

[0167] After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 can include multiple hidden layers (such as...). Figure 4 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0168] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 4 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 4 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0169] It should be noted that, as Figure 4The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0170] It should be understood that this can be adopted. Figure 4 The convolutional neural network (CNN) 200 shown performs the pedestrian detection method of the embodiments of this application, such as... Figure 4 As shown, after the image to be processed by the input layer 210, the convolutional / pooling layer 220 and the fully connected layer 230, the detection result of the image to be processed can be obtained (the bounding box of the pedestrian in the image to be processed and the confidence of the bounding box of the pedestrian in the image).

[0171] Figure 5 A chip hardware structure provided in this application embodiment includes a neural network processor 50. This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 4 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 5 This is achieved in the chip shown.

[0172] The neural network processing unit (NPU) 50 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0173] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0174] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 503 retrieves the corresponding data for matrix B from the weight memory 502 and caches it in each PE (Engineer Component) of the arithmetic circuit 503. The arithmetic circuit 503 retrieves the data for matrix A from the input memory 501 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator 508.

[0175] The vector computation unit 507 can further process the output of the arithmetic circuit 503, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0176] In some implementations, the vector computation unit 507 can store the processed output vector into a unified buffer 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0177] The unified memory 506 is used to store input data and output data.

[0178] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0179] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0180] The instruction fetch buffer 509, which is connected to the controller 504, is used to store the instructions used by the controller 504.

[0181] The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0182] Generally, the unified memory 506, input memory 501, weighted memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0183] Furthermore, in this application, Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507.

[0184] Figure 6 This is a schematic diagram of the system architecture of an embodiment of this application.

[0185] like Figure 6 As shown, the input image is processed by the pedestrian detection network to obtain 2D bounding boxes (also called bounding boxes) that surround pedestrians in the input image. This 2D bounding box information will be sent to subsequent analysis modules for processing, such as to the control unit of the autonomous driving system for obstacle analysis, or to the analysis module of the smart city for matching with missing persons, etc.

[0186] Figure 6 The pedestrian detection network shown includes a backbone module, a region proposal network (RPN) module, an auto-activation module, a basic feature weighting module, a region feature generation module, a region convolutional neural network (RCNN) module, and an output module. Figure 6 The pedestrian detection network in the present application can execute the pedestrian detection method of the present application embodiment. The following is a general description of the process by which the pedestrian detection network processes the input image.

[0187] After acquiring the input image, the backbone network module performs convolution processing on the input image to obtain the basic feature map of the input image; the RPN module processes the basic feature map to obtain pedestrian candidate boxes of the input image; the self-activation module performs further convolution processing and weighted summation processing on the basic feature map to obtain the object visibility map of the input image (this object visibility map can highlight the features of the visible parts of pedestrians in the input image); the basic feature weighting module is used to perform weight processing on the object visibility map and the basic feature map to obtain the enhanced feature map; the region feature module can generate the corresponding features of the candidate boxes based on the candidate boxes and the object visibility map of the input image; the RCNN module processes the features corresponding to the candidate boxes to obtain the pedestrian detection results of the input image. The pedestrian detection results can be the bounding boxes of pedestrians in the image and the confidence of the bounding boxes of pedestrians in the image.

[0188] The pedestrian detection device of this application embodiment can be implemented as an autonomous driving system, terminal device, monitoring device, etc. The present invention is deployed on the computing node of the relevant device, and through software modification, the accuracy of pedestrian recognition can be improved.

[0189] The pedestrian detection device in this application embodiment can be as follows: Figure 5 As shown, the pedestrian detection device mainly consists of a backbone network module 101, an RPN module 102, a self-activation module 103, a basic feature weighting module 104, a region feature generation module 105, an RCNN module 106, and an output module 107.

[0190] Figure 7 The pedestrian detection device shown may or may not include the region feature generation module 105.

[0191] When the pedestrian detection device does not include the region feature generation module 105, the pedestrian detection device and the process of using the pedestrian detection device to perform pedestrian detection can be... Figure 8 As shown.

[0192] like Figure 8 As shown, the RPN module determines candidate boxes of the image based on the basic feature map, the self-activation module 103 generates an object visibility map of the image based on the basic feature map, and the basic feature weighting module 104 performs a weighted summation of the object visibility map and the basic feature map to obtain the enhanced feature map. Next, region of interest (ROI) pooling is used to extract features corresponding to the candidate boxes from the enhanced feature map. Then, the RCNN module processes the features corresponding to the candidate boxes to obtain the pedestrian detection results. It should be understood that in... Figure 8 In this context, the features corresponding to the candidate boxes are the region features of the candidate boxes.

[0193] When the pedestrian detection device includes a region feature generation module 105, the pedestrian detection device and the process of performing pedestrian detection using the pedestrian detection device can be... Figure 9 As shown.

[0194] and Figure 8 compared to, Figure 9 The pedestrian detection device shown utilizes the region feature generation module 105 to generate region features of candidate boxes, as well as region features of the outline region and background region of the candidate boxes. These three features can be fused together to form the features corresponding to the candidate boxes. Next, the RCNN module processes the features corresponding to the candidate boxes to obtain the pedestrian detection results.

[0195] To better understand the execution process of the pedestrian detection method in the embodiments of this application, the following will first describe... Figure 7 The functions of each module in the document will be briefly described.

[0196] Backbone Network Module 101:

[0197] The backbone network module 101 performs a series of convolutional processes on the input image (also known as a picture) to obtain the image's basic feature map. This basic feature map provides the basic features of the image for subsequent modules to perform image detection.

[0198] The backbone network in the backbone network module 101 can be implemented in various ways, such as VGG network, deep residual network (ResNet) and Inception network.

[0199] RPN module 102:

[0200] RPN module 102 is used to predict regions where pedestrians may exist on the base feature map generated by backbone network module 101, and to provide the locations of these regions. The boundaries of these regions can be called candidate boxes (proposals). Generally, the locations of the candidate boxes detected by RPN module 102 are not very accurate. The candidate boxes may fall on the background image, or they may not well surround the pedestrian target in the image.

[0201] Self-activation module 103:

[0202] The self-activation module uses the shared Conv5 convolutional layer in the RCNN module 106 to further convolve the basic feature map generated by the backbone network module 101 to obtain a high-level semantic feature map. Next, the high-level semantic feature map is weighted using the same classifier weights as in the RCNN module 106 to obtain the image's object visibility map. This image's object visibility map shows a strong response to the visible parts of pedestrians, while showing a weak response to the background and occluded parts. The self-activation module 103 is the core module of the pedestrian detection device of this application.

[0203] Basic Feature Weighting Module 104:

[0204] The basic feature weighting module 104 is used to weight the basic feature map of the image generated by the backbone network module 101 and the object visibility map of the image generated by the self-activation module 103 to obtain an enhanced feature map of the image. In the enhanced feature map of the image, the features of the visible part of the pedestrian are enhanced, while the features of the background and the occluders that block the pedestrian are weakened.

[0205] Region Feature Generation Module 105:

[0206] The region feature generation module 105 refers to the object visibility map of the image generated by the self-activation module 103, processes each candidate box generated by the RPN module 102, and generates the outline region image of the pedestrian, the background region image, and the ROI region image corresponding to the current candidate box. The region feature generation module 105 extracts the features of these three region images through the network of interest-pooling module, and then fuses the extracted three region images as the region features of the current candidate box.

[0207] RCNN module 106:

[0208] The RCNN module 106 uses the same Conv5 convolutional layer as the self-activation module 103 to convolve the region features of the candidate boxes generated by the region feature generation module 105 to obtain the image features of the candidate box regions. Then, the global average pooling (GAP) module is used to average the image features of the candidate box regions. Finally, the averaged image features are fed into the box regressor and classifier respectively to predict the final coordinates and confidence of the candidate boxes.

[0209] Output module 107:

[0210] Output module 107 performs non-maximum suppression (NMS) on all candidate boxes output by RCNN module 106, thereby merging highly overlapping candidate boxes and filtering out candidate boxes with low confidence. This outputs 2D boxes (equivalent to bounding boxes mentioned below) reflecting the pedestrian detection results and their confidence scores.

[0211] The following is combined with Figure 10 The pedestrian detection method of the present application embodiments will be described in detail.

[0212] Figure 10 This is a schematic flowchart of a pedestrian detection method according to an embodiment of this application. Figure 10 The method shown can be performed by the pedestrian detection device described in this application. Figure 10 The method shown includes steps 1001 to 1007, which are described in detail below.

[0213] 1001. Obtain the image.

[0214] The image mentioned above may include pedestrians.

[0215] Specifically, the aforementioned images can be various images containing pedestrians, such as images taken by mobile phones or other smart terminals, road scene images obtained by assisted / autonomous driving systems, and surveillance scene images obtained by smart city / video surveillance systems.

[0216] It should be understood that the images captured by mobile phones or smart terminals, as well as road images and surveillance images, generally contain pedestrians. If these images do not contain pedestrians, then the final recognition result can be empty, meaning that recognizing such images that do not contain pedestrians will not identify the bounding box surrounding the pedestrians.

[0217] In step 1001 above, images can be acquired either by capturing images with a camera or by retrieving them from memory. It should be understood that... Figure 1 The method shown can also start directly from step 1002.

[0218] 1002. Perform feature extraction on the image to obtain the basic feature map of the image.

[0219] When obtaining the basic feature map of the image in step 1002, the basic feature map can be obtained by performing a convolution operation on the image (convolution processing), or by further processing the result of the convolution operation on the image (e.g., performing summation, weighting, concatenation, etc.).

[0220] In step 1002 above, the image can be convolved using the backbone network (module) in the neural network to obtain the basic feature map of the image. This backbone network can adopt various convolutional network architectures, such as VGG network (a network proposed by the visual geometry group at Oxford University), deep residual network (ResNet), and Inception network.

[0221] The aforementioned basic feature map can contain multiple channels. In step 1002, when extracting features from the image, the basic feature map can be obtained by performing convolution processing on the image. The process of obtaining the basic feature map through convolution processing can be as follows: Figure 11 As shown.

[0222] like Figure 11 As shown, the basic feature map is a feature map containing multiple channels. Figure 11 In this example, assuming the resolution of the input image is H0*W0*3 (height H0, width W0, and number of channels 3, i.e., three channels of RGB), then after convolution processing, the basic feature map U∈H*W*K can be obtained, where H and K represent the height and width of the basic feature map, respectively, and K represents the number of channels of the basic feature map.

[0223] The following is combined with Figure 12 The process of generating the basic feature map of the image in step 1002 is described in detail.

[0224] like Figure 12 As shown, different convolutional layers of the ResNet18 residual network can be used to perform convolution operations on the input image. Specifically, Figure 12 The convolution operation shown may specifically include the following procedures (1) to (4):

[0225] (1) ResNet18-Conv1 (the first convolutional layer of ResNet18) performs convolution processing on the input image to obtain feature map C1.

[0226] Assuming the input image has a resolution of H0*W0*3 (height H0, width W0, and 3 channels), the feature map C1 obtained after ResNet18-Conv1 convolution can have a resolution of H0 / 4*W0 / 4*64. Specifically, ResNet18-Conv1 can downsample the input image twice (each time the width and height are halved) and expand the number of channels from 3 to 64 to obtain feature map C1.

[0227] (2) ResNet18-Conv2 (the second convolutional layer of ResNet18) continues to convolve the feature map C1 to obtain the feature map C2.

[0228] ResNet18-Conv2 continues to convolve the feature map C1, and the resulting feature map C2 can have the same resolution as the feature map C1, which is H0 / 4*W0 / 4*64.

[0229] (3) ResNet18-Conv3 (the third convolutional layer of ResNet18) continues to convolve the feature map C2 to obtain the feature map C3.

[0230] ResNet18-Conv3 can downsample feature map C2 again and double the number of channels (expanding the number of channels from 64 to 128) to obtain feature map C3, with a resolution of H0 / 8*W0 / 8*128.

[0231] (4) ResNet18-Conv4 (the fourth convolutional layer of ResNet18) continues to convolve the feature map C3 to obtain the feature map C4.

[0232] ResNet18-Conv4 can downsample feature map C3 again and double the number of channels (expanding the number of channels from 128 to 256) to obtain feature map C4, with a resolution of H0 / 16*W0 / 16*256.

[0233] It should be understood that the above Figure 12 The convolution process shown is merely an example. In this embodiment, there are no limitations on the network used for convolution processing, the number of convolution processes, etc.

[0234] by Figure 12 Taking the convolution process shown as an example, the basic feature map in step 1002 can be either... Figure 12 The feature map C4 shown can also be Figure 12 At least one of the convolutional feature maps C1 to C4 shown.

[0235] When the basic feature map is Figure 13 When multiple feature maps of different resolutions are combined as shown, during the subsequent ROI pooling process, these feature maps of different resolutions can be adjusted into feature maps of the same resolution and input into the subsequent RCNN module for further processing.

[0236] 1003. Determine the candidate bounding boxes of the image based on the basic feature map.

[0237] The above candidate boxes are bounding boxes for regions in the image where pedestrians may exist. The region where the candidate box is located can be the region enclosed by the candidate box (the region inside the candidate box), and the region where the candidate box is located is also the region in the image where pedestrians may exist.

[0238] In step 1003, regions where pedestrians may exist can be predicted from the basic feature map of the image through the RPN module, and these regions are framed. The borders that frame these regions are the candidate boxes.

[0239] Generally speaking, the positions of the candidate boxes determined in step 1003 are not very accurate. The candidate boxes may fall on the background image (at this time, there may be no pedestrian targets in the candidate boxes), or may not be able to enclose the pedestrian targets well.

[0240] In step 1003, generally, the RPN module can be used to process the basic feature map to obtain candidate boxes. The following combines Figure 13 A detailed description of the process of the RPN module determining the candidate boxes of the image is given.

[0241] As Figure 13 shown, the RPN module can first perform convolution processing on the basic feature map using a 3×3 convolutional kernel to obtain the RPN hidden feature map (RPN Hidden). Next, two 3×3 convolutional kernels are respectively used to perform convolution processing on the RPN hidden feature map to predict the positions and confidences of each candidate box in the RPN hidden feature map. Generally speaking, the higher the confidence of a candidate box, the greater the probability that there is a pedestrian in this candidate box.

[0242] Next, the RPN module will perform merging processing on the predicted candidate boxes. When performing merging processing, redundant candidate boxes can be removed according to the overlapping degree between the candidate boxes. Among them, when removing redundant candidate boxes, the NMS algorithm or other methods can be used for screening candidate boxes.

[0243] Suppose the RPN module predicts a total of J candidate boxes. Then, N (N < J) candidate boxes with the highest scores can be selected from these J candidate boxes as candidate boxes containing pedestrians, where both N and J are positive integers.

[0244] However, the positions of the candidate boxes of the image determined by the RPN module are generally not very accurate either. As Figure 14 shown, although there are pedestrians in both candidate boxes, these two candidate boxes do not completely enclose the pedestrians within the boxes, nor tightly enclose the pedestrians within the boxes.

[0245] 1004. Process the basic feature map of the image to obtain the object visibility map of the image.

[0246] In this image, the object visibility map shows varying degrees of response to different objects. Specifically, the response to the visible portion of a pedestrian is stronger than the response to the invisible portion. In other words, the object visibility map exhibits a strong response to the visible portion of a pedestrian and a weak response to the invisible portion, with the visible portion showing more prominent features compared to the invisible portion.

[0247] Specifically, in the object visibility map of the image, the pixel values ​​of the visible portion of the pedestrian are greater than the pixel values ​​of the invisible portion of the pedestrian.

[0248] The following is combined with Figure 15 and Figure 14 This section explains the display effect of the object visibility map.

[0249] like Figure 14 As shown, the visible part of the human body is the portion of the pedestrian that can be seen, that is, the part of the pedestrian that is not obscured by other objects. The invisible part of the pedestrian can include other objects that obscure the pedestrian, as well as the background of the image. Figure 15 In the diagram, the visible parts of the human body are brighter, indicating a larger response, while the invisible parts of the human body are darker, indicating a smaller response.

[0250] like Figure 15 As shown, the first row of six images are the original images, and the second row of six images are the corresponding object visibility maps. In the six images shown in the second row, the visible parts of the human body are relatively bright, while the visible parts of the human body are still relatively dark. Figure 16 In the images shown in columns 3 and 4, the lower half of the human body is occluded. The brightness of the color corresponding to the visible part of the human body in the generated object visibility map is greater than the brightness of the color corresponding to the occluded part of the human body. This strengthens the features of the visible part of the human body, while weakening the features of the occluded part of the human body and the background area.

[0251] The object visibility map of the above images can more prominently reflect the features of the visible parts of the pedestrian.

[0252] In addition, in the object visibility map of the above image, the visible part of the pedestrian is the part where the pedestrian image can be seen, while the invisible part of the pedestrian is the part where the pedestrian image cannot be seen.

[0253] Since the object visibility map of the above image has a stronger response to the visible parts of pedestrians, combining the object visibility map of the image can improve the accuracy of pedestrian detection when determining the bounding box containing pedestrians in the image.

[0254] In step 1004 above, in order to obtain the object visibility map of the image, the basic feature map of the image can be convolved first, and then the multiple feature maps obtained by the convolution process can be weighted and summed to obtain the object visibility map of the image.

[0255] In this application, since the object visibility map of an image can more prominently reflect the features of the visible part of a pedestrian, the enhanced feature map obtained by fusing the base feature map and the object visibility map of the image highlights the features of the visible part of the pedestrian, which can improve the accuracy of subsequent pedestrian detection based on the enhanced feature map.

[0256] Optionally, in the object visibility map of the above image, the invisible part of the pedestrian includes the part occluded by the pedestrian.

[0257] When the invisible part of a pedestrian includes the occluded part, the visible part and the occluded part of the pedestrian can be distinguished in the object visibility map. Since the pixel value of the visible part of the pedestrian is greater than that of the occluded part, the features of the visible part of the pedestrian can be highlighted in the object visibility map, while the features of the occluded part are weakened. This reduces the impact of the occluded part of the pedestrian on the detection results in the subsequent pedestrian detection process, making it easier to highlight the features of the visible part of the pedestrian and improve the effectiveness of pedestrian detection.

[0258] Optionally, in the object visibility map of the above image, the invisible portion of the pedestrian includes the background portion of the image.

[0259] The background portion of the aforementioned image may refer to any part of the image other than pedestrians, or it may refer to any part of the image other than pedestrians and main objects (e.g., cars).

[0260] When the invisible part of a pedestrian includes the background of the image, it is possible to distinguish the pedestrian from the background in the object visibility map, highlight the features of the visible part of the pedestrian, weaken the features of the background, reduce the impact of the background on the detection results in the subsequent pedestrian detection process, facilitate the highlighting of the features of the visible part of the pedestrian in subsequent pedestrian detection, and improve the pedestrian detection effect.

[0261] like Figure 8 As shown, the specific process of determining the object visibility map of the image in step 1004 above may include the following steps:

[0262] 1004a. The basic feature maps of the image are convolved using a first convolutional network to obtain multiple first semantic feature maps;

[0263] 1004b. Perform a weighted summation on the above multiple first semantic feature maps to obtain the object visibility map of the image.

[0264] Among them, the aforementioned multiple first semantic feature maps are multiple feature maps with different semantics extracted from the entire base feature map. Specifically, in the aforementioned multiple first semantic feature maps, any two first semantic feature maps correspond to different semantics.

[0265] For example, the aforementioned multiple first semantic feature maps are composed of F1, F2 and F3, where F1 reflects head features, F2 reflects left hand features, and F3 reflects right hand features. The semantics reflected by F1, F2 and F3 are all different.

[0266] The convolution parameters used in step 1004a are the same as those used in step 1007a. In step 1004b, when performing a weighted summation of the multiple first semantic features, the weighting coefficients used are the weighting coefficients in the classifier used to determine the pedestrian score. This classifier refers to... Figure 17 The classifier in module 106 of RCNN.

[0267] Since the convolution parameters in step 1004a are the same as those in step 1007a, the weighting coefficients in step 1004a when performing weighted summation on the multiple first semantic features are the weighting coefficients used in the classifier to determine the pedestrian score. This makes the response of the object visibility map obtained in step 1004b to the visible part of the pedestrian greater than the response to the invisible part of the pedestrian.

[0268] The above steps 1004a and 1004b can be processed by the self-activation module to obtain the object visibility map of the image.

[0269] The following is combined with Figure 17 The above steps 1004a and 1004b are explained in detail.

[0270] like Figure 24 As shown, a Conv5 convolutional layer can be used to further convolve the basic feature map U∈H*W*K to obtain a high-level semantic feature map F∈H*W*K, where H, W, and K are the height, width, and number of channels of the high-level semantic feature map, respectively. The high-level semantic features can reflect features of some key parts of a person (e.g., head features, left hand features, right hand features, etc.).

[0271] Next, the high-level semantic feature map F can be further divided into K first feature maps Fi. k∈H*W, k=1,2,...,K., where each first feature map reflects different features. For example, F1 reflects head features, F2 reflects left hand features, F3 reflects background features, and so on. The importance of features of different parts in pedestrian recognition is different. Some features are very important in pedestrian recognition, while others are less important. For example, head features and facial features are relatively important in pedestrian recognition, while background features are relatively less important.

[0272] In addition, the convolution parameters of the Conv5 convolutional layer in the aforementioned self-activation module are related to the subsequent... Figure 8 The Conv5 convolutional layer in the RCNN module shown has the same convolutional parameters. Since the convolutional parameters of the Conv5 convolutional layer in the RCNN module are generally obtained through multiple training iterations before pedestrian detection, using the same convolutional parameters for the Conv5 convolutional layer in the self-activation module as in the RCNN module allows for better extraction of high-level semantic features from the image. Furthermore, since the classifier's weights establish a mapping between the features and the classification after the Conv5 convolutional layer, in order to reuse the classifier (… Figure 14 For the weights of the RCNN module (106), we need to make the self-activation module undergo the same encoding, so we use Conv5 convolutional layers here.

[0273] After obtaining the above-mentioned multiple first feature maps (F1 to F...) k After that, the multiple first feature maps can be weighted and summed according to formula (2) to obtain the visibility map of the image.

[0274]

[0275] In the above formula (2), w k ∈1*1 represents the first feature map F k ∈H*W weight coefficient, w k The larger the value, the greater the value of F. k The greater the contribution to pedestrian recognition, the better. k The greater the importance, the higher the level of importance.

[0276] When calculating the object visibility map of the image according to the above formula (2), the weight coefficients used (here, the weight coefficients can also be called visualization weights) are the same as the weight coefficients used when weighting and summing the feature maps of different semantics in the subsequent RCNN module. Specifically, the first feature map F kThe weight coefficients of H*W are the same as those used by the subsequent RCNN module when superimposing feature maps with the same semantics. For example, the weight coefficient of the first feature map representing the human head in the above formula (2) is the same as the weight coefficient used by the subsequent RCNN module when performing a weighted summation of the feature maps representing the human head.

[0277] The weighted summation method in formula (2) can highlight features that contribute significantly to human detection and suppress features that contribute little. Experiments have shown that the object visibility map V of the image calculated by formula (2) can produce a strong response to the visible parts of pedestrians and a weak response to the background and occluded parts.

[0278] The object visibility map of the image generated by the above formula (2) can be as follows: Figure 15 and Figure 14 As shown, in Figure 15 and Figure 14 In the image shown, the person's color is brighter, indicating a larger response, while the colors of other objects or the background around the person are darker, indicating a lower response. Overall, Figure 15 and Figure 18 The object visibility map of the image shown exhibits strong suppression of the background. Locally, it shows a strong response to the visible parts of the pedestrian and a weak response to the occluded parts of the pedestrian.

[0279] In this application, when multiple first feature maps are weighted and summed using the same weighting coefficients as those used in the RCNN module to obtain the object visibility map of the image, it is possible to highlight features that contribute significantly to the human body and suppress features that do not contribute significantly.

[0280] 1005. The basic feature map and the object visibility map of the image are fused to obtain the enhanced feature map of the image.

[0281] Before fusing the base feature map and the object visibility map of an image, the object visibility map can be expanded in dimension so that the number of channels in the expanded object visibility map is the same as the number of channels in the base feature map. Then, the base feature map and the object visibility map can be fused. The following three fusion methods can be used for the fusion process.

[0282] The first integration method:

[0283] In the first fusion method, corresponding elements in the base feature map are multiplied by corresponding elements in the expanded object visibility map, and then the product is summed with the corresponding elements in the original base feature map to obtain the enhanced feature map. The value of the corresponding element in.

[0284] Specifically, such as Figure 18 As shown, before dimensional expansion, the object visibility map V typically has only one channel. Dimension expansion ensures that the expanded object visibility map V has the same number of channels as the base feature map U. Then, the (i, j)th element of the base feature map U is multiplied by the (i, j)th element of the expanded base feature map U, and the product is added to the value of the (i, j)th element of the base feature map U to obtain the enhanced feature map. The value that each element can take.

[0285] It should be understood that, Figure 19 The process shown involves weighted summation of feature maps for each channel to obtain an enhanced feature map with the same number of channels as the base feature map. For example, if the base feature map U has 64 channels, the feature map of each channel in the base feature map can be multiplied by the corresponding element in the object visibility map. The result of this multiplication is then summed with the corresponding element in the base feature map to obtain the value of the corresponding element in the enhanced feature map. This process is repeated until the value of the element in the feature map of each of the 64 channels of the enhanced feature map is obtained.

[0286] Specifically, during the weighted processing, formula (3) can be used to weight the object visibility map V and the basic feature map U of the image to obtain the enhanced feature map of the image.

[0287]

[0288] In the above formula (3), This expands V by a factor of K along the channel dimension, and ⊙ represents the multiplication of corresponding elements.

[0289] The enhanced feature map of the image obtained by the self-activation module 103 can enhance the features of the visible parts of the human body in the image, while suppressing the features of the background and occluders, which facilitates high-precision pedestrian detection based on the enhanced image features.

[0290] The second fusion method:

[0291] In the second fusion method, corresponding elements in the base feature map are multiplied by corresponding elements in the dimension-expanded object visibility map. The resulting product is then summed with the corresponding elements of the dimension-expanded object visibility map to obtain the enhanced feature map. The value of the corresponding element in.

[0292] The third type of integration:

[0293] In the third fusion method, the corresponding elements in the basic feature map and the corresponding elements in the dimension-expanded object visibility map can be directly weighted and summed. The value of the element obtained after weighted summation is the value of the corresponding element in the enhanced feature map. When performing weighted summation, the weighting coefficient of the dimension-expanded object visibility map can be greater than the weighting coefficient of the basic feature map, so that the enhanced feature map mainly reflects the features of the object visibility map.

[0294] 1006. Based on the candidate bounding boxes and the enhanced feature map of the image, determine the features corresponding to the candidate bounding boxes.

[0295] The features corresponding to the candidate box obtained in step 1006 above may include the regional features of the candidate box, which are the features of the region located within the candidate box in the enhanced feature map.

[0296] Specifically, the regional features of the aforementioned candidate box can be the features within the region corresponding to the candidate box in the enhanced feature map. When determining the regional features of the candidate box, the position of the candidate box in the enhanced feature map can be determined first. Then, the features of the region enclosed by the candidate box in the enhanced feature map can be determined as the regional features of the candidate box.

[0297] Furthermore, the features of the region enclosed by the candidate box in the enhanced feature map can be sampled (specifically, upsampling or downsampling) to obtain the region features of the candidate box.

[0298] For the features corresponding to the above candidate boxes, in addition to the region features containing the candidate boxes, they can also include features of other regions outside the candidate boxes.

[0299] like Figure 20 As shown, features located within candidate boxes in the image enhancement feature map can be directly used as the features corresponding to the aforementioned candidate boxes, based on the candidate box positions of the image.

[0300] To make the features corresponding to the final candidate boxes more accurate, we can first determine the region of the candidate box by combining the object visibility map of the image, and then determine the features corresponding to the candidate box based on the region of the candidate box and the enhanced feature map of the image.

[0301] like Figure 20 As shown, the candidate box region can be determined from the object visibility map of the image based on the candidate box (position). Then, based on the region (position) of the candidate box, the features located within the candidate box in the enhanced feature map are determined as the features corresponding to the candidate box.

[0302] exist Figure 19In this context, because the object visibility map of an image has a higher response to the visible parts of the human body, the location of the candidate box can be determined more accurately by using the object visibility map, which facilitates the subsequent acquisition of more accurate regional features of the candidate box.

[0303] exist Figure 20 and Figure 21 In this process, after obtaining the regional features of the candidate boxes, the RCNN module can be used to further process the regional features of the candidate boxes to finally obtain the bounding boxes containing pedestrians in the image and the confidence scores of the bounding boxes containing pedestrians in the image.

[0304] In this application, to further improve the accuracy of the final pedestrian detection results, regional features of the area surrounding the candidate box can be extracted and fused with the regional features of the candidate box. Then, an R-CNN module is used to process the fused features to determine the bounding box containing a pedestrian in the image and the confidence level of the bounding box containing a pedestrian. In this application, because both the regional features of the candidate box and the regional features of the area surrounding the candidate box are used to comprehensively determine the bounding box containing a pedestrian in the image and the confidence level of the bounding box containing a pedestrian, the accuracy of pedestrian detection is higher.

[0305] Optionally, the features corresponding to the candidate box also include the regional features of the outline region of the candidate box, where the outline region of the candidate box is the region formed between the reduced candidate box and the candidate box after the candidate box is reduced according to a first preset ratio.

[0306] The region features of the outline area of ​​the above candidate boxes generally include the outline features of pedestrians, and the outline features of pedestrians also play an important role in pedestrian detection.

[0307] like Figure 10 As shown, first, based on the position of the candidate box in the image, the region of the candidate box is determined from the object visibility map of the image. Then, the border of the candidate box is reduced inward according to a certain ratio. The area between the original border of the candidate box and the reduced border of the candidate box is the outline region of the candidate box.

[0308] Alternatively, when determining the outline region of the candidate box, the region of the candidate box can be determined from the enhanced feature map, and then the border of the candidate box can be reduced inward according to a certain ratio. The area between the original border of the candidate box and the reduced border of the candidate box is the outline region of the candidate box.

[0309] The aforementioned reduction of the candidate box according to the first preset ratio can specifically mean that the width and height of the candidate box are reduced by a certain ratio. The ratios for reducing the width and height of the candidate box can be the same or different.

[0310] Optionally, the candidate box is reduced according to a first preset ratio, including: the width of the candidate box is reduced according to the first reduction ratio, and the height of the candidate box is reduced according to a second reduction ratio.

[0311] The first preset ratio includes a first reduction ratio and a second reduction ratio, wherein the first reduction ratio and the second reduction ratio can be the same or different.

[0312] The first and second scaling ratios can be set empirically. For example, appropriate values ​​can be set for the first and second scaling ratios so that the outline region of the candidate box obtained after scaling down the width and height of the candidate box according to the first and second scaling ratios can better extract the outline features of the pedestrian.

[0313] Setting the first and second scaling ratios allows you to select values ​​that are better at extracting pedestrian outlines.

[0314] The first reduction ratio mentioned above can be 1 / 1.1, and the second reduction ratio mentioned above can be 1 / 1.8.

[0315] When shrinking the border of the candidate box, it can be done by taking the center of the candidate box as the center point and shrinking it inward by a certain proportion. The center of the shrunken candidate box can be the maximum value point of the original candidate box in the object visibility map of the image.

[0316] Optionally, the center of the reduced candidate box is kept consistent with the center point of the original candidate box.

[0317] In this application, when the features corresponding to the candidate box also include the regional features of the candidate box's contour region, the contour features of pedestrians can be taken into account when performing pedestrian detection, which facilitates the subsequent comprehensive analysis of pedestrian contour features to better perform pedestrian detection.

[0318] Optionally, Figure 22 The method also includes setting the values ​​of features located within the reduced candidate box to zero in the region features of the candidate box to obtain the region features of the candidate box's outline region.

[0319] In this application, when obtaining the regional features of the outline region of the candidate box, the regional features of the outline region of the candidate box can be obtained quickly and conveniently by directly setting the values ​​of the features located within the reduced candidate box to zero.

[0320] It should be understood that in this application, other methods can also be used to obtain the regional features of the candidate box's outline region. For example, the regional features of the candidate box's outline region can be obtained directly from the enhanced feature map by scanning according to the regional position of the candidate box's outline region.

[0321] Optionally, the features corresponding to the candidate box also include the regional features of the background region of the candidate box. The background region of the candidate box is the region formed between the enlarged candidate box and the candidate box after the candidate box is enlarged according to the second preset ratio.

[0322] The background features of the candidate bounding box generally reflect the features of the background region where the pedestrian is located in the image. These background features can be combined with the pedestrian features to perform pedestrian detection.

[0323] The aforementioned expansion of the candidate box according to the first preset ratio can specifically mean that the width and height of the candidate box are expanded according to a certain ratio. The expansion ratios of the width and height of the candidate box can be the same or different.

[0324] Optionally, the candidate box is enlarged according to a first preset ratio, including: the width of the candidate box is enlarged according to the first enlargement ratio, and the height of the candidate box is enlarged according to a second enlargement ratio.

[0325] The first preset ratio includes a first expansion ratio and a second expansion ratio, wherein the first expansion ratio and the second expansion ratio can be the same or different.

[0326] The first and second enlargement ratios can be set empirically. For example, appropriate values ​​can be set for the first and second enlargement ratios so that the outline region of the candidate box obtained after enlarging the width and height of the candidate box according to the first and second enlargement ratios can better extract the outline features of pedestrians.

[0327] Setting the first and second magnification ratios allows you to select values ​​that are better at extracting pedestrian outlines.

[0328] The first expansion ratio mentioned above can be 1 / 1.1, and the second expansion ratio mentioned above can be 1 / 1.8.

[0329] like Figure 10 As shown, based on the candidate box (position) of the image, the region of the candidate box can be determined from the object visibility map of the image. Then, the border of the candidate box is expanded outward according to a certain ratio. The area between the original border of the candidate box and the expanded border of the candidate box is the background region of the candidate box.

[0330] Alternatively, when determining the background region of a candidate box, the region of the candidate box can be determined from the enhanced feature map of the image based on the position of the candidate box. Then, the border of the candidate box is expanded outward according to a certain ratio. The region between the original border of the candidate box and the expanded border of the candidate box is the background region of the candidate box.

[0331] When enlarging the border of a candidate box, the enlargement ratio can be determined based on experience. For example, when enlarging a candidate box, the width can be increased to 1.1 times its original size, and the height to 1.8 times its original size.

[0332] When expanding the border of a candidate box, it can be done by taking the center of the candidate box as the center point and expanding it outwards according to a certain ratio. The center of the expanded candidate box can be the maximum value point of the original candidate box in the object visibility map of the image.

[0333] Optionally, the center of the expanded candidate box is kept consistent with the center point of the original candidate box.

[0334] In this application, when the features corresponding to the candidate box also include the regional features of the background region of the candidate box, the regional features of the background region can be taken into account when performing pedestrian detection, which facilitates the subsequent comprehensive use of the regional features of the background region to better perform pedestrian detection.

[0335] Optionally, Figure 23 The method also includes: obtaining the regional features of a first region, wherein the regional features of the first region are the regional features located within the extended candidate box in the object visibility map; setting the features located within the candidate box in the regional features of the first region to zero to obtain the regional features of the background region of the candidate box.

[0336] Optionally, the enhanced feature map can be combined to determine the regional features of the background region of the candidate box. Specifically, the regional features of the second region can be obtained, which are the regional features of the region located within the extended candidate box in the enhanced feature map; the features of the second region located within the candidate box are set to zero to obtain the regional features of the background region of the candidate box.

[0337] In this application, when obtaining the regional features of the background region of the candidate box, the region features corresponding to the background region of the candidate box in the enhanced feature map are obtained, and then the features located within the candidate box in the region features are directly set to zero, which can quickly and conveniently obtain the regional features of the outline region of the candidate box.

[0338] It should be understood that the region features, contour features, and background features of the candidate box can be referred to as the features corresponding to the candidate box. That is to say, in this case, the features corresponding to the candidate box include not only the region features within the candidate box itself, but also the region features of other regions (contour region and background region) outside the candidate box itself.

[0339] Specifically, for the features corresponding to the candidate boxes mentioned above, in addition to including the region features of the candidate boxes, the features corresponding to the candidate boxes may also include at least one of the contour features of the candidate boxes and the background features of the candidate boxes.

[0340] When the features corresponding to the candidate box include the region features, the contour features, and the background features, the features corresponding to the candidate box contain the most information. This allows the subsequent pedestrian detection based on the features corresponding to the candidate box using the RCNN module to improve the accuracy of pedestrian detection to a certain extent.

[0341] It should be understood that the features corresponding to the candidate boxes mentioned above can directly include the three individual features of the candidate box region, the candidate box contour, and the candidate box background, or they can include the fused features of the three features. In other words, the three features of the candidate box region, the candidate box contour, and the candidate box background can be fused together, and the fused features are the features corresponding to the candidate boxes.

[0342] The following is combined with Figure 23 Taking the feature corresponding to the candidate box as the fusion of three features—the region feature, the outline feature, and the background feature—as an example, this paper details the process of obtaining the feature corresponding to the candidate box and performing pedestrian detection based on the feature corresponding to the candidate box.

[0343] like Figure 21 As shown, the candidate box region, the outline region of the candidate box, and the background region of the candidate box can be determined based on the candidate box and the object visibility map of the image (the process of determining these regions can be found in [reference]). Figure 22 and Figure 16 ).

[0344] Next, based on the candidate box region, the candidate box contour region, and the candidate box background region, the region features of the candidate box region, the candidate box contour region (which can be simply referred to as contour features), and the candidate box background region (which can be simply referred to as background features) are extracted from the enhanced feature map of the image.

[0345] After obtaining the region features of the candidate bounding box, the region features of the candidate bounding box's outline region, and the region features of the candidate bounding box's background region, these three features can be fused to obtain the fused features, which are the features corresponding to the candidate bounding box. The fusion of these three features can be done using a linear combination (weighted summation of the three features) or a non-linear combination. Before fusing these three features, they can be adjusted to the same size (e.g., 7*7*K, where K is the number of channels) and then fused.

[0346] After obtaining the features corresponding to the candidate boxes, the RCNN module can be used to process the features corresponding to the candidate boxes to finally obtain the bounding boxes containing pedestrians in the image and the confidence scores of the bounding boxes containing pedestrians in the image.

[0347] 1007. Determine the bounding boxes containing pedestrians in the image and their confidence levels based on the features corresponding to the candidate boxes.

[0348] By performing convolution processing and weighted summation on the features corresponding to the candidate boxes, we can better determine the bounding boxes containing pedestrians in the image and the confidence level of the bounding boxes containing pedestrians in the image based on the image features of the corresponding parts of the candidate boxes.

[0349] In this application, since the object visibility map of an image can more prominently reflect the features of the visible parts of pedestrians, the enhanced feature map obtained by fusing the basic feature map and the object visibility map of the image highlights the features of the visible parts of pedestrians, which can improve the accuracy of subsequent pedestrian detection based on the enhanced feature map. Furthermore, this application shows a significant improvement in the accuracy of pedestrian detection in cases of severe pedestrian occlusion.

[0350] In addition, this application improves pedestrian detection accuracy by combining object visibility maps without increasing the amount of labeled training data during training. This application only needs to generate the object visibility map during processing and take the object visibility map into account in subsequent processing. Compared with the scheme of improving pedestrian detection accuracy by increasing the amount of labeled data, it can save the amount of labeled data and reduce the complexity of training.

[0351] The pedestrian detection method of this application embodiment can be applied to various scenarios such as assisted / autonomous driving systems, safe city / video surveillance systems, etc.

[0352] When the pedestrian detection method of this application embodiment is applied to an assisted / autonomous driving system, the acquired data is a road image. Through the processing of the pedestrian detection method of this application embodiment, bounding boxes indicating the presence of pedestrians in the road image and the confidence level of these bounding boxes can be detected. Next, the autonomous vehicle can be controlled based on the bounding boxes indicating the presence of pedestrians in the road image and their confidence levels. For example, when it is determined that there is likely a pedestrian directly in front of the vehicle based on the bounding boxes and their confidence levels, the vehicle can be slowed down and the horn can be sounded, or the vehicle can be controlled to go around the pedestrian, etc.

[0353] When the pedestrian detection method of this application embodiment is applied to a safe city / video surveillance system, the first step is to acquire the surveillance video image. Then, through the processing of the pedestrian detection method of this application embodiment, bounding boxes containing pedestrians in the surveillance video image and their confidence levels are obtained. Next, characteristic individuals can be identified and tracked based on the bounding boxes containing pedestrians in the surveillance video image and their confidence levels. For example, when a specific person (missing person or criminal suspect) is identified in the surveillance video based on the bounding boxes containing pedestrians in the surveillance video image and their confidence levels, this specific person can be tracked through the SkyEye system (which can be considered part of the safe city / video surveillance system).

[0354] In step 1007 above, the features corresponding to the candidate boxes can be convolved first, and then the convolved feature maps obtained after convolution can be weighted and summed. Then, the bounding boxes of pedestrians in the image and the confidence of the bounding boxes of pedestrians in the image can be determined based on the feature maps obtained by weighted summation.

[0355] Specifically, such as Figure 24 As shown, the specific process of determining the bounding box of the image containing a pedestrian and the confidence level of the bounding box containing the pedestrian in step 1007 above may include the following steps:

[0356] 1007a. A second convolutional network is used to perform convolution processing on the features corresponding to the candidate boxes to obtain multiple second semantic feature maps;

[0357] 1007b. A regressor is used to process multiple second semantic features to determine the location of the bounding box containing a pedestrian in the image;

[0358] 1007c. A classifier is used to process multiple second semantic features to obtain the confidence score of the bounding box containing a pedestrian in the image.

[0359] The aforementioned multiple second semantic feature maps represent feature maps with different semantics extracted from the features corresponding to the candidate boxes. The convolution parameters of the aforementioned second convolutional network are the same as those of the first convolutional network. Specifically, in the aforementioned multiple second semantic feature maps, any two second semantic feature maps correspond to different semantics.

[0360] For example, the above-mentioned multiple second semantic feature maps are composed of F1, F2 and F3, where F1 reflects the head features, F2 reflects the left hand features and F3 reflects the leg features. The semantics reflected by F1, F2 and F3 are all different.

[0361] It should be understood that F1, F2, and F3 in this example are different from F1, F2, and F3 in the example of multiple first semantic feature maps mentioned above. Here, F1, F2, and F3 belong to second semantic feature maps, while F1, F2, and F3 in the example of multiple first semantic feature maps mentioned above belong to first semantic feature maps.

[0362] The location of the bounding box containing pedestrians in the above image and the confidence level of the bounding box containing pedestrians in the image can be the detection result of pedestrian detection in the image, and can be called the pedestrian detection result of the image.

[0363] The following is combined with Figure 24 The process of steps 1007a to 1007c above is described in detail.

[0364] like Pedestrian detection method As shown, the features corresponding to the candidate boxes can be convolved using Conv5 in the RCNN module to obtain multiple second semantic feature maps (C1 to Ck). Each of these second semantic feature maps reflects different features. For example, C1 reflects head features, C2 reflects left hand features, C3 reflects background features, and so on. After obtaining multiple second semantic features, the global average pooling (GAP) module can be used to average each second semantic feature map to obtain feature maps P1 to Pk. k Where P1 is obtained by averaging feature map C1, P k It is for feature map C k The result was obtained by averaging.

[0365] After obtaining feature maps P1 to P k Then, classification coefficients can be used in the classifier to classify feature maps P1 to P2. k Perform a weighted summation to obtain the confidence score of the candidate box row.

[0366] Specifically, the feature map P can be analyzed according to formula (4). k We perform a weighted summation to obtain the confidence level of the candidate boxes.

[0367]

[0368] In the above formula (4), w k ∈1*1 is P k The corresponding classifier coefficients, w in the RCNN classifier k The larger the value, the greater the value of P. k The greater the contribution to people, the greater the contribution due to P k It is C k The average, therefore, P k Represents C kTherefore, w k The larger the value, the more likely C is to be added. k The greater the effect on identifying people, the more effective the weight coefficients in the classifier become in relation to C. k It has a selective effect. Therefore, in the self-activation module, the classifier coefficients w shared with RCNN are used. k Weighting the high-level semantic feature map is necessary to form the object visibility map.

[0369] Finally, in the box regressor, similar coefficients are used to obtain more accurate box coordinates (xmin, ymin, width, height).

[0370] Where (xmin, ymin) can be the coordinates of the top-left corner of the box, and W and H represent the width and height of the box, respectively. Additionally, (xmin, ymin) can also be the position of the center point of the box, or the top-right / bottom-left / bottom-right corner of the box.

[0371] After obtaining the positions and coordinates of the candidate boxes, non-maximum suppression (NMS) can be applied to all the output candidate boxes to merge highly overlapping candidate boxes and filter out candidate boxes with too low confidence. The output then reflects the pedestrian detection results, namely, the bounding boxes containing pedestrians in the image and their confidence levels.

[0372] The effectiveness of the pedestrian detection method in this application embodiment is explained below with reference to Table 1. Table 1 shows the miss rate of the proposed solution and existing solutions when performing pedestrian detection on the publicly available dataset (CityPersons). Traditional solution 1 is an adapted faster RCNN solution, proposed at the 2017 IEEE conference on computer vision and pattern recognition (CVPR). Traditional solution 2 is an occlusion-aware RCNN solution, proposed at the 2018 European Conference on Computer Vision (ECCV).

[0373] Additionally, the second column of Table 1 represents the loss rate when performing pedestrian detection on all pedestrians in the image (including pedestrians occluded by other objects and pedestrians not occluded by other objects), and the third column of Table 1 represents the loss rate when performing pedestrian detection on pedestrians with severe occlusion in the image. A lower loss rate indicates better pedestrian detection performance.

[0374] As can be seen from Table 1, the loss rate of the proposed solution is lower than that of the existing solution in both scenarios. Especially in the scenario of severe occlusion, the proposed solution can achieve a performance gain of about 10% compared with the traditional solution, which is a significant improvement in performance.

[0375] Table 1

[0376] Loss rate (all) Loss rate (heavy occlusion) Adapted Faster RCNN (CVPR 17) OR-CNN (ECCV 18) 43.86 50.47 The present application 40.19 51.43 Figure 25 39.26 41.14

[0377] The pedestrian detection method of the present application embodiment has been described in detail above with reference to the accompanying drawings. The pedestrian detection device of the present application embodiment will be described in detail below with reference to the accompanying drawings. It should be understood that the pedestrian detection device described below can perform each step of the pedestrian detection method of the present application embodiment. In order to avoid unnecessary repetition, repeated descriptions will be appropriately omitted when introducing the pedestrian detection device of the present application embodiment.

[0378] Figure 25 This is a schematic block diagram of a pedestrian detection device according to an embodiment of this application. Figure 7 The pedestrian detection device 3000 shown includes an acquisition unit 3001 and a processing unit 3002.

[0379] The acquisition unit 3001 and the processing unit 3002 can be used to execute the pedestrian detection method of the present application embodiment. Specifically, the acquisition unit 3001 can execute the above step 1001, and the processing unit 3002 can execute the above steps 1002 to 1007.

[0380] The aforementioned processing unit 3002 can be divided into multiple modules according to different processing functions. Specifically, the processing unit 3002 can be equivalent to... Figure 7 The pedestrian detection device shown includes a backbone network module 101, an RPN module 102, a self-activation module 103, a basic feature weighting module 104, a region feature generation module 105, an RCNN module 106, and an output module 107. The processing unit 3002 is capable of implementing... Figure 26 The functions of each module in the pedestrian detection device shown are illustrated.

[0381] Figure 26 This is a schematic diagram of the hardware structure of the pedestrian detection device according to an embodiment of this application. Figure 27The pedestrian detection device 4000 shown (specifically, the device 4000 can be a computer device) includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. The memory 4001, processor 4002, and communication interface 4003 are interconnected via the bus 4004.

[0382] The memory 4001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 4001 may store a program, and when the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 is used to execute the various steps of the pedestrian detection method of the embodiments of this application.

[0383] The processor 4002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the pedestrian detection method of the method embodiment of this application.

[0384] The processor 4002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the pedestrian detection method of this application can be completed through the integrated logic circuitry in the processor 4002 or through software instructions.

[0385] The processor 4002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 4001. The processor 4002 reads the information in memory 4001 and, in conjunction with its hardware, completes the functions required by the units included in this pedestrian detection device, or executes the pedestrian detection method of the method embodiments of this application.

[0386] The communication interface 4003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 4000 and other devices or communication networks. For example, an image to be processed can be acquired through the communication interface 4003.

[0387] Bus 4004 may include a pathway for transmitting information between various components of device 4000 (e.g., memory 4001, processor 4002, communication interface 4003).

[0388] Figure 27 This is a schematic diagram of the hardware structure of a neural network training device according to an embodiment of this application. Similar to the device 4000 described above, Figure 26 The neural network training device 5000 shown includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0389] The memory 5001 can be a ROM, a static storage device, or RAM. The memory 5001 can store a program, and when the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 and the communication interface 5003 are used to execute the various steps of the neural network training method of the embodiments of this application.

[0390] The processor 5002 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image processing apparatus of this application embodiment, or to execute the neural network training method of the method embodiment of this application.

[0391] The processor 5002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the neural network training method of this application embodiment can be completed by the integrated logic circuitry in the processor 5002 or by software instructions.

[0392] The processor 5002 described above can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 5001. The processor 5002 reads the information in memory 5001 and, in conjunction with its hardware, completes the functions required by the units included in the image processing apparatus of this application embodiment, or executes the neural network training method of the method embodiment of this application.

[0393] The communication interface 5003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 5000 and other devices or communication networks. For example, an image to be processed can be acquired through the communication interface 5003.

[0394] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0395] It should be noted that although only a memory, processor, and communication interface are shown in the above-described devices 4000 and 5000, those skilled in the art should understand that in specific implementations, devices 4000 and 5000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 4000 and 5000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that devices 4000 and 5000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 27 and ​ All the devices shown.

[0396] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0397] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0398] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0399] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0400] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0401] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0402] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A pedestrian detection method, characterized in that, include: Acquire images; Feature extraction is performed on the image to obtain the basic feature map of the image; Candidate bounding boxes of the image are determined based on the basic feature map, wherein the candidate bounding boxes are the bounding boxes of regions in the image where pedestrians may exist; The basic feature map of the image is convolved to obtain multiple first semantic feature maps. The multiple first semantic feature maps include multiple feature maps with different semantics, which are used to reflect different parts of the pedestrian. The multiple first semantic feature maps are weighted and summed to obtain the object visibility map of the image, wherein the pixel value of the visible part of the pedestrian in the object visibility map is greater than the pixel value of the invisible part of the pedestrian. The basic feature map and the object visibility map of the image are fused to obtain the enhanced feature map of the image; Based on the candidate bounding boxes of the image and the enhanced feature map of the image, the features corresponding to the candidate bounding boxes are determined. The features corresponding to the candidate bounding boxes include the regional features of the candidate bounding boxes, which are the features of the regions located within the candidate bounding boxes in the enhanced feature map. The bounding boxes containing pedestrians in the image and their confidence levels are determined based on the features corresponding to the candidate boxes.

2. The method as described in claim 1, characterized in that, In the object visibility map, the invisible part of the pedestrian includes the part of the pedestrian that is obscured.

3. The method as described in claim 1, characterized in that, In the object visibility map, the invisible portion of the pedestrian includes the background portion of the image.

4. The method as described in claim 1, characterized in that, The step of determining the bounding box containing a pedestrian in the image and the confidence level of the bounding box containing a pedestrian in the image based on the features corresponding to the candidate box includes: A second convolutional network is used to perform convolution processing on the features corresponding to the candidate box to obtain multiple second semantic feature maps. The multiple second semantic feature maps respectively represent multiple feature maps with different semantics extracted from the features corresponding to the candidate box. The convolution parameters of the second convolutional network are the same as those of the first convolutional network. The first convolutional network is used to perform convolution processing on the basic feature map of the image to obtain the multiple first semantic feature maps. A regressor is used to process the multiple second semantic features to determine the position of the bounding box; A classifier is used to process the multiple second semantic features to obtain the confidence score of the bounding box containing a pedestrian in the image. The weighting coefficients used in the weighted summation of the multiple first semantic features are the weight coefficients in the classifier used to determine the pedestrian score.

5. The method as described in claim 1, characterized in that, The features corresponding to the candidate box also include the regional features of the outline region of the candidate box, wherein the outline region of the candidate box is the region formed between the reduced candidate box obtained by reducing the candidate box according to a first preset ratio and the candidate box.

6. The method as described in claim 5, characterized in that, The method further includes: Set the values ​​of the features located within the reduced candidate box in the region features of the candidate box to zero to obtain the region features of the outline region of the candidate box.

7. The method as described in claim 1, characterized in that, The features corresponding to the candidate box also include the regional features of the background region of the candidate box. The background region of the candidate box is the region formed between the expanded candidate box obtained by expanding the candidate box according to the second preset ratio and the candidate box.

8. The method as described in claim 7, characterized in that, The method further includes: Obtain the regional features of the first region, which are the regional features of the object visibility map located within the expanded candidate box. Set the features located within the candidate box in the regional features of the first region to zero to obtain the regional features of the background region of the candidate box.

9. The method as described in claim 1, characterized in that, The step of extracting features from the image to obtain the basic feature map of the image includes: The image is convolved to obtain the basic feature map of the image.

10. The method according to any one of claims 1-9, characterized in that, Determining the features corresponding to the candidate boxes based on the candidate boxes and the enhanced feature map of the image includes: Determine the position of the candidate box in the enhanced feature map; and The features of the region enclosed by the candidate box in the enhanced feature map are determined as the region features of the candidate box.

11. A pedestrian detection device, characterized in that, include: The acquisition unit is used to acquire images; Processing unit, the processing unit is used for: Feature extraction is performed on the image to obtain the basic feature map of the image; Candidate bounding boxes of the image are determined based on the basic feature map, wherein the candidate bounding boxes are the bounding boxes of regions in the image where pedestrians may exist; The basic feature map of the image is convolved to obtain multiple first semantic feature maps. The multiple first semantic feature maps include multiple feature maps with different semantics, which are used to reflect different parts of the pedestrian. The multiple first semantic feature maps are weighted and summed to obtain the object visibility map of the image, wherein the pixel value of the visible part of the pedestrian in the object visibility map is greater than the pixel value of the invisible part of the pedestrian. The basic feature map and the object visibility map of the image are fused to obtain the enhanced feature map of the image; Based on the candidate bounding boxes of the image and the enhanced feature map of the image, the features corresponding to the candidate bounding boxes are determined. The features corresponding to the candidate bounding boxes include the regional features of the candidate bounding boxes, which are the features of the regions located within the candidate bounding boxes in the enhanced feature map. The bounding boxes containing pedestrians in the image and their confidence levels are determined based on the features corresponding to the candidate boxes.

12. The apparatus as claimed in claim 11, characterized in that, In the object visibility map, the invisible part of the pedestrian includes the part of the pedestrian that is obscured.

13. The apparatus as claimed in claim 11, characterized in that, In the object visibility map, the invisible portion of the pedestrian includes the background portion of the image.

14. The apparatus as claimed in claim 11, characterized in that, The step of determining the bounding box containing a pedestrian in the image and the confidence level of the bounding box containing a pedestrian in the image based on the features corresponding to the candidate box includes: A second convolutional network is used to perform convolution processing on the features corresponding to the candidate box to obtain multiple second semantic feature maps. The multiple second semantic feature maps respectively represent multiple feature maps with different semantics extracted from the features corresponding to the candidate box. The convolution parameters of the second convolutional network are the same as those of the first convolutional network. The first convolutional network is used to perform convolution processing on the basic feature map of the image to obtain the multiple first semantic feature maps. A regressor is used to process the multiple second semantic features to determine the position of the bounding box; A classifier is used to process the multiple second semantic features to obtain the confidence score of the bounding box containing a pedestrian in the image. The weighting coefficients used in the weighted summation of the multiple first semantic features are the weight coefficients in the classifier used to determine the pedestrian score.

15. The apparatus as claimed in claim 11, characterized in that, The features corresponding to the candidate box also include the regional features of the outline region of the candidate box, wherein the outline region of the candidate box is the region formed between the reduced candidate box obtained by reducing the candidate box according to a first preset ratio and the candidate box.

16. The apparatus as claimed in claim 15, characterized in that, The processing unit is further configured to set the values ​​of the features located within the reduced candidate box in the regional features of the candidate box to zero, so as to obtain the regional features of the outline region of the candidate box.

17. The apparatus as claimed in claim 11, characterized in that, The features corresponding to the candidate box also include the regional features of the background region of the candidate box. The background region of the candidate box is the region formed between the expanded candidate box obtained by expanding the candidate box according to the second preset ratio and the candidate box.

18. The apparatus as claimed in claim 17, characterized in that, The processing unit is also used for: Obtain the regional features of the first region, which are the regional features of the object visibility map located within the expanded candidate box. Set the features located within the candidate box in the regional features of the first region to zero to obtain the regional features of the background region of the candidate box.

19. The apparatus as claimed in claim 11, characterized in that, The processing unit is used to perform convolution processing on the image to obtain the basic feature map of the image.

20. The apparatus as claimed in any one of claims 11-19, characterized in that, In the process of determining the features corresponding to the candidate boxes based on the candidate boxes and the enhanced feature map of the image, the processing unit is used to: Determine the position of the candidate box in the enhanced feature map; and The features of the region enclosed by the candidate box in the enhanced feature map are determined as the region features of the candidate box.

21. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code for execution by the device, the program code including methods for performing any one of claims 1-10.

22. A chip, characterized in that, The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1-10.

Images