Pedestrian detection method, pedestrian detection device and electronic equipment

Abstract

The invention provides a pedestrian detection method, a pedestrian detection device, electronic equipment and a non-instantaneous computer readable storage medium, and the pedestrian detection method comprises the steps: carrying out the fusion processing of multi-modal target detection image data in response to a pedestrian detection instruction, and obtaining multi-modal fusion data; extracting multi-modal feature data of the multi-modal fusion data; and performing pedestrian detection by using the multi-modal feature data. According to the embodiment of the invention, pedestrian detection is carried out through the image feature data obtained by fusing the multi-modal image data and fusing the different wavebands, compared with single-modal pedestrian detection, the robustness is better, the ability of distinguishing true and false targets and the ability of detecting a mirror surface shielding target are achieved, and the detection accuracy is improved. The method is suitable for target detection of dark light, rain and fog, high camouflage and other scenes.

Description

Technical Field

[0001] The present application relates to the field of pedestrian detection, and in particular, to a pedestrian detection method and a pedestrian detection device, an electronic device, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Traditional pedestrian detection is usually based on visible light images. However, this technology is seriously affected by the environment. When the ambient light changes, when there is rain or fog, or when the target is highly similar to the background to form camouflage, the performance of the visible light image detection algorithm will be seriously reduced. Infrared imaging can capture clear pedestrian images in low light, but when the ambient temperature rises, the contrast between pedestrians and the background decreases, or the target is blocked by a mirror, the performance of the infrared image pedestrian detection algorithm will also decrease. The combination of visible light and infrared can make the two images complement each other in detection. In recent years, with the development of deep learning multimodal networks, the target detection technology combining visible light and infrared dual modalities has been studied.

[0003] In the visible light and infrared multimodal image fusion detection network, the fusion method of the two modalities is the core technology, which is currently divided into three main methods: front-end fusion, intermediate fusion, and post-computation fusion. Front-end fusion directly fuses infrared and visible light data. This method has high requirements for the alignment of the two data, and the data fusion method is also the key. Intermediate fusion is that the network extracts a certain degree of features from infrared and visible light respectively and then fuses the features. Back-end fusion usually optimizes the infrared detection results and the visible light detection results to determine the final detection results. Although the latter two methods have lower requirements for data alignment, the network is usually very large and occupies a high memory.

[0004] The body weight of micro-nano UAVs is usually in the order of hundreds of grams or even tens of grams. Under this limitation, on the one hand, the storage and computing resources of the chip that can be equipped are very limited, and on the other hand, the performance of the configurable visible light and infrared image sensors is also very limited. In order to enable micro-nano UAVs to perform pedestrian detection in a variety of environments, the fusion detection of visible light and infrared modes is an important technical route.

[0005] Considering the current development status of chip technology, if we want to make the long-distance multimodal pedestrian deep learning detection algorithm meet the requirements of high real-time performance, we need to limit the computational complexity of the detection model to less than 1GOPs, the number of parameters to less than 5M, and the time consumption of the entire algorithm to less than 100ms. However, the currently used multimodal pedestrian detection model is large in scale and it is difficult to meet the above requirements. Summary of the invention

[0006] The present application aims to propose a pedestrian detection method and a pedestrian detection device, an electronic device, an electronic device and a non-transitory computer-readable storage medium to solve the problem that the current micro-nano UAV only uses a single modality for target detection when performing target detection, and does not fuse infrared and visible light image data for target detection, as well as the problem of large computational complexity in the current visible light and infrared dual-modality data target detection technology.

[0007] According to one aspect of the present application, a pedestrian detection method is proposed, comprising: in response to a pedestrian detection instruction, fusing multimodal target detection image data to obtain multimodal fusion data; extracting multimodal feature data of the multimodal fusion data; and performing pedestrian detection using the multimodal feature data.

[0008] According to some embodiments, the multimodal target detection image data includes infrared image data and visible light image data.

[0009] According to some embodiments, before fusing the multimodal target detection image data to obtain the multimodal fusion data, the pedestrian detection method further includes: correcting the multimodal target detection image data.

[0010] According to some embodiments, multimodal target detection image data is fused to obtain multimodal fusion data, including: performing feature shifting and data exchange on the infrared image data and the visible light image data according to a preset first step length.

[0011] According to some embodiments, before using the multimodal feature data for pedestrian detection, the pedestrian detection method also includes: performing feature shifting and data exchange on the infrared image data and the visible light image data in the multimodal feature data according to a preset second step size to obtain multimodal feature fusion data; performing bilinear interpolation processing on the multimodal feature fusion data according to a preset third step size, and using the bilinear interpolation processing result as correction data for the multimodal feature fusion data.

[0012] According to some embodiments, extracting multimodal feature data of the multimodal fusion data includes: extracting the multimodal feature data of the multimodal fusion data using the backbone network structure of yolov7-tiny.

[0013] According to some embodiments, before extracting the multimodal feature data of the multimodal fusion data, the pedestrian detection method further includes: performing data enhancement and / or Gaussian blur processing on the multimodal fusion data.

[0014] According to one aspect of the present application, a pedestrian detection device is proposed, comprising: an image data fusion processing module, which is used to perform fusion processing on multimodal target detection image data in response to a pedestrian detection instruction to obtain multimodal fusion data; a multimodal feature data extraction and correction module, which is used to extract and correct the multimodal feature data of the multimodal fusion data; and a target estimation module, which is used to perform pedestrian detection using the multimodal feature data.

[0015] According to one aspect of the present application, an electronic device is proposed, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor executes the pedestrian detection method as described in any of the previous embodiments.

[0016] According to one aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor executes the pedestrian detection method as described in any of the previous embodiments.

[0017] According to the embodiments of the present application, pedestrians are detected by fusing image feature data of different bands obtained by fusing multi-module image data. Compared with single-modality pedestrian detection, this method has better robustness and the ability to distinguish between true and false targets and the ability to detect targets occluded by mirrors. It is suitable for target detection in scenes such as dark light, rain and fog, and high camouflage.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. By describing the exemplary embodiments in detail with reference to the drawings, the above and other objectives, features and advantages of the present application will become more apparent.

[0020] Figure 1 A flow chart of a pedestrian detection method according to an exemplary embodiment of the present application is shown.

[0021] Figure 2 A schematic diagram of a multimodal target detection image data fusion processing process according to an exemplary embodiment of the present application is shown.

[0022] Figure 3 A schematic diagram of a feature data extraction process of multimodal fusion data according to an exemplary embodiment of the present application is shown.

[0023] Figure 4 A schematic diagram of a target estimation network structure training process according to an exemplary embodiment of the present application is shown.

[0024] Figure 5 A block diagram of a pedestrian detection device according to an exemplary embodiment of the present application is shown.

[0025] Figure 6 An electronic device according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. In the figures, the same figures show the same or similar parts, and thus their repeated description will be omitted.

[0027] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices or operations may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0029] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0030] Before describing the embodiments of the present application, the terms appearing in the present application are first explained.

[0031] LLVIP dataset: A visible-infrared image pairing dataset for various vision tasks in low-light conditions, such as image fusion, pedestrian detection, and image translation.

[0032] FLIR dataset: provides synchronously annotated infrared thermal images and unannotated visible light images.

[0033] ControlNet: A neural network technology for enhancing image generation that increases control over the generated content without changing the core structure of the generative model. The main idea is to guide the generation process by adding additional control signals, such as edges, sketches, depth maps, or other morphological information of the image. By introducing these signals, ControlNet can better meet the user's specific requirements for generated images.

[0034] YOLO (You Only Look Once): is a target detection algorithm, and YOLOv7-Tiny is its lightweight version.

[0035] COCO dataset: is an object detection, segmentation and key point detection dataset. The dataset is mainly captured from complex daily scenes, and the targets in the images are calibrated.

[0036] VOC dataset: is a computer vision dataset mainly used for tasks such as target detection, image segmentation, and image classification.

[0037] MPII dataset: is an image dataset for human pose estimation, including single-person pose estimation, multi-person pose estimation and pose recognition.

[0038] Mosaic: It is a data enhancement method that helps improve the detection performance of small objects by stitching four images together for training.

[0039] The specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0040] Figure 1 A flow chart of a pedestrian detection method according to an exemplary embodiment of the present application is shown. Figure 1 The pedestrian detection method shown includes steps S101 , S103 and S105 .

[0041] like Figure 1 As shown, in step S101, in response to a pedestrian detection instruction, multimodal target detection image data is fused to obtain multimodal fusion data.

[0042] According to an embodiment of the present application, the multimodal target detection image data includes visible light image data and infrared image data. In step S101, correction processing is also required for the visible light image data and the infrared image data.

[0043] In some embodiments, when the visible light image data and the infrared image data are corrected, one channel is infrared data (1, 320, 256), and the other channel is visible light data (1, 320, 256). The visible light image data and the infrared image data are arranged in parallel to obtain data of (2, 320, 256). In this embodiment, the visible light data adopts the Bayer format instead of the RGB format, thereby eliminating the RGB encoding, packaging, transmission and corresponding RGB decoding processes, which can greatly reduce the overall processing time of a single frame of data and reduce memory usage.

[0044] In micro-nano drone devices, there are large differences in the field of view angle, focal length and distortion parameters of infrared cameras and visible light cameras. In addition, due to the different assembly positions, the directly obtained infrared and visible light images will also have spatial misalignment. In other embodiments, when correcting the visible light image data and infrared image data, it is also necessary to use Zhang's calibration method to calibrate the infrared camera and the visible light camera, and then, according to the calibrated camera parameters, the captured visible light image data and infrared image data are subjected to epipolar correction, dedistortion, point projection and / or correction mapping, so that the objects in the visible light image data and the infrared image data are aligned, and the same object is kept the same size and in exactly the same position in the two visible light image data and infrared image data.

[0045] According to an embodiment of the present application, in step S101, feature shifting and data exchange are performed on infrared image data and visible light image data according to a preset first step length.

[0046] In a specific embodiment, the infrared image data and the visible light image data are feature shifted and fused again using a fusion sub-feature equalization method on the width W and height H of the data, and the moving step length of the fusion sub-feature equalization method is set to a preset first step length.

[0047] For example, Figure 2 As shown, it is assumed that the size of the fusion sub-image is 2*2*2, wherein the preset moving step is 2. The dimensions of the infrared image data and the visible light image data are (1, 320, 256) and (1, 320, 256), respectively. Before step S101, first, the two types of data are connected together according to the number of channels to obtain data (2, 320, 256), wherein the infrared image data is in the first channel and the visible light image data is in the second channel.

[0048] In step S101, when the infrared image data and the visible light image data are fused using the 2*2*2 fusion sub-unit, the infrared image data of the 1st and 2nd rows and the 1st and 2nd columns on the first channel are exchanged with the visible light data of the 1st and 2nd rows and the 1st and 2nd columns on the second channel. After that, the infrared image data and the visible light image data are sequentially fused horizontally and vertically according to the moving step 2 until the last row and column of the image data are reached, so as to balance the features between the infrared image data and the visible light image data, so as to fully fuse them and achieve the best detection effect.

[0049] In some embodiments, the specific operation of the moving step size of 2 is: when the fusion sub-module is used horizontally, the next exchange is to exchange the infrared data of the 1st and 2nd rows and the 5th and 6th columns on the first channel with the visible light data of the 1st and 2nd rows and the 5th and 6th columns on the second channel; when the fusion sub-module is used vertically, the next exchange is to exchange the infrared data of the 3rd and 4th rows and the 3rd and 4th columns on the first channel with the visible light data of the 3rd and 4th rows and the 3rd and 4th columns on the second channel.

[0050] In step S103, multimodal feature data of the multimodal fusion data is extracted.

[0051] According to an embodiment of the present application, the backbone network structure of yolov7-tiny is used to extract multimodal feature data of multimodal fusion data.

[0052] In a specific embodiment, a backbone network structure of yolov7-tiny is adopted, which is composed of 30 convolutional network layers, 10 batch normalization layers, and 1 maximum pooling layer.

[0053] Since the resolution of the multimodal feature data will be reduced after step S103, the feature points of the multimodal feature data will be offset. Therefore, in order to reduce the displacement problem of the feature points of the multimodal feature data and thus improve the overall performance of the backbone network structure, in some embodiments, in step S103, according to the preset convolution step size, the back layer of the convolution network layer of the backbone network structure also includes a feature optimization module.

[0054] According to an embodiment of the present application, the feature optimization module is composed of a fusion sub-core and a correction core.

[0055] In a specific embodiment, the feature optimization module first performs feature shift and data exchange on the infrared image data and the visible light image data in the multimodal feature data according to a preset second step size to obtain multimodal feature fusion data; then, bilinear interpolation processing is performed on the multimodal feature fusion data according to a preset third step size, and the bilinear interpolation processing result is used as the correction data of the multimodal feature fusion data.

[0056] For example, Figure 3 As shown, the size of the fusion sub-element is 3*3*2, and the step size is 2. The correction kernel is 2*2 in size and consists of 1 bilinear interpolation operator.

[0057] The specific operation of the fusion is as described in step S101 and will not be repeated here.

[0058] The specific operation of the correction kernel is:

[0059] Assume that the feature dimension of each channel obtained after calculation in any convolutional network layer is (a, b), and perform bilinear interpolation on the data in rows 1 and 2 and columns 1 and 2 to obtain a result v. The specific operation with a step size of 2 is:

[0060] When the correction kernel is applied horizontally, the next correction is to perform bilinear interpolation on the data of the 1st and 2nd rows and the 5th and 6th columns, and write the results to the back of v by row; when the correction kernel is applied vertically, the next correction is to perform bilinear interpolation on the data of the 3rd and 4th rows and the 3rd and 4th columns, and write the results to the back of v by column. The correction kernel is continuously applied to the data for correction horizontally and vertically according to the step size until the last row and column are calculated. In this implementation, each feature optimization module is inserted after the convolution network layer with a step size of 2 in the extraction module. The reason for setting it here is that after the feature optimization module performs calculations with a convolution kernel of 3 and a step size of 2, the feature resolution will be reduced to 1 / 4 of the original, and the feature points in the multimodal feature data will be offset. At this time, the feature optimization module can be used to perform better fusion by the fusion sub, and the correction kernel can reduce the degree of displacement of the feature points, thereby improving the overall performance of the network.

[0061] In step S105, pedestrian detection is performed using multimodal feature data.

[0062] According to an embodiment of the present application, in step S105, the target estimation network structure for pedestrian detection adopts the detection head network structure of yolov7-tiny.

[0063] In a specific embodiment, the detection head network structure consists of a 28-layer convolutional network, a 10-layer batch normalization layer and a 2-layer upsampling layer.

[0064] According to an embodiment of the present application, before step S105, the detection head network structure needs to be trained first.

[0065] According to other embodiments of the present application, in step S105, it is also necessary to perform non-maximum suppression (e.g., NMS: Non Maximum Suppression) processing on the output result of the detection head network structure to obtain pedestrian detection information. The pedestrian detection information includes the current detection frame coordinates, target confidence, and detection target category. In this embodiment, the detection target category is pedestrians.

[0066] according to Figure 1 The embodiment shown detects pedestrians by fusing image feature data of different bands obtained by fusing multi-module image data. Compared with single-modality pedestrian detection, it has better robustness and the ability to distinguish between true and false targets and the ability to detect targets occluded by mirrors. It is suitable for target detection in scenes such as dark light, rain and fog, and high camouflage.

[0067] According to the embodiments of the present application, the detection accuracy of drones in various scenarios is improved, the problems of misidentification and missed identification of targets are reduced, and the real-time detection effect is improved.

[0068] In this embodiment, by using visible light image data in Bayer format, memory usage can be reduced and the real-time operation of the entire algorithm can be accelerated.

[0069] Figure 4 A schematic diagram of a target estimation network structure training process according to an exemplary embodiment of the present application is shown below. Figure 4 Taking as an example, a target estimation network structure training process according to an exemplary embodiment of the present application is described in detail.

[0070] like Figure 4 As shown, in step S401, a training data set is prepared to obtain a training data set FD and a training data set FFD.

[0071] According to an embodiment of the present application, the LLVIP dataset and the Flir dataset are mixed and used as a training dataset FD of the model. When training the target estimation network structure, the infrared image data and visible light image data in the training dataset FD data are used as inputs of the target estimation network structure. In a specific embodiment, the bounding box in the corresponding text label in each infrared-visible light image data of the same field of view in the training dataset FD is used as a training target.

[0072] In a specific embodiment, the visible light image data adopts Bayer format data to omit the RGB encoding, packaging, transmission and corresponding RGB decoding process of the collected data, reduce the overall processing time of a single frame of data, and reduce memory usage. Since the visible light image data in the LLVIP dataset and the Flir dataset are RGB data converted from Bayer format data, the visible light RGB data in the training dataset FD is preprocessed and converted into Bayer format data, and the corresponding data labels can be kept unchanged.

[0073] Since the total number of images in the FD dataset is small, large-scale datasets with the same amount of data as the COCO dataset usually only contain visible light image data and lack corresponding infrared images. In some embodiments, the image style transfer model is used to transfer the visible light images of the COCO, VOC, and MPII datasets to infrared style using ControlNet to generate corresponding pseudo infrared-visible light image data pairs, and save the corresponding target labels as the dataset FFD. Since the pseudo infrared-visible light image data pairs are not real corresponding infrared images, the dataset FFD is only used for pre-training of the target estimation network structure, and its function is to set the initial value of the target estimation network structure when it actually starts training to the optimal initial value as much as possible.

[0074] In order to save memory, in some embodiments, the visible light image data of the COCO, VOC, and MPII datasets are converted from RGB data to Bayer data format.

[0075] In step S403, the multimodal data is corrected.

[0076] In a specific embodiment, if the infrared and visible light image data have been aligned, the infrared and visible light image data are arranged in sequence in the channel dimension as input data of step S405.

[0077] If the target estimation network structure is used on a micro-nano UAV, due to the large differences in the field of view angle, focal length, distortion parameters and other parameters of the infrared camera and the visible light camera, and the different assembly positions of the infrared camera and the visible light camera, the directly obtained infrared and visible light image data will also have spatial misalignment. Therefore, in some embodiments, it is necessary to first calibrate the infrared camera and the visible light camera using the Zhang calibration method, and then, according to the calibrated camera parameters, perform epipolar correction, dedistortion, point projection and / or correction mapping processing on the captured infrared-visible light image data, so that the objects in the infrared and visible light images are aligned.

[0078] For small target images, for example, image data with a target size less than 0.1 times the original image size, or in the COCO dataset, image data with a target size less than 32*32 pixels, in some embodiments, in order to increase the number of small target images, it is also necessary to perform data enhancement processing on the multimodal data, for example, rotating, cropping, scaling, and / or horizontally flipping the image data.

[0079] In a specific embodiment, samples containing small objects are oversampled to increase the number of images containing small objects, and small objects are copied, randomly scaled and angled, and then pasted at different positions in the image to increase the number of anchor boxes matched to the small objects during training. In other implementations, when Mosaic data augmentation is used, the size of each image in the mosaic is allowed to be different.

[0080] Since the quality of images collected by the visible light sensor configured in the unmanned equipment of the micro-nano platform is poor, in some embodiments, it is also necessary to perform Gaussian blur processing on the visible light image data and the infrared image data respectively.

[0081] In step S405, the multimodal data is fused.

[0082] Since the process of fusing the multimodal data in step S405 is the same as that in step S101 , it will not be described again for the sake of simplicity.

[0083] In step S407, feature extraction is performed on the fused multimodal data.

[0084] Since the feature extraction process of step S407 is the same as the processing process of step S103, it will not be repeated here for the sake of brevity.

[0085] In step S409, the target estimation network structure is pre-trained using the training data set FFD.

[0086] According to an embodiment of the present application, the target estimation network structure adopts a yolov7-tiny detection head network structure consisting of a 28-layer convolutional network, a 10-layer batch normalization layer and a 2-layer upsampling layer.

[0087] In a specific embodiment, the computational complexity of the constructed target estimation network structure is 385 MOPs, and the parameter quantity is 2.9M.

[0088] According to an embodiment of the present application, step S409 includes:

[0089] In sub-step S4091, the target estimation network structure parameters are randomly initialized, and the target estimation network structure is pre-trained using the data set FFD, so that the initial values ​​of the target estimation network structure when the actual training begins are set to the optimal initial values ​​as much as possible.

[0090] In sub-step S4093, supervised training is performed on the target estimation network structure.

[0091] In a specific embodiment, in step S4093, the loss function is used to back-propagate and update weights and biases through the gradient descent method to obtain a target estimation network structure.

[0092] In this embodiment, sub-step S4093 includes a forward propagation stage and a backward propagation stage.

[0093] The forward propagation stage includes initializing the weights and biases between the neuron nodes in the target estimation network structure; the deep neural network performs forward propagation. During the forward propagation process, the activation function is used to increase the nonlinear relationship between the networks, and finally a nonlinear mapping between the input and output results is generated.

[0094] The back-propagation phase includes calculating the loss function of the multimodal target estimation network structure and updating the parameters of the target estimation network structure by gradient descent.

[0095] The loss function of the target estimation network structure is shown in formula (1).

[0096] L=L box +L obj +L cls ′ (1)

[0097] Among them, L box is the loss function of the detection box; L obj is the object loss function, L cls ′ is the category loss function.

[0098] In a specific embodiment, uncertain learning is added to sub-step S4093 for training, and the category loss function is shown in formula (2).

[0099]

[0100] Where N is the number of samples in the training set, P i is the output probability corresponding to the true category of the i-th sample, c i The network outputs the certainty of the i-th sample, y i is the true value of the i-th sample.

[0101] In some embodiments, the parameters of the target estimation network structure are updated by gradient descent through the following process:

[0102] First, the parameters within the target estimation network structure are fixed and the gradient of the output layer loss function is calculated;

[0103] Then, calculate the gradient corresponding to each layer when the number of network layers is l = L-1, L-2, ..., 2;

[0104] Finally, the weights and biases of the entire target estimation network structure are updated.

[0105] In step S411, the target estimation network structure is trained using the training data set FD.

[0106] In step S411, based on step S409, the target estimation network structure is further trained using the training data set FD, and the weights, biases and other parameters obtained in step S409 are imported as initial parameters of the retraining model.

[0107] Since the training process of step S411 is the same as that of step S409, it will not be described again here for the sake of brevity.

[0108] Finally, through step S411, the final weight and bias parameters of the target estimation network structure are obtained, which can be directly used for pedestrian detection in step S105.

[0109] The above mainly introduces the embodiments of the present application from the perspective of the method. Those skilled in the art should easily appreciate that, in combination with the operations or steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Those skilled in the art can use different ways to implement the described functions for each specific operation or method, and such implementation should not be considered to be beyond the scope of the present application.

[0110] The following describes the device embodiments of the present application. For details not described in the device embodiments of the present application, reference can be made to the method embodiments of the present application.

[0111] Figure 5 A block diagram of a pedestrian detection device according to an exemplary embodiment of the present application is shown. Figure 5 The pedestrian detection device includes an image data fusion processing module 501, a multimodal feature data extraction and correction module 503, and a target estimation module 505. The image data fusion processing module 501 is used to respond to the pedestrian detection instruction, perform fusion processing on the multimodal target detection image data, and obtain multimodal fusion data; the multimodal feature data extraction module 503 is used to extract and correct the multimodal feature data of the multimodal fusion data; the target estimation module 505 is used to perform pedestrian detection using the multimodal feature data.

[0112] Figure 6 An electronic device according to an exemplary embodiment of the present application is shown. Figure 6 The electronic device 200 according to this embodiment of the present application is described. Figure 6 The electronic device 200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0113] like Figure 6As shown, the electronic device 200 is in the form of a general computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0114] The storage unit stores program codes, which can be executed by the processing unit 210, so that the processing unit 210 executes the methods described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 210 can execute the following: Figure 1 The method shown in .

[0115] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2201 and / or a cache memory unit 2202 , and may further include a read-only memory unit (ROM) 2203 .

[0116] The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0117] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0118] The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 200, and / or communicate with any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 250. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation method of the present application.

[0120] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0121] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0122] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0123] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0124] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0125] According to an embodiment of the present application, a computer program is provided, including a computer program or an instruction. When the computer program or the instruction is executed by a processor, the method described above can be executed.

[0126] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. A pedestrian detection method, characterized in that: include: In response to a pedestrian detection instruction, the multimodal target detection image data is fused to obtain multimodal fusion data; Extracting multimodal feature data of the multimodal fusion data; Pedestrian detection is performed using the multimodal feature data.

2. The pedestrian detection method according to claim 1, characterized in that: The multimodal target detection image data includes infrared image data and visible light image data.

3. The pedestrian detection method according to claim 1, characterized in that: Before fusing the multimodal target detection image data to obtain the multimodal fusion data, the pedestrian detection method further includes: The multimodal target detection image data is corrected.

4. The pedestrian detection method according to claim 2, characterized in that: The multimodal target detection image data is fused to obtain multimodal fusion data, including: The infrared image data and the visible light image data are subjected to feature shifting and data exchange according to a preset first step length.

5. The pedestrian detection method according to claim 4, characterized in that: Before using the multimodal feature data to perform pedestrian detection, the pedestrian detection method further includes: Performing feature shifting and data exchange on the infrared image data and the visible light image data in the multimodal feature data according to a preset second step length to obtain multimodal feature fusion data; The multimodal feature fusion data is subjected to bilinear interpolation processing according to a preset third step length, and the bilinear interpolation processing result is used as correction data of the multimodal feature fusion data.

6. The pedestrian detection method according to claim 5, characterized in that: Extracting multimodal feature data of the multimodal fusion data includes: The backbone network structure of yolov7-tiny is used to extract multimodal feature data of the multimodal fusion data.

7. The pedestrian detection method according to claim 1, characterized in that: Before extracting the multimodal feature data of the multimodal fusion data, the pedestrian detection method further includes: Perform data enhancement and / or Gaussian blur processing on the multimodal fusion data.

8. A pedestrian detection device, characterized in that: include: An image data fusion processing module is used to perform fusion processing on multi-modal target detection image data in response to a pedestrian detection instruction to obtain multi-modal fusion data; A multimodal feature data extraction and correction module, used to extract and correct the multimodal feature data of the multimodal fusion data; The target estimation module is used to detect pedestrians using the multimodal feature data.

9. An electronic device, comprising: processor; as well as A memory storing a computer program, which, when executed by the processor, enables the processor to execute the pedestrian detection method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the pedestrian detection method according to any one of claims 1 to 7.

Images