Trolley detection method and device for aerial image of unmanned aerial vehicle, electronic equipment and storage medium
Through drone aerial photography combined with dual-stream multi-scale information feature extraction and cross-modal fusion of visible light and infrared images, the problem of low accuracy in trolley detection in harsh environments is solved, and the accuracy of car recognition in complex backgrounds is significantly improved.
Patent Information
- Application Number
- CN202411058199.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-26
AI Technical Summary
In harsh environments, individual visible light images cannot effectively detect hidden cars, while infrared images are ineffective against cold backgrounds, making it difficult to improve the accuracy of car recognition in complex backgrounds.
Visible light and infrared images were obtained through drone aerial photography, dual-stream multi-scale information was used for feature extraction, feature fusion was combined with channel stitching and global average pooling, and cross-modal feature fusion and position regression processing were carried out to improve the accuracy of trolley detection.
It improves the accuracy of car recognition in complex backgrounds, especially in rainy, snowy and foggy weather, and improves detection accuracy.
Smart Images

Figure CN119027835A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer artificial intelligence and image recognition technology, and in particular to a method, device, electronic equipment and storage medium for detecting a vehicle in an image taken by an unmanned aerial vehicle. Background Art
[0002] With the development of computer vision technology, computer algorithms such as image segmentation and target detection are emerging in an endless stream. Visible light images alone cannot meet the needs of complex visual tasks. In many harsh environments, such as smoke-filled environments, visible light images can only reflect the surrounding environmental information, but the car target hidden in the smoke cannot be presented in the visible light image. Although infrared images can obtain information about objects with thermal radiation, they cannot display cold backgrounds. Therefore, whether it is possible to use infrared sensors and visible light sensors for imaging at the same time to integrate the advantages of the two images and improve the target detection effect of hidden cars has become an urgent problem to be solved. Summary of the invention
[0003] The main purpose of the embodiments of the present invention is to provide a method, device, electronic device and storage medium for detecting a car in an unmanned aerial vehicle aerial image, thereby improving the accuracy of identifying a car in a complex background.
[0004] One aspect of the present invention provides a method for detecting a vehicle in an unmanned aerial vehicle aerial image, comprising:
[0005] Taking aerial photos of the target area by using a drone to obtain a visible light image and an infrared light image of the target area;
[0006] Extracting features of the visible light image and the infrared light image through dual-stream multi-scale information to obtain target infrared features and target visible light features;
[0007] During each feature extraction, the target infrared feature and the target visible light feature are mapped, mixed and compressed to obtain an infrared weight vector and a visible light weight vector;
[0008] The infrared features of the target are filtered by using an infrared weight vector using channel splicing and global average pooling to obtain filtered infrared features, the visible light features are filtered by using a visible light weight vector to obtain filtered visible light features, and the filtered infrared features and the filtered visible light features are cross-modally fused to obtain fused features;
[0009] The fused features are subjected to position regression processing to obtain a detection frame of the car, and the fused features are subjected to category regression to obtain a detection category of the car. The car detection result of the target area is determined according to the detection frame and the detection category.
[0010] According to the method for detecting a car in an unmanned aerial vehicle aerial image, the method further comprises:
[0011] Performing a first feature extraction on the visible light image and the infrared light image to obtain a first infrared feature and a first visible light feature, and performing a first fusion of the first infrared feature and the first visible light feature to obtain a first fusion feature;
[0012] Extracting the first infrared feature and the first visible light feature for a second time to obtain a second infrared feature and a second visible light feature, and fusing the second infrared feature and the second visible light feature for a second time to obtain a second fused feature;
[0013] Extracting the second infrared feature and the second visible light feature for a third time to obtain a third infrared feature and a third visible light feature, and fusing the third infrared feature and the third visible light feature for a third time to obtain a third fused feature;
[0014] The first fusion feature, the second fusion feature and the third fusion feature are extracted for a fourth time to obtain the first feature to be detected, the second feature to be detected and the third feature to be detected. The first extraction, the second extraction and the third extraction are extracted using a dual-stream multi-scale information method. The first fusion, the second fusion and the third fusion are feature fused using a cross-modal feature fusion method.
[0015] According to the method for detecting a car in an unmanned aerial vehicle aerial image, the visible light image and the infrared light image are subjected to feature extraction through dual-stream multi-scale information to obtain target infrared features and target visible light features, including:
[0016] The input feature is copied into three copies to obtain the first feature, the second feature and the third feature, wherein the input feature is the feature obtained by the first extraction, the second extraction and the third extraction;
[0017] The first feature is processed through a dilated convolution layer with a dilation rate of 1 to obtain a first sub-feature; the second feature is processed through a dilated convolution layer with a dilation rate of 3 to obtain a second sub-feature; the third feature is processed through a dilated convolution layer with a dilation rate of 5 to obtain a third sub-feature;
[0018] The first sub-feature, the second sub-feature and the third sub-feature are spliced on the channel to obtain a fourth fused feature, and the fourth fused feature is processed through a convolution layer with a convolution kernel of 3 to obtain a target infrared feature and a target visible light feature.
[0019] According to the vehicle detection method for drone aerial images, channel stitching and global average pooling are used to filter the target infrared features through infrared weight vectors to obtain filtered infrared features, and the visible light features are filtered through visible light weight vectors to obtain filtered visible light features. The filtered infrared features and the filtered visible light features are cross-modally fused to obtain fused features, including:
[0020] The target visible light feature and target infrared feature maps are mixed in the first stage by channel splicing and global average pooling to obtain a mixed vector. The formula is:
[0021] F fu1 =GAP(concat(F ir ,F vi ))
[0022] Among them, F fu1 is a compressed mixed vector, GAP() represents global average pooling, and concat() represents concatenation along the channel, where F ir is the target visible light feature, F vi is the target infrared signature;
[0023] The mixed vector is compressed through two fully connected layers to obtain the infrared weight vector for fusion and the visible light weight vector for fusion. The formula is:
[0024]
[0025] where q inf represents the infrared weight vector used for fusion, q vis Represents the visible light weight vector used for fusion, FC a→b represents the fully connected layer that squeezes the weight channel from a to b, c represents the number of channels of each feature for feature fusion, and r represents the scaling rate.
[0026] According to the vehicle detection method for drone aerial images, channel stitching and global average pooling are used to filter the infrared features through infrared weight vectors, and the visible light features are filtered through visible light weight vectors, and the filtered infrared features are fused with the filtered visible light features to obtain fused features, including:
[0027] The channel weight vector is used to filter the infrared features on the channel. The formula is:
[0028] F inf1 =q inf ☉F inf
[0029] Among them, F inf1represents the second infrared feature after channel screening, ⊙ represents channel-by-channel multiplication;
[0030] The same method is used to screen the visible light features on the channel;
[0031] The second stage feature blending is performed using the formula
[0032] F fu2 =concat(F vis +q vis ☉F vis , F inf +q inf ☉F inf )
[0033] Among them, F inf2 represents the mixed features obtained in the second stage;
[0034] Through the MLP layer, F inf2 Compression is performed on the channel dimension to obtain a mixed feature with 2 channels. Then this feature is separated on the channel to obtain the final spatial weight map:
[0035] MAP = conv(conv(F fu2 ))
[0036] MAP vis , MAP inf =splite(MAP)
[0037] Among them, MAP represents the features after MLP compression, conv() represents the convolution layer with a convolution kernel of 3, and splite() represents separation along the channel;
[0038] The features filtered in the first stage are fused in the second stage using the following formula:
[0039]
[0040] Among them, F fused Represents the output fusion features.
[0041] According to the method for detecting a car in an unmanned aerial vehicle aerial image, the fused features are subjected to position regression processing to obtain a detection frame of the car, and the fused features are subjected to category regression to obtain a detection category of the car, including:
[0042] dividing the fused feature into a first fused feature and a second fused feature;
[0043] The first fusion feature is subjected to feature extraction through three repeated convolution layers with a convolution kernel of 3 to obtain a third fusion feature, and the third fusion feature is subjected to position regression processing to obtain a detection frame;
[0044] Extract the second fused feature through three repeated convolution layers with a convolution kernel of 3 to obtain a fourth fused feature, and perform category regression processing on the fourth fused feature to obtain a detection category;
[0045] The detection box is constrained by the loss function, where
[0046]
[0047] Where L BCE represents the cross entropy loss, L REG represents the bounding box loss, L BCE for
[0048] L BCE =-(ylog(p(x)+(1-y)log(1-p(x))))
[0049] Where p(x) is the category of the target box generated by the network, and y represents the true label of the object;
[0050] Where L REG The formula is
[0051]
[0052] Where y represents the coordinate value of the box obtained by network regression, y i Indicates the new coordinate position estimated based on the true label.
[0053]
[0054] in Represents the coordinates of the real box of each car.
[0055] According to the method for detecting a car in an unmanned aerial vehicle aerial image, the number of pixels of the car in the visible light image or the infrared light image is less than or equal to 100.
[0056] Another aspect of the present invention provides a vehicle detection device for drone aerial images, comprising:
[0057] The first module is used to take aerial photos of the target area by using a drone to obtain a visible light image and an infrared light image of the target area;
[0058] The second module is used to extract features of the visible light image and the infrared light image through dual-stream multi-scale information to obtain target infrared features and target visible light features;
[0059] The third module is used to map, mix and compress the target infrared feature and the target visible light feature each time the feature is extracted to obtain an infrared weight vector and a visible light weight vector;
[0060] The fourth module is used to filter the target infrared features through the infrared weight vector by using channel stitching and global average pooling to obtain filtered infrared features, filter the visible light features through the visible light weight vector to obtain filtered visible light features, and perform cross-modal feature fusion on the filtered infrared features and the filtered visible light features to obtain fused features;
[0061] The fifth module is used to perform position regression processing on the fused features to obtain a detection frame of the car, and to perform category regression on the fused features to obtain a detection category of the car, and to determine the car detection result in the target area based on the detection frame and the detection category.
[0062] Another aspect of an embodiment of the present invention provides an electronic device, including a processor and a memory;
[0063] The memory is used to store programs;
[0064] The processor executes the program to implement the method described above.
[0065] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method described above.
[0066] The beneficial effects of the present invention are as follows: a dual-stream multi-scale information extraction method is adopted, feature enhancement is performed during extraction, and features of different modes are fully mixed to improve the accuracy of car detection in rainy, snowy and foggy weather; and the accuracy of car recognition under complex backgrounds is improved by combining infrared and visible light for recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0068] Figure 1 It is a schematic diagram of the process of a method for detecting a vehicle in an unmanned aerial vehicle aerial image according to an embodiment of the present invention.
[0069] Figure 2 Schematic diagram of a dual-stream multi-scale information extraction block according to an embodiment of the present invention.
[0070] Figure 3 Schematic diagram of a cross-modal feature fusion block according to an embodiment of the present invention.
[0071] Figure 4 Schematic diagram of a detection head according to an embodiment of the present invention.
[0072] Figure 5 It is a schematic diagram of the overall network architecture of a vehicle detection in a drone aerial image according to an embodiment of the present invention.
[0073] Figure 6 2 is a diagram of a device for detecting and analyzing a vehicle in an unmanned aerial vehicle (UAV) aerial image according to an embodiment of the present invention. DETAILED DESCRIPTION
[0074] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. In the subsequent description, the use of suffixes such as "module", "component" or "unit" used to represent elements is only for the purpose of facilitating the description of the present invention, and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner. "First", "second" and the like are only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the continuous numbering of the method steps is for the convenience of review and understanding. In combination with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the implementation order between the steps will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0075] refer to Figure 1 , which is a flow chart of a method for detecting a vehicle in an unmanned aerial vehicle aerial image according to an embodiment of the present invention, including but not limited to steps S100 to S500:
[0076] S100, using a drone to take aerial photos of a target area to obtain visible light images and infrared light images of the target area.
[0077] In some embodiments, a pair of registered infrared images and visible light images are obtained by an infrared sensor and a visible light sensor carried on a drone.
[0078] S200, extracting features of the visible light image and the infrared light image through dual-stream multi-scale information to obtain target infrared features and target visible light features.
[0079] S300, mapping, mixing and compressing the target infrared features and the target visible light features each time the features are extracted, to obtain an infrared weight vector and a visible light weight vector.
[0080] In some embodiments, the process of extracting dual-stream multi-scale information is as follows:
[0081] Performing a first feature extraction on the visible light image and the infrared light image to obtain a first infrared feature and a first visible light feature, and performing a first fusion of the first infrared feature and the first visible light feature to obtain a first fusion feature;
[0082] Extracting the first infrared feature and the first visible light feature for a second time to obtain a second infrared feature and a second visible light feature, and fusing the second infrared feature and the second visible light feature for a second time to obtain a second fused feature;
[0083] Extracting the second infrared feature and the second visible light feature for the third time to obtain a third infrared feature and a third visible light feature, and fusing the third infrared feature and the third visible light feature for the third time to obtain a third fused feature;
[0084] The first fusion feature, the second fusion feature and the third fusion feature are extracted for the fourth time to obtain the first feature to be detected, the second feature to be detected and the third feature to be detected. The first extraction, the second extraction and the third extraction are performed using a dual-stream multi-scale information method. The first fusion, the second fusion and the third fusion are performed using a cross-modal feature fusion method.
[0085] In some embodiments, reference Figure 2 The schematic diagram of the dual-stream multi-scale information extraction block shown in the figure has the following process:
[0086] The feature extraction module consists of three dilated convolutions with different dilation rates, a channel splicing module, and a convolution module with a convolution kernel of 3. First, the input feature is copied into three parts, and then three dilated convolutions with different dilation rates are used to extract the copied features. Then, the three extracted features are spliced along the channel, and finally a convolution layer is used to add further transformation to the spliced features, where the input features are the features extracted for the first, second, and third times.
[0087] Preferably, three dilated convolutions with different dilation rates are used, and the dilation rates are set to 1, 3, and 5. The convolution kernel size is set to 3, the convolution step size is set to 1, and the convolution zero padding is set to 1.
[0088] The infrared image and the visible light image are respectively subjected to feature extraction by three feature extraction modules to obtain vis1 and inf1, and then continue to pass through a feature extraction module to obtain vis2 and inf2, and then pass through a feature extraction module again to obtain vis3 and inf3.
[0089] S400 uses channel splicing and global average pooling to filter the target infrared features through the infrared weight vector to obtain filtered infrared features, filters the visible light features through the visible light weight vector to obtain filtered visible light features, and performs cross-modal feature fusion on the filtered infrared features and the filtered visible light features to obtain fused features.
[0090] In some embodiments, reference Figure 3 The cross-modal feature fusion block shown has the following workflow:
[0091] In order to fully mix the features of different modalities, the embodiment of the present invention discloses a two-stage feature fusion module. Through channel splicing and global average pooling, the extracted visible light features and infrared features are mapped into a mixed vector:
[0092] F fu1 =GAP(concat(F ir ,F vi ))
[0093] where F fu1 is a compressed mixed vector, GAP() represents global average pooling, and concat() represents concatenation along the channel, where F ir is the target visible light feature, F vi is the target infrared feature; then the mixed vector is compressed through two fully connected layers to obtain the infrared weight vector for fusion:
[0094]
[0095] where q inf represents the infrared weight vector used for fusion, q vis Represents the visible light weight vector used for fusion, FC a→b Indicates the fully connected layer that squeezes the weight channel from a to b. In this process, c represents the number of channels of each feature of the input feature fusion module, and r represents the scaling rate, which is set to 16. The visible light weight for fusion is obtained in the same way as the previous step. The channel weight vector is used to filter the infrared features on the channel:
[0096] F inf1 =q inf ☉F inf
[0097] Among them, F inf1 represents the infrared feature after channel screening, ⊙ represents channel-by-channel multiplication; the visible light feature is screened on the channel in the same way; the following formula is used for the second stage feature mixing:
[0098] F fu2 =concat(F vis +q vis ☉F vis , F inf +q inf ☉F inf )
[0099] Among them, F inf2 represents the mixed features obtained in the second stage; the spatial distribution of different modes is not independent of each other, so we use an MLP layer to transform F inf2 Compression is performed on the channel dimension to obtain a feature with a channel number of 2, and then this feature is separated on the channel to obtain the final spatial weight map:
[0100] MAP = conv(conv(F fu2 ))
[0101] MAP vis , MAP inf =splite(MAP)
[0102] Among them, MAP represents the features after MLP compression, conv() represents the convolution layer with a convolution kernel of 3, and splite() represents separation along the channel; the features after the first stage screening are fused in the second stage through the following formula:
[0103]
[0104] Among them, F fused Represents the fused features of the output fusion module.
[0105] S500, performing position regression processing on the fused features to obtain a detection frame of the car, and performing category regression on the fused features to obtain a detection category of the car, and determining the car detection result in the target area according to the detection frame and the detection category.
[0106] In some embodiments, reference Figure 4 The detection head shown is used to perform position regression processing on the fused features to obtain the detection frame of the car, and to perform category regression on the fused features to obtain the detection category of the car based on the detection frame and the detection category.
[0107] like Figure 4The detection head shown in the figure has the following workflow: copy the input feature into two copies, recorded as feature 1 and feature 2; extract feature 1 through three repeated convolution layers with a convolution kernel of 3 to obtain feature 1-1; extract feature 2 through three repeated convolution layers with a convolution kernel of 3 to obtain feature 2-1; use feature 1-1 to perform position regression to obtain a detection frame; use feature 2-1 to perform category regression to obtain a detection category.
[0108] In some embodiments, it can be understood that the detection box and the detection category refer to the vehicle in the target area.
[0109] In some embodiments, the vehicle detection based on drone aerial images is performed on complex scenes, where the complex scenes include but are not limited to rain, smoke, low light and other environments.
[0110] In some embodiments, the small vehicle detection is performed using a drone to detect small vehicles on the ground, and the number of pixels occupied by the small vehicle in the aerial image is less than 100.
[0111] The embodiment of the present invention uses visible light images and infrared images for detection at the same time, which improves the detection accuracy of the detection method in a complex interference environment. The present invention fully mixes the features of different modes by designing a feature enhancement module to improve the detection accuracy.
[0112] In some embodiments, reference Figure 5 The overall network architecture of a vehicle detection in a drone aerial image is shown. Based on this network architecture, the embodiment of the present invention also includes the following process:
[0113] In order to enable the detection model to detect ground vehicles using infrared and visible light aerial images, it is first necessary to use a drone to collect several pairs of infrared and visible light images. Preferably, the number of image pairs is 10,000. Then, the vehicles in these image pairs are manually annotated. Then, the images are quantitatively enhanced using mosaic enhancement technology. Subsequently, these images are input into the model proposed in the present invention, and the loss function is calculated using the following formula: Calculate and use the Adam optimizer to optimize the network parameters:
[0114]
[0115] Where L BCE represents the cross entropy loss, L REG represents the bounding box loss, preferably, α=1, β=1.
[0116] Where L BCE The specific expression formula is:
[0117] LBCE =-(ylog(p(x)+(1-y)log(1-p(x))))
[0118] Where p(x) is the category of the target box generated by the network, and y represents the true label of the object.
[0119] Where L REG The specific expression formula is:
[0120]
[0121] Where y represents the coordinate value of the box obtained by network regression, y i Represents the new coordinate position estimated based on the true label. The estimated coordinate position satisfies:
[0122]
[0123] in Indicates the coordinates of the real box of each car. In this way, when the edge of the car is not clear, the network can select the optimal prediction box through learning, instead of having to completely match the real box.
[0124] Figure 6 6 is a diagram of a vehicle detection and analysis device for drone aerial images according to an embodiment of the present invention. The device includes a first module 610, a second module 620, a third module 630, a fourth module 640 and a fifth module 650.
[0125] Among them, the first module is used to take aerial photos of the target area through a drone to obtain visible light images and infrared images of the target area; the second module is used to extract features from the visible light images and infrared images through dual-stream multi-scale information to obtain target infrared features and target visible light features; the third module is used to map, mix and compress the target infrared features and target visible light features each time the features are extracted to obtain infrared weight vectors and visible light weight vectors; the fourth module is used to filter the target infrared features through the infrared weight vector using channel splicing and global average pooling to obtain filtered infrared features, filter the visible light features through the visible light weight vector to obtain filtered visible light features, and perform cross-modal feature fusion on the filtered infrared features and the filtered visible light features to obtain fused features; the fifth module is used to perform position regression processing on the fused features to obtain the detection frame of the car, and to perform category regression on the fused features to obtain the detection category of the car, and determine the car detection result of the target area according to the detection frame and the detection category.
[0126] Exemplarily, with the cooperation of the first module in the device, the embodiment device can implement any of the aforementioned methods for detecting cars in drone aerial images, that is, taking aerial photos of the target area by a drone to obtain visible light images and infrared images of the target area; performing feature extraction on the visible light image and the infrared image through dual-stream multi-scale information to obtain target infrared features and target visible light features; in each feature extraction, mapping, mixing and compressing the target infrared features and the target visible light features to obtain infrared weight vectors and visible light weight vectors; using channel splicing and global average pooling to filter the target infrared features through the infrared weight vector to obtain filtered infrared features, filtering the visible light features through the visible light weight vector to obtain filtered visible light features, cross-modal feature fusion of the filtered infrared features and the filtered visible light features to obtain fused features; performing position regression processing on the fused features to obtain a detection frame of the car, and performing category regression on the fused features to obtain a detection category of the car, and determining the car detection result of the target area according to the detection frame and the detection category. The beneficial effects of the present invention are as follows: a dual-stream multi-scale information extraction method is adopted, feature enhancement is performed during extraction, and features of different modes are fully mixed to improve the detection accuracy of cars in rainy, snowy and foggy weather; and the detection accuracy of complex scenes such as hidden cars is improved by combining infrared and visible light for recognition.
[0127] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory;
[0128] The memory stores a program;
[0129] The processor executes the program to execute the aforementioned method for detecting the car in the drone aerial images; the electronic device has the function of carrying and running the software system for detecting the car in the drone aerial images provided by the embodiment of the present invention, for example, a personal computer, a minicomputer, a main frame, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicating with a charged particle tool or other imaging device, etc.
[0130] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method for detecting a vehicle in an aerial image taken by a drone as described above.
[0131] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0132] The embodiment of the present invention also discloses a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned vehicle detection method for drone aerial images.
[0133] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0136] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0137] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0138] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0139] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0140] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for detecting cars in drone aerial images. It is characterized in that include: Taking aerial photos of the target area by using a drone to obtain a visible light image and an infrared light image of the target area; Extracting features of the visible light image and the infrared light image through dual-stream multi-scale information to obtain target infrared features and target visible light features; During each feature extraction, the target infrared feature and the target visible light feature are mapped, mixed and compressed to obtain an infrared weight vector and a visible light weight vector; The infrared features of the target are filtered by using an infrared weight vector using channel splicing and global average pooling to obtain filtered infrared features, the visible light features are filtered by using a visible light weight vector to obtain filtered visible light features, and the filtered infrared features and the filtered visible light features are cross-modally fused to obtain fused features; The fused features are subjected to position regression processing to obtain a detection frame of the car, and the fused features are subjected to category regression to obtain a detection category of the car. The car detection result of the target area is determined according to the detection frame and the detection category.
2. The method for detecting a vehicle in an unmanned aerial vehicle aerial image according to claim 1, It is characterized in that The method further comprises: Performing a first feature extraction on the visible light image and the infrared light image to obtain a first infrared feature and a first visible light feature, and performing a first fusion of the first infrared feature and the first visible light feature to obtain a first fusion feature; Extracting the first infrared feature and the first visible light feature for a second time to obtain a second infrared feature and a second visible light feature, and fusing the second infrared feature and the second visible light feature for a second time to obtain a second fused feature; Extracting the second infrared feature and the second visible light feature for a third time to obtain a third infrared feature and a third visible light feature, and fusing the third infrared feature and the third visible light feature for a third time to obtain a third fused feature; The first fusion feature, the second fusion feature and the third fusion feature are extracted for a fourth time to obtain the first feature to be detected, the second feature to be detected and the third feature to be detected. The first extraction, the second extraction and the third extraction are extracted using a dual-stream multi-scale information method. The first fusion, the second fusion and the third fusion are feature fused using a cross-modal feature fusion method.
3. The method for detecting a vehicle in an unmanned aerial vehicle aerial image according to claim 2, It is characterized in that The extracting features of the visible light image and the infrared light image through dual-stream multi-scale information to obtain target infrared features and target visible light features includes: The input feature is copied into three copies to obtain the first feature, the second feature and the third feature, wherein the input feature is the feature obtained by the first extraction, the second extraction and the third extraction; The first feature is processed through a dilated convolution layer with a dilation rate of 1 to obtain a first sub-feature; the second feature is processed through a dilated convolution layer with a dilation rate of 3 to obtain a second sub-feature; the third feature is processed through a dilated convolution layer with a dilation rate of 5 to obtain a third sub-feature; The first sub-feature, the second sub-feature and the third sub-feature are spliced on the channel to obtain a fourth fused feature, and the fourth fused feature is processed through a convolution layer with a convolution kernel of 3 to obtain a target infrared feature and a target visible light feature.
4. The method for detecting a vehicle in an unmanned aerial vehicle aerial image according to claim 1, It is characterized in that The channel stitching and global average pooling are used to filter the target infrared features through the infrared weight vector to obtain the filtered infrared features, the visible light features are filtered through the visible light weight vector to obtain the filtered visible light features, and the filtered infrared features and the filtered visible light features are cross-modally fused to obtain the fused features, including: The target visible light feature and target infrared feature maps are mixed in the first stage by channel splicing and global average pooling to obtain a mixed vector. The formula is: F fu1 =GAP(concat(F ir F vi )) Among them, F fu1 is a compressed mixed vector, GAP() represents global average pooling, and concat() represents concatenation along the channel, where F ir is the target visible light feature, F vi is the target infrared signature; The mixed vector is compressed through two fully connected layers to obtain the infrared weight vector for fusion and the visible light weight vector for fusion. The formula is: where q inf represents the infrared weight vector used for fusion, q vis Represents the visible light weight vector used for fusion, FC a→b represents the fully connected layer that squeezes the weight channel from a to b, c represents the number of channels of each feature for feature fusion, and r represents the scaling rate.
5. The method for detecting a vehicle in an unmanned aerial vehicle aerial image according to claim 4, It is characterized in that The channel stitching and global average pooling are used to filter the infrared features through the infrared weight vector, the visible light features are filtered through the visible light weight vector, and the filtered infrared features are fused with the filtered visible light features to obtain fused features, including: The channel weight vector is used to filter the infrared features on the channel. The formula is F inf1 =q inf ☉F inf Among them, F inf1 represents the second infrared feature after channel screening, ⊙ represents channel-by-channel multiplication; The same method is used to screen the visible light features on the channel; The second stage feature blending is performed using the formula F fu2 =concat(F vis +q vis ☉F vis ,F inf +q inf ☉F inf ) Among them, F inf2 represents the mixed features obtained in the second stage; Through the MLP layer, F inf2 Compression is performed on the channel dimension to obtain a mixed feature with 2 channels. Then this feature is separated on the channel to obtain the final spatial weight map: MAP=conv(conv(F fu2 )) MAP vis ,MAP inf =splite(MAP) Among them, MAP represents the features after MLP compression, conv() represents the convolution layer with a convolution kernel of 3, and splite() represents separation along the channel; The features filtered in the first stage are fused in the second stage using the following formula: Among them, F fused Represents the output fusion features.
6. The method for detecting a vehicle in an unmanned aerial vehicle aerial image according to claim 5, It is characterized in that The performing position regression processing on the fused features to obtain a detection frame of the car, and the performing category regression on the fused features to obtain a detection category of the car, include: dividing the fused feature into a first fused feature and a second fused feature; The first fusion feature is subjected to feature extraction through three repeated convolution layers with a convolution kernel of 3 to obtain a third fusion feature, and the third fusion feature is subjected to position regression processing to obtain a detection frame; Extract the second fused feature through three repeated convolution layers with a convolution kernel of 3 to obtain a fourth fused feature, and perform category regression processing on the fourth fused feature to obtain a detection category; Through the loss function Constrain the detection box, where Where L BCE represents the cross entropy loss, L REG represents the bounding box loss, L BCE for L BCE =-(ylog(p(x)+(1-y)log(1-p(x)))) Where p(x) is the category of the target box generated by the network, and y represents the true label of the object; Where L REG The formula is Where y represents the coordinate value of the box obtained by network regression, y i Indicates the new coordinate position estimated based on the true label. in Represents the coordinates of the real box of each car.
7. The method for detecting a vehicle in an unmanned aerial vehicle aerial image according to claim 1, It is characterized in that The number of pixels of the vehicle in the visible light image or the infrared light image is less than or equal to 100.
8. A vehicle detection device for drone aerial images, It is characterized in that include: The first module is used to take aerial photos of the target area by using a drone to obtain a visible light image and an infrared light image of the target area; The second module is used to extract features of the visible light image and the infrared light image through dual-stream multi-scale information to obtain target infrared features and target visible light features; The third module is used to map, mix and compress the target infrared feature and the target visible light feature each time the feature is extracted to obtain an infrared weight vector and a visible light weight vector; The fourth module is used to filter the target infrared features through the infrared weight vector by using channel stitching and global average pooling to obtain filtered infrared features, filter the visible light features through the visible light weight vector to obtain filtered visible light features, and perform cross-modal feature fusion on the filtered infrared features and the filtered visible light features to obtain fused features; The fifth module is used to perform position regression processing on the fused features to obtain a detection frame of the car, and to perform category regression on the fused features to obtain a detection category of the car, and to determine the car detection result in the target area based on the detection frame and the detection category.
9. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method for detecting a vehicle in an unmanned aerial vehicle aerial image according to any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method for detecting a vehicle in an unmanned aerial vehicle aerial image according to any one of claims 1 to 7.
