Foreign body intrusion detection method, device and storage medium based on convolutional neural network
Through the improved YOLOv5 network, the multi-scale fusion of radar and visual features is solved, and the problems of sparse feature information loss and small target detection accuracy in the prior art are improved, and the perception ability and accuracy of foreign object intrusion detection are improved.
Patent Information
- Application Number
- CN202311454959.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-03
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-11-03
AI Technical Summary
The existing millimeter-wave radar and visual fusion detection methods will filter out most of the sparse radar feature information in foreign object intrusion detection, resulting in reduced perception and recognition capabilities and low detection accuracy of small targets.
Using the improved YOLOv5 network, the features of the radar projected image are extracted by adding new feature extraction branches and stitching modules, and the attention module is introduced in the feature fusion part to perform multi-scale fusion and feature extraction of radar and visual features to generate higher resolution result feature map.
It improves the detection and perception ability of invading targets, enhances the detection accuracy of small targets, makes full use of the advantages of radar and vision sensors, reduces noise and false alarm problems, and improves the reliability and detection performance of target recognition.
Smart Images

Figure CN117237613B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of foreign body intrusion detection, and in particular to a foreign body intrusion detection method, device and storage medium based on convolutional neural network. Background Art
[0002] Foreign Object Debris (FOD) on an airport runway usually refers to any object or substance found on the runway and surrounding areas that does not belong to the area. These foreign objects may include but are not limited to stones, debris, tools, parts, animals, stray pedestrians, etc. These foreign objects can cause a lot of damage to the aircraft's engine, landing gear and many other components, and will also pose a serious safety hazard to the aircraft's takeoff and landing. Every year, aircraft maintenance and flight delays caused by FOD are a huge loss.
[0003] Currently, due to cost and technical difficulty considerations, the mainstream FOD detection methods are mainly divided into three types: visual detection technology based on deep learning, FOD detection technology based on millimeter-wave radar, and multi-sensor hybrid detection methods.
[0004] An airport runway is a long strip of area paved with materials such as asphalt and concrete. In FOD monitoring, the detection distance varies from 0 to 100 meters, and the size of runway foreign objects can usually be as small as 1 to 2 cm. In visual sensors (such as high-definition cameras), the FOD target size is between 12 and 42 pixels, and the target size is mainly small. Visual sensors have some difficulties and limitations in solving the problem of FOD small target detection, which makes it impossible to completely solve the problem. The resolution of a visual sensor determines its ability to capture and represent detailed information in an image. For small targets, their features are usually weak and are easily affected by background noise and interference, making it difficult to distinguish and extract effective features. This may cause the target to be confused with the background, resulting in false detection or missed detection. Although target detection technology based on visual sensors has a wide range of applications in FOD detection, there are technical limitations in solving the problem of FOD small target detection. At present, the performance of most FOD detection algorithms based on visual sensors is more suitable for medium-sized targets, and there are challenges in the detection accuracy of small targets.
[0005] At the same time, under extreme weather conditions (such as heavy rain, heavy snow and fog), the appearance of obstacles will become blurred, resulting in a significant reduction in the detection performance of the FOD detection algorithm based on visual sensors; at the same time, due to changes in lighting and the similarity of FOD to the environment color, missed detection may occur. Due to the special environment of the airport itself, relying solely on a single visual sensor technology for FOD detection cannot meet actual needs.
[0006] Millimeter-wave radar has high-precision and accurate target location detection capabilities and is not easily affected by weather conditions. However, since millimeter-wave radar cannot provide color characteristics and contour information of the target, millimeter-wave radar signals may be affected by multipath effects during propagation, that is, multiple arrival times caused by reflection and refraction of signals through different paths, which may lead to false targets or misdetection in complex environments. The method of relying solely on millimeter-wave radar for FOD detection has limited effect.
[0007] Regarding the advantages and disadvantages of radar detection technology and optical image detection technology, multi-sensor hybrid detection technology uses radar, visible light, infrared and other multi-sensor technologies to effectively improve the detection accuracy of FOD under different conditions. In the millimeter-wave radar and visual fusion detection method based on decision-level fusion, the millimeter-wave radar collects radar data and predicts FOD by analyzing point cloud data through clustering algorithm; at the same time, the visual sensor collects video images and inputs them into the FOD target detection network to obtain the area of interest of FOD and extract features, and detect the area of interest, predict FOD targets and obtain the confidence that the target is FOD; then, with the help of coordinate transformation method, the targets predicted by radar and visual detection parts are projected into the same coordinate system; finally, the detection results after conversion of the two sensors are fused and weighted to obtain the final detection result. The decision-level fusion detection method can reduce the possible wrong decisions made by a single sensor by integrating the decision results of different sensors, and improve the overall decision accuracy.
[0008] In the millimeter-wave radar and vision fusion method based on data-level fusion, the millimeter-wave radar obtains the position and distance information of the FOD; with the help of the coordinate conversion method, the radar points are projected onto the visual image; the visual partial target detection algorithm obtains the corresponding FOD region of interest in the image based on the projected radar points; finally, the corresponding area of the visual image is extracted based on the ROI, and the feature extractor and classifier are used to perform target detection on these images. The data-level fusion solution can reduce the search area in the camera coordinates based on the information of the millimeter-wave radar, thereby reducing computing resources.
[0009] The existing FOD radar-visual fusion detection methods all focus on the fusion of decision-level and data-level. The decision-level fusion method usually adopts the traditional filtering-based strategy, which processes the radar and image data separately and fuses their results to obtain the final result; the data-level fusion method generates the region of interest (ROI) of the target object through the data collected by a sensor (in most cases, the radar sensor), and then extracts the corresponding area of the visual image based on the ROI. Both of the above FOD radar-visual fusion strategies may ignore the complementary information between sensors, resulting in the loss or underutilization of information; at the same time, they do not consider that by fusing the feature information of different sensors, the robustness of the system can be enhanced and the adaptability to complex scenes such as illumination changes and occlusion can be improved.
[0010] Most of these hybrid detection methods integrate the data of different sensors and weight the detection results of multiple sensors to obtain the final detection results. This decision-level hybrid detection method directly relies on the decision results of the sensor and may lose some feature information. This may lead to the inability to fully utilize the information provided by each sensor in some cases, reducing the perception and recognition capabilities of the system. At the same time, there may be certain differences in the decision results of different sensors. The decision-level hybrid detection method needs to solve the problem of decision consistency to ensure the reliability of the overall decision results. The multi-sensor data-level fusion detection method simply merges the raw data of different sensors, does not fully utilize the feature information of the data, and does not perform feature extraction or optimization on the data, which may lead to information redundancy, noise accumulation and inconsistency. The results of data-level fusion are also affected by the quality of each sensor. If one of the sensors has poor performance or fails, it may have a negative impact on the entire fusion result. In addition, the performance differences of different sensors may also cause the results of data-level fusion to be biased towards a certain sensor, reducing the accuracy and robustness of fusion. Summary of the invention
[0011] The purpose of the present invention is to provide a foreign body intrusion detection method, device and storage medium based on convolutional neural network, so as to solve the problem that in the existing method of detecting foreign body intrusion by fusion of millimeter wave radar and vision, most of the sparse radar feature information will be filtered out during the intrusion target detection process by decision-level fusion and data-level fusion, resulting in the inability to fully utilize the information provided by each sensor, reducing the perception and recognition capabilities, and the low small target detection accuracy of the existing foreign body intrusion detection algorithm.
[0012] The present invention solves the above technical problems through the following technical solutions: a foreign body intrusion detection method based on convolutional neural network, comprising the following steps:
[0013] Acquire radar point cloud data and visual images of intrusion targets;
[0014] Performing time alignment on the radar point cloud data and the visual image to obtain multiple groups of radar point cloud data and visual images;
[0015] The radar point cloud data in each group are processed to obtain a radar projection image, where the radar projection image in each group is aligned with the visual image in time and space and conforms to the input mode of the target detection model;
[0016] Constructing a target detection model, wherein the target detection model adopts an improved YOLOv5 network, wherein the improved YOLOv5 network adds a feature extraction branch in the feature extraction part and adds a splicing module in the original feature extraction branch, wherein the newly added splicing module includes a first splicing module, a second splicing module, and a third splicing module; and adds an attention module and a feature fusion module in the feature fusion part, wherein the newly added attention module includes a first attention module, a second attention module, and a third attention module;
[0017] The newly added feature extraction branch is used to extract the features of the radar projection image to obtain radar feature maps of different scales; the original feature extraction branch is used to extract the features of the visual image to obtain visual feature maps of different scales; the newly added splicing module is used to splice the radar feature map and the visual feature map corresponding to the scale to obtain a fused feature map; the newly added feature fusion module and the original feature fusion module are both used to extract features, sample, and splice the fused feature map output by the newly added splicing module; the newly added attention module is used to extract features and splice the fused feature map output by the newly added feature fusion module and the original feature fusion module, and the radar feature maps of different scales to obtain the result feature map;
[0018] The radar projection image and the visual image of each group are simultaneously input into the target detection model, and the radar projection image and the visual image are detected by using the target detection model to obtain the intrusion target image.
[0019] Furthermore, the radar point cloud data and the visual image are time aligned, specifically including:
[0020] Adding a timestamp to the radar point cloud data and the visual image to record the sampling time point of the radar point cloud data and the visual image;
[0021] The radar point cloud data and the visual image are aligned according to the timestamp to obtain multiple sets of time-aligned radar point cloud data and visual images.
[0022] Furthermore, the radar point cloud data includes location information and status information of the intrusion target, and the radar point cloud data in each group is processed, specifically including:
[0023] Convert the position of each point cloud in the radar point cloud data from the radar coordinate system to the camera coordinate system;
[0024] Project the position of each point cloud in the camera coordinate system to the pixel coordinate system to obtain the position of each point cloud in the pixel coordinate system to achieve spatial alignment;
[0025] Map the point cloud data in the pixel coordinate system to the image plane to generate an RGB mask image;
[0026] The pixel values of the RGB three channels are calculated according to the state information of the intrusion target. The specific calculation formula is:
[0027]
[0028] Among them, r is the pixel value of the R channel, g is the pixel value of the G channel, and b is the pixel value of the B channel. d i is the spatial distance from the point cloud to the radar, v i is the radial velocity of the point cloud relative to the radar, p i is the signal-to-noise ratio of the point cloud;
[0029] The pixel values of the RGB three channels are used to assign values to the pixel points in the RGB mask image to obtain the radar projection image.
[0030] Furthermore, the point cloud data in the pixel coordinate system is mapped to the image plane in the form of a circle, a rectangle or a triangle. When mapped to the image plane in the form of a circle, the radius of the circle is:
[0031] ;
[0032] Among them, radius is the radius of the circle, Z c is the spatial distance from the intrusion target to the visual sensor.
[0033] Furthermore, the improved YOLOv5 network includes a feature extraction part, a feature fusion part and a feature detection part, and the feature detection part includes a first feature detection module, a second feature detection module and a third feature detection module;
[0034] The newly added feature extraction branch and the original feature extraction branch both include a first scale feature extraction module, a second scale feature extraction module and a third scale feature extraction module, the first scale feature extraction module, the second scale feature extraction module and the third scale feature extraction module of the newly added feature extraction branch are connected in sequence, a first splicing module is provided between the first scale feature extraction module and the second scale feature extraction module of the original feature extraction branch, a second splicing module is provided between the second scale feature extraction module and the third scale feature extraction module of the original feature extraction branch, a third splicing module is provided between the third scale feature extraction module of the original feature extraction branch and the spatial pyramid pooling layer, and the first scale feature extraction module of the newly added feature extraction branch is also connected to the first splicing module and the first attention module, the second scale feature extraction module of the newly added feature extraction branch is also connected to the second splicing module and the second attention module, and the third scale feature extraction module of the newly added feature extraction branch is also connected to the third splicing module and the third attention module;
[0035] The first splicing module is connected to the newly added feature fusion module, the second splicing module and the spatial pyramid pooling layer are respectively connected to the original feature fusion module, the newly added feature fusion module is connected to the original feature fusion module, the newly added feature fusion module is also connected to the first attention module, and the original feature fusion module is connected to the second attention module and the third attention module; the first attention module is connected to the first feature detection module, the second attention module is connected to the second feature detection module, and the third attention module is connected to the third feature detection module.
[0036] Furthermore, the target detection model is used to detect the radar projection image and the visual image, specifically including:
[0037] Using the newly added feature extraction branch to perform feature extraction on the radar projection image, and sequentially obtain a first-scale radar feature map, a second-scale radar feature map, and a third-scale radar feature map;
[0038] Using the original feature extraction branch to extract features from the visual image, and sequentially obtain a first-scale visual feature map, a second-scale visual feature map, and a third-scale visual feature map;
[0039] The first scale radar feature map and the first scale visual feature map are spliced by the first splicing module to obtain a first fused feature map; the second scale radar feature map and the second scale visual feature map are spliced by the second splicing module to obtain a second fused feature map; the third scale radar feature map and the third scale visual feature map are spliced by the third splicing module to obtain a third fused feature map;
[0040] The third fused feature map is processed by the spatial pyramid pooling layer and then the original feature fusion module is used to perform feature extraction and upsampling to obtain a fourth fused feature map; the second fused feature map and the fourth fused feature map are spliced and feature extracted using the original feature fusion module to obtain a fifth fused feature map;
[0041] Using the newly added feature fusion module to perform feature extraction and upsampling on the fifth fused feature map to obtain a sixth fused feature map, and using the newly added feature fusion module to perform splicing and feature extraction on the first fused feature map and the sixth fused feature map to obtain a seventh fused feature map;
[0042] Using the newly added feature fusion module to extract features from the seventh fused feature map, to obtain an eighth fused feature map; using the newly added feature fusion module to splice the eighth fused feature map and the feature map obtained by the convolution layer processing of the fifth fused feature map, to obtain a ninth fused feature map;
[0043] Using the original feature fusion module to perform feature extraction on the ninth fusion feature map to obtain a tenth fusion feature map;
[0044] The feature map obtained by the spatial pyramid pooling layer and the tenth fused feature map are processed by the convolution layer respectively, and then spliced and feature extracted to obtain the eleventh fused feature map;
[0045] Using the first attention module to extract features from the first scale radar feature map and the seventh fusion feature map, and splicing them to obtain a first result feature map; using the second attention module to extract features from the second scale radar feature map and the tenth fusion feature map, and splicing them to obtain a second result feature map; using the third attention module to extract features from the third scale radar feature map and the eleventh fusion feature map, and splicing them to obtain a third result feature map;
[0046] Feature detection is performed on the first result feature map, the second result feature map, and the third result feature map, respectively, to obtain a first intrusion target image, a second intrusion target image, and a third intrusion target image.
[0047] Furthermore, the first attention module, the second attention module and the third attention module each include a first convolution module, a second convolution module, a first adder, a second adder, a third adder and a multiplier; the first convolution module and the second convolution module each include three parallel convolution layers; the output end of the first convolution module is connected to the first adder, the output end of the second convolution module is connected to the second adder, the first adder and the second adder are respectively connected to the third adder, and the input end of the multiplier is connected to the output end of the third adder, the output end of the newly added feature fusion module or the output end of the original feature fusion module.
[0048] Furthermore, the loss function of the improved YOLOv5 network adopts a border regression loss function, and the specific expression is:
[0049]
[0050]
[0051] in, L is the loss value, t is the coordinate axis, All are intermediate quantities. is the center point coordinate of the real frame, is the center point coordinate of the predicted box, IoU is the intersection over union ratio of the real box and the predicted box, a is a positive balance parameter, s is the consistency of the aspect ratio between the predicted box and the real box, are the width and height of the minimum bounding rectangle of the real box and the predicted box, respectively. w g and h g are the width and height of the real frame respectively, w and h are the width and height of the prediction box respectively, An intermediate amount.
[0052] Based on the same concept, the present invention also provides an electronic device, the device comprising:
[0053] Memory for storing computer programs;
[0054] A processor is used to implement the steps of the foreign body intrusion detection method based on convolutional neural network as described above when executing the computer program.
[0055] Based on the same concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the foreign body intrusion detection method based on convolutional neural network as described above are implemented.
[0056] Beneficial Effects
[0057] Compared with the prior art, the advantages of the present invention are:
[0058] The present invention converts radar point cloud data into radar projection images, adds a feature extraction branch for extracting radar projection image features in the improved YOLOv5 network, makes full use of radar feature information, and fuses radar feature maps at multiple scales in the feature extraction stage; before the feature detection stage, generates a result feature map (radar visual space attention map) through an attention module, thereby improving the detection and perception capabilities of intrusion targets; the improved YOLOv5 network uses higher resolution feature maps for detection, thereby improving the detection accuracy of small targets.
[0059] The present invention integrates the characteristic information of millimeter-wave radar and visual sensor, which can fully utilize the advantages of millimeter-wave radar and visual sensor, has more comprehensive and accurate target perception and recognition capabilities, reduces the problems caused by noise and false alarms of a single sensor, improves the reliability of target recognition, and improves the perception capability and detection performance as a whole. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 is a flow chart of the FOD detection method in an embodiment of the present invention;
[0062] Figure 2 is a flow chart of radar point cloud data processing in an embodiment of the present invention;
[0063] Figure 3 It is a diagram of the improved YOLOv5 network structure in an embodiment of the present invention;
[0064] Figure 4 It is a structural diagram of the RCAM module in an embodiment of the present invention. DETAILED DESCRIPTION
[0065] The following is a clear and complete description of the technical solutions in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0067] The embodiment of the present invention takes the airport runway foreign object intrusion (i.e. FOD) as an example to illustrate the specific steps of the foreign object intrusion detection method based on the convolutional neural network. Figure 1 As shown, the FOD detection method based on convolutional neural network provided in this embodiment includes the following steps:
[0068] Step 1: Obtain radar point cloud data and visual images of the FOD target.
[0069] This embodiment takes the airport runway FOD target detection as an example, integrates the millimeter wave radar and the visual sensor into the same device, and installs the device outside the airport runway (usually the device is about 3 meters away from the edge of the airport runway), and the device is perpendicular to the runway. The device also includes a turntable, which can be controlled to achieve scanning detection within the runway range, with a scanning range of -135° to 135°.
[0070] In this embodiment, the visual sensor is a high-definition camera, which is used to collect visual images of the airport runway, and a millimeter-wave radar is used to collect radar point cloud data of the locomotive runway. The radar point cloud data includes the position information and status information of the FOD target. The position of each point cloud in the radar point cloud data is based on the radar coordinate system, and the status information includes the spatial distance from the FOD target to the radar, the radial velocity of the FOD target relative to the radar, and the signal-to-noise ratio of the point cloud.
[0071] Step 2: Time-align the radar point cloud data and visual images to obtain multiple groups of radar point cloud data and visual images.
[0072] In order to better integrate the characteristic information of radar and visual sensors and improve the detection accuracy of FOD targets, a set of radar point cloud data and visual images that need to be integrated should be aligned in time and space, that is, each set of radar point cloud data and visual images collect FOD targets at the same time point and collect them in the same space. In this embodiment, the radar point cloud data and visual images are time-aligned, specifically including:
[0073] Step 2.1: Add timestamps to the radar point cloud data and visual images to record the sampling time points of the radar point cloud data and visual images;
[0074] Step 2.2: Align the radar point cloud data with the visual image according to the timestamp to obtain multiple sets of time-aligned radar point cloud data and visual images.
[0075] In step 2.1, the system's hardware clock or other time synchronization devices can be used to obtain accurate timestamps. In order to ensure that the FOD targets collected by the radar and visual sensors are aligned in time, it is necessary to ensure that the sampling times of the radar and visual sensors are aligned within a certain range. Therefore, when collecting data, it is necessary to record the timestamp of each frame of data, and the alignment of the radar point cloud data and the visual image is achieved based on the timestamp.
[0076] Step 3: Process the radar point cloud data in each group to obtain a radar projection image.
[0077] Step 2 obtains the time-aligned radar point cloud data and visual images. For a set of time-aligned radar point cloud data and visual images, since the radar point cloud data cannot be directly input into the target detection model for feature extraction, it is also necessary to process the radar point cloud data to obtain the RGB three-channel radar projection image, which is input into the target detection model at the same time as the visual image for FOD target detection.
[0078] In order to achieve spatial matching between the FOD targets in the radar point cloud data and the FOD targets in the visual image, it is necessary to first convert the radar point cloud data from the radar coordinate system to the pixel coordinate system by means of a coordinate conversion method, and then convert the point cloud data in the pixel coordinate system into a radar projection image to adapt to the input mode of the target detection model. In this embodiment, the radar point cloud data in each group is processed to obtain a radar projection image, such as Figure 2 As shown, specifically including:
[0079] Step 3.1: With the help of the rotation and translation matrix, the position of each point cloud in the radar point cloud data is converted from the radar coordinate system to the camera coordinate system. The specific conversion formula is:
[0080] (1)
[0081] Among them, X r , Y r , Z r is the position of the point cloud in the radar coordinate system, X c , Y c , Z c is the position of the point cloud in the camera coordinate system, d x , d y , d z are the translation amounts of the point cloud from the radar coordinate system to the camera coordinate system on the X, Y, and Z coordinate axes, respectively. is the rotation matrix. The calculation process of the rotation matrix is: first measure the three rotation angles around the X, Y, and Z axes of the radar coordinate system, and substitute the three rotation angles measured into the three matrices respectively: , , , and then calculate the rotation matrix of the corresponding angles around the X, Y, and Z coordinate axes respectively, and finally multiply the three rotation matrices to get the final rotation matrix .
[0082] Step 3.2: With the help of the camera intrinsic parameter matrix, project the position of each point cloud in the camera coordinate system to the pixel coordinate system to obtain the position of each point cloud in the pixel coordinate system to achieve spatial alignment. The specific projection formula is:
[0083] (2)
[0084] Among them, Z c is the spatial distance from the FOD target to the HD camera, u , v is the position of the point cloud in the pixel coordinate system, f x , f y represents the scaling factor of the focal length on the image plane, c x , c y Represents the pixel coordinates of the principal point, which is the intersection of the image diagonals.
[0085] Step 3.3: Map the point cloud data in the pixel coordinate system to the image plane to generate an RGB mask image.
[0086] The point cloud data in the pixel coordinate system can be mapped to the image plane in the form of a circle, rectangle, triangle, etc. This embodiment uses a circle as the projection form of the point cloud data, and the state information of the FOD target detected by the radar is expressed as the pixel value of the point cloud projection point. The point cloud data in the pixel coordinate system is mapped to the image plane in the form of a circle to generate an RGB mask image. Figure 2 Where H is the height of the RGB mask image, and W is the width of the RGB mask image. Considering the principle that objects near the image are larger and far away are smaller, this embodiment uses the spatial distance Z from the FOD object to the high-definition camera as c Calculate the radius of the circle. The area of the circle is the mapping range of the point cloud. The larger the radius, the larger the range represented by the corresponding point cloud. The radius of the circle is:
[0087] (3)
[0088] Here, radius is the radius of the circle.
[0089] Step 3.4: Calculate the pixel values of the RGB three channels according to the status information of the FOD target. The specific calculation formula is:
[0090] (4)
[0091] (5)
[0092] (6)
[0093] Among them, r is the pixel value of the R channel, g is the pixel value of the G channel, and b is the pixel value of the B channel. d i is the spatial distance from the point cloud to the radar, v i is the radial velocity of the point cloud relative to the radar, p i is the signal-to-noise ratio of the point cloud.
[0094] Three state information are selected from the state information such as spatial distance, radial velocity, signal-to-noise ratio, and scattering cross section detected by the radar to assign RGB three-channel pixel values to obtain a radar projection image. This embodiment uses spatial distance, radial velocity, and signal-to-noise ratio to assign RGB three-channel pixel values.
[0095] Step 3.5: Use the pixel values of the RGB three channels to assign values to the pixel points in the RGB mask image to obtain the radar projection image.
[0096] According to equations (4) to (6), the state information of each point cloud is converted into the pixel values of the three channels of the projection circle RGB corresponding to the point cloud. That is, the calculated pixel values are assigned to all pixel points within the range of each channel circle, and the pixel values at the positions where there is no point cloud mapping are set to 0, and finally the radar projection image is obtained.
[0097] The radar projection image and the corresponding visual image are aligned in time and space, and conform to the input mode of the target detection model. In order to ensure that the radar projection image read by the target detection model corresponds to the visual image, the radar projection image and the visual image aligned in time and space are generated using the same name but different formats, that is, the file name is the same but the image file format is different, such as the visual image and the radar projection image are in the form of 0.jpg and 0.png respectively. When detecting FOD targets, two visual images and radar projection images aligned in time and space are read at the same time according to the file name.
[0098] Step 4: Build a target detection model. The target detection model uses an improved YOLOv5 network.
[0099] The traditional YOLOv5 network solidifies the 3-channel image input and can only read one image. In order to make full use of the feature information of radar and visual sensors, the target detection model of the present invention needs to read a set of radar projection images and visual images at the same time, so the traditional YOLOv5 network is improved. Figure 3 As shown in the figure, the improved YOLOv5 network adds a feature extraction branch in the feature extraction part. The original feature extraction branch is used to extract the features of the visual image to obtain visual feature maps of different scales; the new feature extraction branch is used to extract the features of the radar projection image to obtain radar feature maps of different scales. In order to fuse the features of the radar projection image and the visual image at multiple scales and improve the model perception ability, a new splicing module is added to the original feature extraction branch. The new splicing module includes a first splicing module, a second splicing module and a third splicing module. The new splicing module is used to splice the radar feature map corresponding to the scale with the visual feature map to obtain a fused feature map. In order to improve the detection performance of the model, enhance the resistance to interference, and enhance the stability and reliability of FOD detection, the improved YOLOv5 network introduces an attention module before the feature detection stage, that is, a new attention module (Radar-CameraAttention Module, RCAM) is added to the feature fusion part. The new attention module includes the first attention module, the second attention module, and the third attention module. The new attention module extracts and splices the fusion feature map output by the new feature fusion module and the original feature fusion module, and the radar feature map of different scales to obtain the result feature map. In order to improve the detection accuracy of small targets, the feature extraction, fusion, and detection parts of the original large targets are cut off, and a new feature fusion module (i.e., small target detection) is added to the feature fusion part. While improving the detection accuracy of small targets, the amount of network reasoning calculations is effectively reduced, and the FOD detection speed is improved.
[0100] like Figure 3 As shown, the improved YOLOv5 network includes a feature extraction part, a feature fusion part and a feature detection part, and the feature detection part includes a first feature detection module, a second feature detection module and a third feature detection module.
[0101] Both the newly added feature extraction branch and the original feature extraction branch include a first scale feature extraction module, a second scale feature extraction module and a third scale feature extraction module. The first scale feature extraction module, the second scale feature extraction module and the third scale feature extraction module of the newly added feature extraction branch are connected in sequence. A first splicing module is provided between the first scale feature extraction module and the second scale feature extraction module of the original feature extraction branch. A second splicing module is provided between the second scale feature extraction module and the third scale feature extraction module of the original feature extraction branch. A third splicing module is provided between the third scale feature extraction module of the original feature extraction branch and the spatial pyramid pooling layer (Spatial Pyramid Pooling – Fast, SPPF). The first scale feature extraction module of the newly added feature extraction branch is also connected to the first splicing module and the first attention module. The second scale feature extraction module of the newly added feature extraction branch is also connected to the second splicing module and the second attention module. The third scale feature extraction module of the newly added feature extraction branch is also connected to the third splicing module and the third attention module. The first splicing module is connected to the newly added feature fusion module, the second splicing module and the spatial pyramid pooling layer are respectively connected to the original feature fusion module, the newly added feature fusion module is connected to the original feature fusion module, the newly added feature fusion module is also connected to the first attention module, and the original feature fusion module is connected to the second attention module and the third attention module; the first attention module is connected to the first feature detection module, the second attention module is connected to the second feature detection module, and the third attention module is connected to the third feature detection module.
[0102] like Figure 3As shown, the first-scale feature extraction modules of the newly added feature extraction branch and the original feature extraction branch are both two convolutional layers Conv and 3 C3 modules, the second-scale feature extraction modules of the newly added feature extraction branch and the original feature extraction branch are both 1 convolutional layer Conv and 6 C3 modules, the third-scale feature extraction modules of the newly added feature extraction branch and the original feature extraction branch are both 1 convolutional layer Conv and 9 C3 modules, and the first splicing module, the second splicing module and the third splicing module are all splicing layers Concat. The original feature fusion module includes a first branch and a second branch. The first branch includes a convolutional layer Conv, an upsampling layer Upsample, a splicing layer Concat and three C3 modules connected in sequence. The second branch includes three C3 modules, a convolutional layer Conv, a splicing layer Concat and three C3 modules connected in sequence. The new feature fusion module includes a third branch and a fourth branch. The third branch includes a convolutional layer Conv, an upsampling layer Upsample, a splicing layer Concat and three C3 modules connected in sequence. The fourth branch includes a convolutional layer Conv and a splicing layer Concat connected in sequence. The spatial pyramid pooling layer SPPF is connected to the convolutional layer Conv of the first branch, the second splicing module Concat2 is connected to the splicing layer Concat of the first branch, the first splicing module Concat1 is connected to the splicing layer Concat of the third branch, the convolutional layer Conv of the first branch is also connected to the splicing layer Concat of the second branch, and the convolutional layer Conv of the third branch is also connected to the splicing layer Concat of the fourth branch. The C3 module of the third branch is also connected to the first attention module RCAM1, the C3 module at the head end of the second branch is also connected to the second attention module RCAM2, and the C3 module at the end of the second branch is also connected to the third attention module RCAM3.
[0103] In this embodiment, the newly added feature extraction branch has the same network structure as the original feature extraction branch, and both use the feature extraction network of the traditional YOLOv5. In another embodiment of the present invention, the original feature extraction branch uses the feature extraction network of the traditional YOLOv5, and the newly added feature extraction branch uses the ResNet network (i.e., residual neural network), VGGNet network, or a radar feature extraction network constructed for sparse radar point cloud data. In this embodiment, the first splicing module, the second splicing module, and the third splicing module all use the Concat operation. In another embodiment of the present invention, the first splicing module, the second splicing module, and the third splicing module can also use fusion operations such as Add or mul.
[0104] Both the radar projection image and the visual image have the spatial features of the FOD target in the pixel plane. The present invention designs an attention module RCAM and introduces the RCAM module before the feature detection part. Figure 4As shown, the first attention module, the second attention module and the third attention module all include a first convolution module, a second convolution module, a first adder, a second adder, a third adder and a multiplier; the first convolution module and the second convolution module all include three parallel convolution layers (i.e., Conv1×1, Conv3×3, Conv5×5); the output end of the first convolution module is connected to the first adder, the output end of the second convolution module is connected to the second adder, the first adder and the second adder are respectively connected to the third adder, and the input end of the multiplier is connected to the output end of the third adder, the output end of the newly added feature fusion module or the output end of the original feature fusion module.
[0105] In another embodiment of the present invention, the first attention module, the second attention module and the third attention module may also adopt a classic spatial attention module or an attention module that integrates the spatial attention mechanism and the channel attention mechanism, and utilize radar features to enhance the ability to focus on the target.
[0106] The radar feature maps of different scales and the fused feature maps output by the newly added feature fusion module or the original feature fusion module are simultaneously input into the RCAM module. The radar feature map of C×H×W size is generated through three parallel convolutional layers Conv1×1, Conv3×3, and Conv5×5 to generate three 1×H×W feature maps. Then, the three feature maps are added pixel by pixel through the first adder to obtain a 1×H×W radar spatial attention feature map. Secondly, the fused feature map output by the newly added feature fusion module or the original feature fusion module is generated through three parallel convolutional layers Conv1×1, Conv3×3, and Conv5 ×5 generates three 1×H×W feature maps, and then adds the three feature maps pixel by pixel through the second adder to obtain a 1×H×W visual space attention feature map; the radar space attention feature map and the visual space attention feature map are simultaneously input into the third adder, and the final radar space attention feature map is obtained by weighting; finally, the 1×H×W radar space attention feature map is multiplied with the fusion feature map output by the newly added feature fusion module or the original feature fusion module to obtain the final result feature map used for detection, so that the model pays more attention to the target area and improves the detection and perception ability of the visual branch to the FOD target. The RCAM module improves the detection and perception ability of the visual branch to the FOD through the radar weighted attention mechanism.
[0107] Step 5: Input each group of radar projection images and visual images into the target detection model at the same time, and use the target detection model to detect the radar projection image and the visual image to obtain the FOD target image.
[0108] In this embodiment, the radar projection image and the visual image are detected using the target detection model, specifically including:
[0109] Step 5.1: Use the newly added feature extraction branch to extract features from the radar projection image, and obtain the first-scale radar feature map, the second-scale radar feature map, and the third-scale radar feature map in turn.
[0110] Step 5.2: Use the original feature extraction branch to perform feature extraction on the visual image, and obtain a first-scale visual feature map, a second-scale visual feature map, and a third-scale visual feature map in sequence.
[0111] The newly added feature extraction branch is connected in parallel with the original feature extraction branch, and the first scale feature extraction module, the second scale feature extraction module and the third scale feature extraction module of the newly added feature extraction branch are used to extract features of the radar projection image, and the first scale radar feature map, the second scale radar feature map and the third scale radar feature map are obtained in sequence; the first scale feature extraction module, the second scale feature extraction module and the third scale feature extraction module of the original feature extraction branch are used to extract features of the visual image, and the first scale visual feature map, the second scale visual feature map and the third scale visual feature map are obtained in sequence.
[0112] Step 5.3: Use the first splicing module to splice the first-scale radar feature map and the first-scale visual feature map to obtain the first fused feature map P1 (i.e., the fused feature map of the first scale); use the second splicing module to splice the second-scale radar feature map and the second-scale visual feature map to obtain the second fused feature map P2 (i.e., the fused feature map of the second scale); use the third splicing module to splice the third-scale radar feature map and the third-scale visual feature map to obtain the third fused feature map P3 (i.e., the fused feature map of the third scale).
[0113] Step 5.4: The third fused feature map P3 is processed by the spatial pyramid pooling layer SPPF and then the original feature fusion module is used for feature extraction and upsampling (i.e., the convolution layer Conv and the upsampling layer Upsample of the first branch) to obtain the fourth fused feature map P4; the second fused feature map P2 and the fourth fused feature map P4 are concatenated and feature extracted using the original feature fusion module (i.e., the concatenation layer Concat and 3 C3 modules of the first branch) to obtain the fifth fused feature map P5;
[0114] Step 5.5: Use the newly added feature fusion module to extract features and upsample the fifth fused feature map (i.e., corresponding to the convolution layer Conv and upsampling layer Upsample of the third branch) to obtain the sixth fused feature map P6. Use the newly added feature fusion module to splice and extract features from the first fused feature map P1 and the sixth fused feature map P6 (i.e., corresponding to the splicing layer Concat and 3 C3 modules of the third branch) to obtain the seventh fused feature map P7.
[0115] Step 5.6: Use the newly added feature fusion module to extract features from the seventh fused feature map P7 (i.e., the convolution layer Conv corresponding to the fourth branch) to obtain the eighth fused feature map P8; use the newly added feature fusion module to splice the feature maps obtained by processing the eighth fused feature map P8 and the fifth fused feature map P5 through the convolution layer (i.e., the convolution layer Conv corresponding to the third branch) (i.e., the splicing layer Concat corresponding to the fourth branch) to obtain the ninth fused feature map P9.
[0116] Step 5.7: Use the original feature fusion module to extract features from the ninth fusion feature map P9 (i.e., corresponding to the three C3 modules and the convolution layer Conv of the second branch), and obtain the tenth fusion feature map P10;
[0117] Step 5.8: The feature map obtained by the spatial pyramid pooling layer SPPF and the tenth fused feature map P10 are processed by the convolution layer respectively and then spliced and feature extracted (i.e., corresponding to the concatenation layer Concat and 3 C3 modules of the second branch) to obtain the eleventh fused feature map P11.
[0118] Step 5.9: Use the first attention module RCAM1 to extract features from the first scale radar feature map and the seventh fusion feature map P7 and splice them to obtain a first result feature map; use the second attention module RCAM2 to extract features from the second scale radar feature map and the tenth fusion feature map P10 and splice them to obtain a second result feature map; use the third attention module RCAM3 to extract features from the third scale radar feature map and the eleventh fusion feature map P11 and splice them to obtain a third result feature map;
[0119] Step 5.10: Use the first feature detection module Detect1, the second feature detection module Detect2 and the third feature detection module Detect3 to perform feature detection on the first result feature map, the second result feature map and the third result feature map respectively to obtain the first FOD target image, the second FOD target image and the third FOD target image.
[0120] In the feature extraction part, the radar feature map corresponding to the scale is fused with the visual feature map to achieve multi-scale fusion, and the corresponding scale fusion feature map (i.e., P1~P3) is obtained, which improves the network perception ability. In the feature fusion part, the RCAM module can simultaneously obtain the radar spatial attention map and the visual spatial attention map, and then obtain the radar visual spatial attention map through weighted summation, and then multiply it with the corresponding fusion feature map (i.e., the seventh fusion feature map P7, the tenth fusion feature map P10 and the eleventh fusion feature map P11), thereby improving the visual branch detection performance. In view of the small size of FOD targets, the small proportion of camera imaging, and the low detection accuracy of small targets caused by the traditional YOLOv5 network, the traditional YOLOv5 network is improved. In the field of FOD detection, it mainly detects medium and small-sized targets, and there is no need to use deep convolution to extract the detailed feature information of large targets. Therefore, in the feature extraction part, the feature extraction block Conv+C3*3 ( Figure 4 In the feature fusion part, the bottom-up Conv+UpSample+Concat+C3*3 module (not shown in the figure) of the large target is cut off accordingly, which reduces the network parameters and speeds up the FOD detection. In the feature fusion part, a new feature fusion module is added to fuse the first fusion feature map P1 (i.e., the fusion feature map of the first scale) through the Concat operation of the third branch, and then the seventh fusion feature map P7 (i.e., the fusion feature map of the first scale) is obtained through the C3*3 module. The seventh fusion feature map P7 is used as the input of the RCAM1 module of the first scale. Finally, the RCAM1 module outputs the result feature map of the first scale for detection by Detect1. Since small targets are easier to detect in high resolution, the traditional YOLOv5 detection of 20×20, 40×40, and 80×80 resolution feature maps is improved to detection of 40×40, 80×80, and 160×160 resolution feature maps. Under the premise of ensuring real-time detection of FOD, the detection accuracy of small FOD targets is improved.
[0121] The YOLOv5 network uses CIoU Loss as the border loss function, which comprehensively considers the coverage area, center point distance and aspect ratio, can measure their relative positions well, and can solve the problem of horizontal and vertical optimization of the prediction box, but does not consider the matching of the direction between the prediction box and the real box. This deficiency leads to a slow convergence speed. To solve this problem, the present invention redefines the distance loss from the vector perspective based on CIoU Loss and designs the following border regression loss function:
[0122] (7)
[0123] (8)
[0124] (9)
[0125] (10)
[0126] (11)
[0127] (12)
[0128] in, L is the loss value, t is the coordinate axis, All are intermediate quantities. is the center point coordinate of the real frame, is the center point coordinate of the predicted box, IoU is the intersection over union ratio of the real box and the predicted box, a is a positive balance parameter, s is the consistency of the aspect ratio between the predicted box and the real box, are the width and height of the minimum bounding rectangle of the real box and the predicted box, respectively. w g and h g are the width and height of the real frame respectively, w and h are the width and height of the prediction box respectively, An intermediate amount.
[0129] The border loss function designed by the present invention defines the distance loss from a vector perspective and adds consideration of the consistency of the aspect ratio, which can effectively reduce the degrees of freedom of regression, accelerate network convergence, and further improve the regression accuracy.
[0130] The present invention adopts the fusion detection technology of millimeter-wave radar and visual sensor, and comprehensively applies the millimeter-wave radar detection technology and the visual detection technology based on deep learning. It not only overcomes the shortcomings of single sensor detection technology, but also combines the advantages of millimeter-wave radar with high-precision and accurate target position detection capability and strong anti-interference ability as well as the visualization characteristics of visual detection technology, so that the target detection model has better detection accuracy and robustness.
[0131] The present invention extracts radar and visual features respectively, and fuses radar point cloud data and visual images at the feature level. Compared with decision-level and data-level fusion strategies, the present invention can fuse feature information from different sensors, fully utilize the advantages of each sensor, and provide more comprehensive and accurate target perception and recognition capabilities; it can fuse context information from different sensors, enhance the ability to understand the target or environment, and improve the accuracy and robustness of decision-making; it can reduce problems caused by noise and false alarms of a single sensor, and improve the reliability of target detection and recognition.
[0132] The present invention improves YOLOv5 on small target detection and adopts the radar-vision fusion technology to achieve accurate detection of small and medium-sized targets. The present invention is applicable to the field of small and medium-sized target detection such as foreign objects on airport runways, has great application prospects, and can effectively solve the problems of missed detection and false detection of foreign objects on airport runways while ensuring real-time performance.
[0133] An embodiment of the present invention also provides an electronic device, which includes: a processor and a memory storing a computer program, wherein the processor is configured to implement the steps of the FOD detection method based on a convolutional neural network as described above when executing the computer program.
[0134] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) or the program and / or data loaded from the storage part to the random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. Various programs and data required for the operation of the electronic device are also stored in the RAM. The processor, ROM and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.
[0135] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.
[0136] Although not shown, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the FOD detection method based on a convolutional neural network as described above are implemented.
[0137] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0138] This invention is funded by the Hunan Science and Technology Innovation Program, Project No. 2023RC3230.
[0139] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.
Claims
1. A foreign body intrusion detection method based on convolutional neural network, characterized in that: The detection method comprises the following steps: Acquire radar point cloud data and visual images of intrusion targets; Performing time alignment on the radar point cloud data and the visual image to obtain multiple groups of radar point cloud data and visual images; The radar point cloud data in each group are processed to obtain a radar projection image, where the radar projection image in each group is aligned with the visual image in time and space and conforms to the input mode of the target detection model; Constructing a target detection model, wherein the target detection model adopts an improved YOLOv5 network, wherein the improved YOLOv5 network adds a feature extraction branch in the feature extraction part and adds a splicing module in the original feature extraction branch, wherein the newly added splicing module includes a first splicing module, a second splicing module, and a third splicing module; and adds an attention module and a feature fusion module in the feature fusion part, wherein the newly added attention module includes a first attention module, a second attention module, and a third attention module; The newly added feature extraction branch is used to extract the features of the radar projection image to obtain radar feature maps of different scales; the original feature extraction branch is used to extract the features of the visual image to obtain visual feature maps of different scales; the newly added splicing module is used to splice the radar feature map with the visual feature map corresponding to the scale to obtain a fused feature map; the newly added feature fusion module and the original feature fusion module are both used to extract features, sample, and splice the fused feature map output by the newly added splicing module; the newly added attention module is used to extract features and fuse the fused feature maps output by the newly added feature fusion module and the original feature fusion module, and radar feature maps of different scales to obtain a result feature map; Inputting the radar projection image and the visual image of each group into the target detection model at the same time, and using the target detection model to detect the radar projection image and the visual image to obtain the intrusion target image; Wherein, the improved YOLOv5 network includes a feature extraction part, a feature fusion part and a feature detection part, and the feature detection part includes a first feature detection module, a second feature detection module and a third feature detection module; The newly added feature extraction branch and the original feature extraction branch both include a first scale feature extraction module, a second scale feature extraction module and a third scale feature extraction module, the first scale feature extraction module, the second scale feature extraction module and the third scale feature extraction module of the newly added feature extraction branch are connected in sequence, a first splicing module is provided between the first scale feature extraction module and the second scale feature extraction module of the original feature extraction branch, a second splicing module is provided between the second scale feature extraction module and the third scale feature extraction module of the original feature extraction branch, a third splicing module is provided between the third scale feature extraction module of the original feature extraction branch and the spatial pyramid pooling layer, and the first scale feature extraction module of the newly added feature extraction branch is also connected to the first splicing module and the first attention module, the second scale feature extraction module of the newly added feature extraction branch is also connected to the second splicing module and the second attention module, and the third scale feature extraction module of the newly added feature extraction branch is also connected to the third splicing module and the third attention module; The first splicing module is connected to the newly added feature fusion module, the second splicing module and the spatial pyramid pooling layer are respectively connected to the original feature fusion module, the newly added feature fusion module is connected to the original feature fusion module, the newly added feature fusion module is also connected to the first attention module, and the original feature fusion module is connected to the second attention module and the third attention module; the first attention module is connected to the first feature detection module, the second attention module is connected to the second feature detection module, and the third attention module is connected to the third feature detection module.
2. The foreign body intrusion detection method based on convolutional neural network according to claim 1 is characterized in that: The radar point cloud data and the visual image are time-aligned, specifically including: Adding a timestamp to the radar point cloud data and the visual image to record the sampling time point of the radar point cloud data and the visual image; The radar point cloud data and the visual image are aligned according to the timestamp to obtain multiple sets of time-aligned radar point cloud data and visual images.
3. The foreign body intrusion detection method based on convolutional neural network according to claim 1 is characterized in that: The radar point cloud data includes the location information and status information of the intrusion target, and the radar point cloud data in each group is processed, specifically including: Convert the position of each point cloud in the radar point cloud data from the radar coordinate system to the camera coordinate system; Project the position of each point cloud in the camera coordinate system to the pixel coordinate system to obtain the position of each point cloud in the pixel coordinate system to achieve spatial alignment; Map the point cloud data in the pixel coordinate system to the image plane to generate an RGB mask image; The pixel values of the RGB three channels are calculated according to the state information of the intrusion target. The specific calculation formula is: Among them, r is the pixel value of the R channel, g is the pixel value of the G channel, b is the pixel value of the B channel, d i is the spatial distance from the point cloud to the radar, v i is the radial velocity of the point cloud relative to the radar, p i is the signal-to-noise ratio of the point cloud; The pixel values of the RGB three channels are used to assign values to the pixel points in the RGB mask image to obtain the radar projection image.
4. The foreign body intrusion detection method based on convolutional neural network according to claim 3 is characterized in that: The point cloud data in the pixel coordinate system is mapped to the image plane in the form of a circle, rectangle or triangle. When mapped to the image plane in the form of a circle, the radius of the circle is: Among them, radius is the radius of the circle, Z c is the spatial distance from the intrusion target to the visual sensor.
5. The foreign body intrusion detection method based on convolutional neural network according to any one of claims 1 to 4, characterized in that: The target detection model is used to detect the radar projection image and the visual image, specifically including: Using the newly added feature extraction branch to perform feature extraction on the radar projection image, and sequentially obtain a first-scale radar feature map, a second-scale radar feature map, and a third-scale radar feature map; Using the original feature extraction branch to extract features from the visual image, and sequentially obtain a first-scale visual feature map, a second-scale visual feature map, and a third-scale visual feature map; The first scale radar feature map and the first scale visual feature map are spliced by the first splicing module to obtain a first fused feature map; the second scale radar feature map and the second scale visual feature map are spliced by the second splicing module to obtain a second fused feature map; the third scale radar feature map and the third scale visual feature map are spliced by the third splicing module to obtain a third fused feature map; The third fused feature map is processed by the spatial pyramid pooling layer and then the original feature fusion module is used to perform feature extraction and upsampling to obtain a fourth fused feature map; the second fused feature map and the fourth fused feature map are spliced and feature extracted using the original feature fusion module to obtain a fifth fused feature map; Using the newly added feature fusion module to perform feature extraction and upsampling on the fifth fused feature map to obtain a sixth fused feature map, and using the newly added feature fusion module to perform splicing and feature extraction on the first fused feature map and the sixth fused feature map to obtain a seventh fused feature map; Using the newly added feature fusion module to extract features from the seventh fused feature map, to obtain an eighth fused feature map; using the newly added feature fusion module to splice the eighth fused feature map and the feature map obtained by the convolution layer processing of the fifth fused feature map, to obtain a ninth fused feature map; Using the original feature fusion module to perform feature extraction on the ninth fusion feature map to obtain a tenth fusion feature map; The feature map obtained by the spatial pyramid pooling layer and the tenth fused feature map are processed by the convolution layer respectively, and then spliced and feature extracted to obtain the eleventh fused feature map; The first attention module is used to extract and fuse the first scale radar feature map and the seventh fusion feature map to obtain a first result feature map; the second attention module is used to extract and fuse the second scale radar feature map and the tenth fusion feature map to obtain a second result feature map; the third attention module is used to extract and fuse the third scale radar feature map and the eleventh fusion feature map to obtain a third result feature map; Feature detection is performed on the first result feature map, the second result feature map, and the third result feature map, respectively, to obtain a first intrusion target image, a second intrusion target image, and a third intrusion target image.
6. The foreign body intrusion detection method based on convolutional neural network according to claim 5 is characterized in that: The first attention module, the second attention module and the third attention module all include a first convolution module, a second convolution module, a first adder, a second adder, a third adder and a multiplier; the first convolution module and the second convolution module all include three parallel convolution layers; the output end of the first convolution module is connected to the first adder, the output end of the second convolution module is connected to the second adder, the first adder and the second adder are respectively connected to the third adder, and the input end of the multiplier is connected to the output end of the third adder, the output end of the newly added feature fusion module or the output end of the original feature fusion module.
7. The foreign body intrusion detection method based on convolutional neural network according to claim 5 is characterized in that: The first attention module, the second attention module and the third attention module adopt a classic spatial attention module or an attention module that integrates the spatial attention mechanism and the channel attention mechanism.
8. The foreign body intrusion detection method based on convolutional neural network according to any one of claims 1 to 4, characterized in that: The loss function of the improved YOLOv5 network adopts the border regression loss function, and the specific expression is: Among them, L is the loss value, t is the coordinate axis, ρx and ρy are both intermediate quantities, is the coordinate of the center point of the real box, b and b are the coordinates of the center point of the predicted box, IoU is the intersection over union ratio of the real box and the predicted box, a is the positive balance parameter, and s is the predicted cx cy The consistency of the aspect ratio between the measured frame and the real frame, cw and ch are the width and height of the minimum bounding rectangle of the real frame and the predicted frame respectively, w g and h g are the width and height of the real box, w and h are the width and height of the predicted box, and γ is the intermediate value.
9. An electronic device, characterized in that: The device comprises: Memory for storing computer programs; A processor is used to implement the steps of the foreign body intrusion detection method based on convolutional neural network as described in any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the foreign body intrusion detection method based on a convolutional neural network as described in any one of claims 1 to 9 are implemented.
