A visual-based vehicle-mounted multi-target intrusion perception system without blind area

Abstract

The present application relates to a kind of visual-based vehicle-mounted multi-target intrusion perception system without blind area, belong to target identification technical field, solve the problem of poor night target identification capability in prior art. Including: image acquisition module, far field environment perception module;Wherein, image acquisition module includes far field image collector, for obtaining the RGB image, infrared image and depth image containing target object in far field;Far field environment perception module is used to respectively pre-process the RGB image, infrared image and depth image obtained by far field image collector, and input trained multi-modal feature extraction fusion network to carry out multi-modal feature extraction and fusion, and multi-modal feature extraction fusion network outputs the spatial positioning parameter of each target object;The size and position of each target object are obtained using the each spatial positioning parameter. Realize accurate multi-target intrusion perception.

Description

Technical Field

[0001] This invention relates to the field of target recognition technology, and in particular to a vision-based vehicle-mounted multi-target intrusion perception system with no blind spots. Background Technology

[0002] In key locations in the field that require long-term guard duty, such as mining areas or protected areas, unmanned substations, and military sites, if intrusion is discovered, the intrusion information must be recorded, reported, and the guards notified to conduct a patrol.

[0003] Traditional security systems mostly use electronic fences and pan-tilt cameras. Electronic fences can detect intrusions, but they cannot identify the intruder's location or direction of movement. Furthermore, when there are many intruders, electronic fences cannot determine their exact number. Pan-tilt cameras can identify the intruder's location and direction of movement, and can roughly estimate the number of intruders. However, due to the limited field of view, they cannot cover all areas, resulting in blind spots. They also struggle to determine the geometric dimensions of intruders. Additionally, pan-tilt cameras are significantly affected by lighting conditions, especially in low-light environments, making it difficult to effectively detect obstacles around the vehicle. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a vision-based vehicle-mounted multi-target intrusion perception system with no blind spots, in order to solve the problem of poor target recognition capability at night in the prior art.

[0005] This invention provides a vision-based vehicle-mounted multi-target intrusion detection system with no blind spots, comprising: an image acquisition module and a far-field environment perception module; wherein,

[0006] The image acquisition module includes a far-field image acquisition unit, used to obtain RGB images, infrared images, and depth images containing the target object in the far field;

[0007] The far-field environment perception module is used to preprocess the RGB images, infrared images and depth images acquired by the far-field image acquisition device, and input them into the trained multimodal 3D target detection network for multimodal feature extraction and fusion. The multimodal feature extraction and fusion network outputs the spatial positioning parameters of each target object; the size and position of each target object are obtained using the spatial positioning parameters.

[0008] Based on further improvements to the above method, the multimodal feature extraction and fusion network includes a first, second, and third backbone network module and a feature fusion module; wherein,

[0009] The first backbone network module is used to extract multi-scale features from the preprocessed RGB image based on the interaction information between the RGB image and the infrared image;

[0010] The second backbone network module is used to extract multi-scale features from the preprocessed infrared image based on the interaction information between the RGB image and the infrared image;

[0011] The third backbone network module is used to extract multi-scale features from the preprocessed depth image.

[0012] The feature fusion module is used to add RGB feature maps and infrared feature maps of the same scale and then stitch them together with depth feature maps of the corresponding scale to obtain a stitched feature map; and to further extract fused features from the stitched feature map to obtain the spatial positioning parameters of each target object.

[0013] Based on further improvements to the above method, the first, second, and third backbone networks each include five convolutional fusion modules, used to obtain the first to fifth RGB feature maps, infrared feature maps, and depth feature maps, respectively; wherein,

[0014] The first convolutional fusion module includes a ConvBNSiLU layer; the second to fourth convolutional fusion modules include a ConvBNSiLU layer and a Bottleneck CSP layer connected in sequence; the fifth convolutional fusion module of the first and second backbone network modules includes a ConvBNSiLU layer and a Bottleneck CSP layer; the fifth convolutional fusion module of the third backbone network module includes a ConvBNSiLU layer; the outputs of the fifth convolutional fusion modules of the first and second backbone network modules are added together and then input into the SPPF layer.

[0015] The ConvBNSiLU layer is used to extract features from the input image, the BottleneckCSP layer is used to prevent gradient vanishing during backpropagation, and the SPPF layer is used to extract features from the summed RGB and infrared feature maps of the input.

[0016] Based on a further improvement of the above method, the feature fusion module includes first to fourth feature fusion modules; wherein,

[0017] The first feature fusion module includes a first Concat layer; the first Concat layer is used to stitch together the summed third RGB feature map, infrared feature map and depth feature map of the corresponding scale to obtain the stitched first stitched feature map;

[0018] The second feature fusion module includes a second Concat layer; the second Concat layer is used to stitch the summed fourth RGB feature map, infrared feature map and corresponding scale depth feature map together to obtain the stitched second stitched feature map;

[0019] The third feature fusion module includes a third Concat layer; the third Concat layer is used to concatenate the feature map output by SPPF with the depth feature map of the corresponding scale to obtain the concatenated third concatenated feature map.

[0020] The fourth feature fusion module is used to further extract fusion features from the stitched first to third stitched feature maps to obtain the spatial positioning parameters of each target object.

[0021] Based on further improvements to the above method, multi-scale feature extraction is performed on the preprocessed RGB and infrared images based on interactive information, including:

[0022] Transformer modules are set at the outputs of the second, third, and fourth convolutional fusion modules in the first and second backbone networks, respectively.

[0023] The Transformer module is used to exchange information between RGB feature maps and infrared feature maps of the same scale, resulting in fused RGB feature maps and fused infrared feature maps after the exchange information is obtained.

[0024] The fused RGB feature map is added to the RGB feature map of the input Transformer module and used as the input to the next level convolutional fusion module in the first backbone network; the fused infrared feature map is added to the infrared feature map of the input Transformer module and used as the input to the next level convolutional fusion module in the second backbone network.

[0025] Based on further improvements to the above method, the fourth feature fusion module includes an FPN module, a PAN module, and a Detect module; wherein,

[0026] The FPN module includes a first ConvBNSiLU layer, a first upsampling layer, a fourth Concat layer, a first BottleneckCSP layer, a second ConvBNSiLU layer, a second upsampling layer, a fifth Concat layer, and a second BottleneckCSP layer connected in sequence; wherein, the first ConvBNSiLU layer is used to extract features from the input third concatenated feature map, the other input of the fourth Concat layer is connected to the second concatenated feature map, and the other input of the fifth Concat layer is connected to the first concatenated feature map;

[0027] The PAN module includes a third ConvBNSiLU layer, a sixth Concat layer, a third BottleneckCSP layer, a fourth ConvBNSiLU layer, a seventh Concat layer, and a fourth BottleneckCSP layer connected in sequence; wherein, another input of the third ConvBNSiLU layer is the output of the second ConvBNSiLU layer in the FPN module; and another input of the fourth ConvBNSiLU layer is the output of the first ConvBNSiLU layer in the FPN module.

[0028] The Detect module includes a first Detect module, a second Detect module, and a third Detect module, used to regress the spatial positioning parameters of each target object; wherein,

[0029] The input to the first Detect module is the output of the second Bottleneck CSP layer in the FPN module;

[0030] The input to the second Detect module is the output of the third Bottleneck CSP layer in the PAN module;

[0031] The input to the third Detect module is the output of the fourth Bottleneck CSP layer in the PAN module;

[0032] The outputs of the first, second, and third Detect modules are the spatial positioning parameters of each target object.

[0033] Based on a further improvement of the above method, the spatial positioning parameters include:

[0034] The target category, center point coordinates, center point depth, center point offset, outer border, and yaw angle of each target object.

[0035] Based on a further improvement of the above method, the image acquisition module also includes a near-field image acquisition unit, used to obtain a fisheye image containing the target object in the near field;

[0036] The near-field image acquisition device is a fisheye camera vertically mounted on each side of the vehicle body, used to obtain fisheye images of the target object in the near field; wherein, one fisheye camera is installed at the center of the front and rear sides of the vehicle body, and two fisheye cameras are evenly installed on the left and right sides of the vehicle body; the height of each fisheye camera is the same.

[0037] Further improvements to the above method include a near-field environment perception module, which is used to obtain 3D information of the target object from each fisheye image based on the BEVformer algorithm.

[0038] Based on a further improvement of the above method, the far-field image acquisition device is a centralized panoramic camera;

[0039] The base of the centralized panoramic camera is a regular hexagonal prism, which is vertically installed at the center of the top of the vehicle at a fixed height. One side of the base faces the front of the vehicle. Each side of the base is equipped with an RGB low-light camera, an infrared camera, and a TOF camera, which are used to acquire RGB images, infrared images, and depth images containing the target object in the far field, respectively. Each camera is perpendicular to the corresponding side of the base.

[0040] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0041] 1. The far-field image acquisition unit acquires RGB, infrared, and depth images of the target object in the far field, which improves the system's nighttime perception capability and enables the system to maintain high detection accuracy in low light conditions, thus expanding the system's application scenarios.

[0042] 2. To address the issue of blind spots in the near-field region caused by the far-field image acquisition device, a near-field image acquisition device was added. This device utilizes multiple fisheye cameras to acquire near-field images from all directions and angles, ensuring the comprehensiveness of the system.

[0043] 3. The system utilizes a self-developed multimodal feature extraction and fusion network to process the acquired far-field images, extracting features from the images through multi-scale convolutional layers; it uses a feature fusion module to fuse RGB features, infrared features, and depth features; it further achieves multi-scale feature fusion through FPN and PAN modules; and it uses a Detect module to accurately extract the spatial positioning parameters of the target object. The system's data processing is stable and highly adaptable to various environments.

[0044] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0045] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0046] Figure 1 This is a schematic diagram of a vision-based vehicle-mounted multi-target intrusion perception system with no blind spots, as shown in this embodiment.

[0047] Figure 2This is a schematic diagram of the camera installation for the vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system shown in this embodiment.

[0048] Figure 3 This is a top view of the vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system shown in this embodiment. Detailed Implementation

[0049] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0050] One specific embodiment of the present invention discloses a vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system, such as... Figure 1 As shown. It includes: an image acquisition module and a far-field environment perception module; wherein,

[0051] The image acquisition module includes a far-field image acquisition unit, used to obtain RGB images, infrared images, and depth images containing the target object in the far field;

[0052] The far-field environment perception module is used to preprocess the RGB images, infrared images and depth images acquired by the far-field image acquisition device, and input them into the trained multimodal feature extraction and fusion network for multimodal feature extraction and fusion. The multimodal feature extraction and fusion network outputs the spatial positioning parameters of each target object; the size and position of each target object are obtained using the spatial positioning parameters.

[0053] When implementing, such as Figure 2 As shown, the image acquisition module includes a far-field image acquisition unit and a near-field image acquisition unit.

[0054] Specifically, the far-field image acquisition device is a centralized panoramic camera. The base of the centralized panoramic camera is a regular hexagonal prism, vertically mounted at the center of the vehicle's roof at a fixed height, with one side of the base facing the front of the vehicle. Each side of the base is equipped with an RGB low-light camera, an infrared camera, and a TOF camera, used to acquire RGB images, infrared images, and depth images of the target object in the far field, respectively. Each camera is perpendicular to its corresponding side of the base. For example, the base is 3 meters above the vehicle roof, and the far field refers to the area from 4 meters to 100 meters away from the vehicle body. Infrared images can avoid the impact of insufficient lighting on the images.

[0055] A near-field image acquisition unit is used to obtain fisheye images containing the target object in the near field. The near-field image acquisition unit consists of fisheye cameras vertically mounted on each side of the vehicle body, used to obtain individual fisheye images of the target object in the near field. Specifically, one fisheye camera is installed at the center of the front and rear sides of the vehicle body, and two fisheye cameras are evenly installed on the left and right sides of the vehicle body. All fisheye cameras are at the same height, with a single fisheye camera having a field of view of 180° x 110° and a resolution of 1080P. Searchlights are arranged around each fisheye camera. Since the fisheye camera's field of view can reach 180°, this layout ensures that the system can perceive the vehicle body at close range with a 360° view without blind spots. Furthermore, the light source around the fisheye camera helps to improve the illumination around the vehicle, thereby improving the imaging effect of the fisheye camera. The near field refers to the area within 4 meters of the vehicle body.

[0056] like Figure 3 As shown, the far-field image acquisition unit and the near-field image acquisition unit together divide the spatial scene into six parts, and each part has a common viewing area of ​​the far-field image acquisition unit and the near-field image acquisition unit.

[0057] Furthermore, the system shown in this embodiment also includes a far-field environment perception module and a near-field environment perception module.

[0058] It is worth noting that each fisheye camera needs to be calibrated before use; among them

[0059] The calibration of each fisheye camera includes calibrating the external parameters of each fisheye camera with reference to the vehicle body coordinate system.

[0060] Furthermore, the cameras on each side of the centralized panoramic camera are calibrated before use; wherein, the calibration of the camera on one side of the centralized panoramic camera includes:

[0061] The intrinsic parameters of RGB low-light cameras, infrared cameras, and TOF depth cameras were calibrated using Zhang Zhengyou's calibration method.

[0062] Zhang Zhengyou's calibration algorithm was used to calibrate the extrinsic parameters of an infrared camera and a TOF depth camera relative to an RGB camera.

[0063] Using the extrinsic parameters between the fisheye camera and the vehicle body coordinate system within the common viewing area, and based on 2D-2D feature point matching in SFM 3D reconstruction technology, the extrinsic parameters of the RGB low-light camera, infrared camera, and TOF camera with the vehicle body as the reference are obtained based on the extrinsic parameters of the infrared camera and TOF depth camera relative to the RGB camera.

[0064] The extrinsic parameter calibration between the camera coordinate system and the vehicle coordinate system on each side is relatively complex. Using calibration boards would require arranging too many boards, and the positional transformation relationship between the calibration boards and the vehicle coordinate system is difficult to measure. Therefore, this embodiment utilizes the extrinsic parameters between the fisheye camera and the vehicle coordinate system within the common viewing area. Based on 2D-2D feature point matching in SFM 3D reconstruction technology, the extrinsic parameters of the RGB low-light camera, infrared camera, and TOF camera on the corresponding side, with the vehicle body as the reference, are obtained. For example, SIFT, ORB, or KAZA algorithms are used for feature point matching to obtain the camera coordinate system transformation matrix. Then, the feature point matching relationship between the RGB image and the corresponding area fisheye-corrected image is obtained, leading to the extrinsic parameter transformation relationship between the fisheye camera and the RGB camera. Since the extrinsic parameters between the fisheye camera and the vehicle coordinate system have been calibrated, the extrinsic parameters between the coordinate systems of each camera in the centralized panoramic camera and the vehicle coordinate system can also be obtained.

[0065] Specifically, the near-field environment perception module is used to obtain 3D information of the target object from various fisheye images based on the BEVformer algorithm. The specific steps include:

[0066] Fisheye distortion image correction can restore the distorted parts of an image to a state close to the real viewpoint, thereby improving image quality and measurement accuracy, and enhancing the performance of subsequent algorithms.

[0067] Based on the BEVformer algorithm, the 3D information of the target object is obtained from each fisheye image after distortion correction. The BEVformer algorithm has good robustness and adaptability, and can perceive the target object well and obtain relevant 3D information.

[0068] Furthermore, the far-field environment perception module is used to preprocess the RGB images, infrared images and depth images acquired by the far-field image acquisition device, and input them into the trained multimodal feature extraction and fusion network to obtain the spatial positioning parameters of each target object.

[0069] Specifically, the preprocessing of the RGB image, infrared image, and depth image acquired by the far-field image acquisition device includes:

[0070] Based on the intrinsic parameters of the RGB low-light camera, infrared camera, and TOF depth camera, and the extrinsic parameters of the infrared camera and TOF depth camera relative to the RGB camera, the RGB images, infrared images, and depth feature images acquired by each camera in the centralized panoramic camera are registered to obtain the registered RGB images, infrared images, and depth feature images.

[0071] The registered RGB image, infrared image, and depth feature image are cropped separately to remove non-overlapping regions, resulting in preprocessed RGB, infrared, and depth images. These preprocessed images are then input into a trained multimodal feature extraction and fusion network.

[0072] It is worth noting that the multimodal feature extraction and fusion network includes a first, second, and third backbone network module and a feature fusion module; wherein,

[0073] The first backbone network module is used to extract multi-scale features from the preprocessed RGB image based on the interaction information between the RGB image and the infrared image;

[0074] The second backbone network module is used to extract multi-scale features from the preprocessed infrared image based on the interaction information between the RGB image and the infrared image;

[0075] The third backbone network module is used to extract multi-scale features from the preprocessed depth image.

[0076] The feature fusion module is used to add RGB feature maps and infrared feature maps of the same scale and then stitch them together with depth feature maps of the corresponding scale to obtain a stitched feature map; and to further extract fused features from the stitched feature map to obtain the spatial positioning parameters of each target object.

[0077] Specifically, the first, second, and third backbone networks each include five convolutional fusion modules, used to obtain the first to fifth RGB feature maps, infrared feature maps, and depth feature maps, respectively; among them...

[0078] The first convolutional fusion module includes a ConvBNSiLU layer; the second to fourth convolutional fusion modules include a ConvBNSiLU layer and a Bottleneck CSP layer connected in sequence; the fifth convolutional fusion module of the first and second backbone network modules includes a ConvBNSiLU layer and a Bottleneck CSP layer; the fifth convolutional fusion module of the third backbone network module includes a ConvBNSiLU layer; the outputs of the fifth convolutional fusion modules of the first and second backbone network modules are added together and then input into the SPPF layer.

[0079] The ConvBNSiLU layer is used to extract features from the input image, the BottleneckCSP layer is used to prevent gradient vanishing during backpropagation, and the SPPF layer is used to extract features from the summed RGB and infrared feature maps of the input.

[0080] Specifically, the feature fusion module includes first to fourth feature fusion modules; wherein,

[0081] The first feature fusion module includes a first Concat layer; the first Concat layer is used to stitch together the summed third RGB feature map, infrared feature map and depth feature map of the corresponding scale to obtain the stitched first stitched feature map;

[0082] The second feature fusion module includes a second Concat layer; the second Concat layer is used to stitch together the summed fourth RGB feature map, infrared feature map and depth feature map of the corresponding scale to obtain the stitched second stitched feature map;

[0083] The third feature fusion module includes a third Concat layer; the third Concat layer is used to concatenate the feature map output by SPPF with the depth feature map of the corresponding scale to obtain the concatenated third concatenated feature map.

[0084] The fourth feature fusion module is used to further extract fusion features from the stitched first to third stitched feature maps to obtain the spatial positioning parameters of each target object.

[0085] It is worth noting that, based on the interactive information, multi-scale feature extraction is performed on the preprocessed RGB and infrared images, including:

[0086] Transformer modules are set at the outputs of the second, third, and fourth convolutional fusion modules in the first and second backbone networks, respectively.

[0087] The Transformer module is used to exchange information between RGB feature maps and infrared feature maps of the same scale, resulting in fused RGB feature maps and fused infrared feature maps after the exchange information is obtained.

[0088] The fused RGB feature map is added to the RGB feature map of the input Transformer module and used as the input to the next level convolutional fusion module in the first backbone network; the fused infrared feature map is added to the infrared feature map of the input Transformer module and used as the input to the next level convolutional fusion module in the second backbone network.

[0089] Specifically, the fourth feature fusion module includes an FPN module, a PAN module, and a Detect module; wherein,

[0090] The FPN module includes a first ConvBNSiLU layer, a first upsampling layer, a fourth Concat layer, a first BottleneckCSP layer, a second ConvBNSiLU layer, a second upsampling layer, a fifth Concat layer, and a second BottleneckCSP layer connected in sequence; wherein, the first ConvBNSiLU layer is used to extract features from the input third concatenated feature map, the other input of the fourth Concat layer is connected to the second concatenated feature map, and the other input of the fifth Concat layer is connected to the first concatenated feature map;

[0091] The PAN module includes a third ConvBNSiLU layer, a sixth Concat layer, a third BottleneckCSP layer, a fourth ConvBNSiLU layer, a seventh Concat layer, and a fourth BottleneckCSP layer connected in sequence; wherein, another input of the third ConvBNSiLU layer is the output of the second ConvBNSiLU layer in the FPN module; and another input of the fourth ConvBNSiLU layer is the output of the first ConvBNSiLU layer in the FPN module.

[0092] The Detect module includes a first Detect module, a second Detect module, and a third Detect module, used to regress the spatial positioning parameters of each target object; wherein,

[0093] The input to the first Detect module is the output of the second Bottleneck CSP layer in the FPN module;

[0094] The input to the second Detect module is the output of the third Bottleneck CSP layer in the PAN module;

[0095] The input to the third Detect module is the output of the fourth Bottleneck CSP layer in the PAN module;

[0096] The outputs of the first, second, and third Detect modules are the spatial positioning parameters of each target object.

[0097] For example, the first Detect module corresponds to a large-scale feature map, the second Detect module corresponds to a medium-scale feature map, and the third Detect module corresponds to a small-scale feature map, so that the spatial positioning parameters of each target object include features of each scale.

[0098] Specifically, each Detect module includes a Detect head, which comprises a class cls prediction branch, a target keypoint center coordinate (u,v) prediction branch, and an offset (δ) based on the keypoint coordinates. x ,δ y Predictive branch, a depth offset δ d The system employs three prediction branches: a prediction branch for bounding box dimensions (w, h, l), and a prediction branch for yaw angle α. The resulting spatial positioning parameters include: target category, center point coordinates, center point depth, center point offset, bounding box, and yaw angle. The bounding box size represents the size of the target object.

[0099] Furthermore, the position of the target object is obtained using the following formula based on the aforementioned spatial positioning parameters:

[0100]

[0101]

[0102] Where K represents the camera's intrinsic parameters, T represents the camera's extrinsic parameters relative to the vehicle's coordinate system, σ represents the Sigmoid function; (x,y,z) represents the target's position, (u,v) represents the center point coordinates, and δ... d Indicates the depth at the center point, (δ) x ,δ y ) represents the center point offset, d c This represents the depth value of a 3D object in the camera coordinate system, (u c v c () represents the pixel coordinates of the 3D target projected onto the camera plane. From this, the size and position of the target can be obtained.

[0103] Compared with existing technologies, the far-field image acquisition device provided in this embodiment acquires RGB, infrared, and depth images containing the target object in the far field, improving the system's nighttime perception capability and maintaining high detection accuracy even in low-light conditions, thus expanding the system's application scenarios. To address the blind spot issue in the near-field region of the far-field image acquisition device, a near-field image acquisition device is added, utilizing multiple fisheye cameras to acquire near-field images from all directions and angles, ensuring the system's comprehensiveness. A self-developed multimodal feature extraction and fusion network is used to process the acquired far-field images, extracting features through multi-scale convolutional layers. A feature fusion module is used to fuse RGB, infrared, and depth features. Multi-scale feature fusion is further achieved through FPN and PAN modules. The Detect module accurately extracts the spatial positioning parameters of the target object. This system exhibits stable data processing and strong environmental adaptability.

[0104] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0105] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system, characterized in that, include: Image acquisition module, far-field environment perception module; among which, The image acquisition module includes a far-field image acquisition unit, used to obtain RGB images, infrared images, and depth images containing the target object in the far field; The far-field environment perception module is used to preprocess the RGB images, infrared images and depth images acquired by the far-field image acquisition device, and input them into the trained multimodal 3D target detection network for multimodal feature extraction and fusion. The multimodal feature extraction and fusion network outputs the spatial positioning parameters of each target object; the size and position of each target object are obtained using the spatial positioning parameters. The multimodal feature extraction and fusion network includes a first, second, and third backbone network module and a feature fusion module. The first backbone network module performs multi-scale feature extraction on the preprocessed RGB image based on the interaction information between the RGB and infrared images. The second backbone network module performs multi-scale feature extraction on the preprocessed infrared image based on the interaction information between the RGB and infrared images. The third backbone network module performs multi-scale feature extraction on the preprocessed depth image. The feature fusion module adds RGB and infrared feature maps of the same scale and then stitches them together with the corresponding depth feature map to obtain a stitched feature map. Furthermore, it extracts fusion features from the stitched feature map to obtain the spatial positioning parameters of each target object. The first, second, and third backbone networks each include five convolutional fusion modules, used to obtain first to fifth RGB feature maps, infrared feature maps, and depth feature maps, respectively. The first convolutional fusion module includes a ConvBNSiLU layer; the second to fourth convolutional fusion modules include sequentially connected ConvBNSiLU layers and BottleneckCSP layers; the fifth convolutional fusion module of the first and second backbone network modules includes a ConvBNSiLU layer and a BottleneckCSP layer; the fifth convolutional fusion module of the third backbone network module includes a ConvBNSiLU layer; the outputs of the fifth convolutional fusion modules of the first and second backbone network modules are summed and then input into an SPPF layer. The ConvBNSiLU layer is used for feature extraction from the input image; the BottleneckCSP layer is used to prevent gradient vanishing during backpropagation; and the SPPF layer is used for feature extraction from the summed RGB and infrared feature maps. The feature fusion module includes a first to a fourth feature fusion module; wherein, the first feature fusion module includes a first Concat layer; the first Concat layer is used to stitch together the summed third RGB feature map, infrared feature map and corresponding scale depth feature map to obtain a stitched first stitched feature map; the second feature fusion module includes a second Concat layer; the second Concat layer is used to stitch together the summed fourth RGB feature map, infrared feature map and corresponding scale depth feature map to obtain a stitched second stitched feature map; the third feature fusion module includes a third Concat layer; the third Concat layer is used to stitch together the feature map output by SPPF and corresponding scale depth feature map to obtain a stitched third stitched feature map; the fourth feature fusion module is used to further extract fusion features from the stitched first to third stitched feature maps to obtain the spatial positioning parameters of each target object; Based on interactive information, multi-scale feature extraction is performed on the preprocessed RGB and infrared images, including: setting Transformer modules at the outputs of the second, third, and fourth convolutional fusion modules in the first and second backbone networks respectively; using the Transformer modules to perform information interaction between RGB feature maps and infrared feature maps of the same scale to obtain fused RGB feature maps and fused infrared feature maps after interaction; adding the fused RGB feature map to the RGB feature map input to the Transformer module as the input to the next-level convolutional fusion module in the first backbone network; adding the fused infrared feature map to the infrared feature map input to the Transformer module as the input to the next-level convolutional fusion module in the second backbone network; The fourth feature fusion module includes an FPN module, a PAN module, and a Detect module; wherein, the FPN module includes a first ConvBNSiLU layer, a first upsampling layer, a fourth Concat layer, a first BottleneckCSP layer, a second ConvBNSiLU layer, a second upsampling layer, a fifth Concat layer, and a second BottleneckCSP layer connected in sequence; wherein, the first ConvBNSiLU layer is used to extract features from the input third concatenated feature map, the other input of the fourth Concat layer is connected to the second concatenated feature map, and the other input of the fifth Concat layer is connected to the first concatenated feature map; The PAN module includes a third ConvBNSiLU layer, a sixth Concat layer, a third BottleneckCSP layer, a fourth ConvBNSiLU layer, a seventh Concat layer, and a fourth BottleneckCSP layer connected in sequence. Another input to the third ConvBNSiLU layer is the output of the second ConvBNSiLU layer in the FPN module; another input to the fourth ConvBNSiLU layer is the output of the first ConvBNSiLU layer in the FPN module. The Detect module includes a first Detect module, a second Detect module, and a third Detect module, used to regress the spatial positioning parameters of each target object. The input to the first Detect module is the output of the second BottleneckCSP layer in the FPN module; the input to the second Detect module is the output of the third BottleneckCSP layer in the PAN module; the input to the third Detect module is the output of the fourth BottleneckCSP layer in the PAN module; and the outputs of the first, second, and third Detect modules are the spatial positioning parameters of each target object. The spatial positioning parameters include: target category, center point coordinates, center point depth, center point offset, outer bounding box, and yaw angle for each target object.

2. The vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system according to claim 1, characterized in that, The image acquisition module also includes a near-field image acquisition unit, used to obtain fisheye images containing the target object in the near field; The near-field image acquisition device is a fisheye camera vertically mounted on each side of the vehicle body, used to obtain fisheye images of the target object in the near field; wherein, one fisheye camera is installed at the center of the front and rear sides of the vehicle body, and two fisheye cameras are evenly installed on the left and right sides of the vehicle body; the height of each fisheye camera is the same.

3. The vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system according to claim 2, characterized in that, It also includes a near-field environment perception module, which is used to obtain the 3D information of the target object using each fisheye image based on the BEVformer algorithm.

4. The vision-based vehicle-mounted blind-spot-free multi-target intrusion perception system according to claim 1, characterized in that, The far-field image acquisition device is a centralized panoramic camera; The base of the centralized panoramic camera is a regular hexagonal prism, which is vertically installed at the center of the top of the vehicle at a fixed height. One side of the base faces the front of the vehicle. Each side of the base is equipped with an RGB low-light camera, an infrared camera, and a TOF camera, which are used to acquire RGB images, infrared images, and depth images containing the target object in the far field, respectively. Each camera is perpendicular to the corresponding side of the base.

Images