Bridge anti-collision detection method and system based on multi-modal fusion of object imaging

Through the multimodal fusion method of object imaging and spatiotemporal clustering algorithm, camera and millimeter wave radar data are processed and bridge anti-collision warning information is generated, which solves the problem of long data processing time in the existing technology, and improves the real-time detection and Accuracy.

CN118604817BActive Publication Date: 2024-11-26SOUTH CHINA NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411081523.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-26
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

In the prior art, the fusion complementary method of camera and millimeter wave radar data increases the complexity of algorithms and the amount of calculations, making it difficult to meet the real-time requirements of bridge collision-proof application scenarios.

Method used

Using a multimodal fusion method based on object imaging, camera detection data is processed through object imaging algorithm to obtain the image of the area of ​​the object of interest, and millimeter wave radar detection data is processed through spatiotemporal clustering algorithm, combining the two to generate multimodal information fusion results , generate early warning information based on preset bridge anti-collision safety rules.

Benefits of technology

It effectively reduces the amount of algorithm calculation, improves the real-time and accuracy of ship detection data in bridge collision prevention application scenarios, and meets the real-time requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118604817B_ABST
    Figure CN118604817B_ABST
Patent Text Reader

Abstract

The present application provides a bridge anti-collision detection method and system based on multimodal fusion of object imaging, which can be applied to the field of transportation technology. The present application processes the camera detection data through an object imaging algorithm to obtain an image of the object area of ​​interest, and processes the millimeter-wave radar detection data through a spatiotemporal clustering algorithm to obtain target detection information, and then fuses the image of the object area of ​​interest and the target detection information to obtain a multimodal information fusion result, and then generates bridge anti-collision warning information based on the multimodal information fusion result and the preset bridge anti-collision safety rules; the present application uses the area of ​​interest for bridge anti-collision detection, so there is no need to process irrelevant data, effectively reducing the amount of algorithm calculation, reducing data processing time, and thus improving the real-time performance of ship detection data in bridge anti-collision application scenarios, and at the same time uses a combination of camera data and millimeter-wave radar data for bridge anti-collision detection, effectively improving the accuracy of detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of transportation technology, and in particular to a bridge anti-collision detection method and system based on multi-modal fusion of object imaging. Background Art

[0002] Bridges play an important role as hubs in transportation networks. However, bridges are often damaged or collapsed due to ship collisions. In order to reduce the number of ship collisions with bridges, camera sensors or lidars can be used for ship detection. However, camera sensors are easily affected by the environment, while lidars are not affected by the environment but are expensive. In related technologies, in order to improve detection accuracy and reduce prices, the information of cameras and millimeter-wave radars is fused and complemented to detect the position of ships in real time. However, the method of fusing and complementing the two types of information greatly increases the complexity and amount of calculation of the algorithm, thereby increasing the data processing time, making it difficult to meet the real-time requirements in bridge collision avoidance application scenarios.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a bridge collision avoidance detection method and system based on multi-modal fusion of object imaging, which can reduce the amount of algorithm calculation and improve the real-time and accuracy of ship detection data in bridge collision avoidance application scenarios.

[0005] To achieve the above object, an embodiment of the present application provides a bridge collision avoidance detection method based on multi-modal fusion of object imaging, the method comprising the following steps:

[0006] Acquire camera detection data and millimeter wave radar detection data, wherein the camera detection data includes ship image information, and the millimeter wave radar detection data includes ship distance information;

[0007] Processing the camera detection data by an object imaging algorithm to obtain an image of the object area of ​​interest;

[0008] Processing the millimeter wave radar detection data by a spatiotemporal clustering algorithm to obtain target detection information;

[0009] Fusing the object of interest region image and the target detection information to obtain a multimodal information fusion result;

[0010] Bridge anti-collision warning information is generated according to the multimodal information fusion result and preset bridge anti-collision safety rules.

[0011] In some embodiments, after the step of acquiring the camera detection data and the millimeter wave radar detection data, the method further includes the following steps:

[0012] Performing time synchronization processing on the camera detection data and the millimeter wave radar detection data;

[0013] The camera detection data and the millimeter wave radar detection data after time synchronization processing are aligned in spatial coordinates.

[0014] In some embodiments, the processing of the camera detection data by an object imaging algorithm to obtain an image of the object region of interest includes:

[0015] Training a preset neural network to obtain convolution kernel parameters, wherein the convolution kernel parameters include features of an object of interest;

[0016] Constructing a feature filter according to the convolution kernel parameters;

[0017] The camera detection data is subjected to background filtering by the feature filter to obtain the object of interest region image, wherein the object of interest region image includes the ship position information.

[0018] In some embodiments, before the step of processing the millimeter wave radar detection data by using a spatiotemporal clustering algorithm, the method further includes the following steps:

[0019] The millimeter-wave radar detection data is preprocessed to obtain first radar detection data to be clustered.

[0020] In some embodiments, the processing of the millimeter wave radar detection data by a spatiotemporal clustering algorithm to obtain target detection information includes:

[0021] Determine a preset clustering threshold according to the spatial feature information;

[0022] Performing a first filtering on all the first radar detection data to be clustered according to the preset clustering threshold to obtain second radar detection data to be clustered;

[0023] A second filtering is performed on all the second radar detection data to be clustered according to the time frame feature to obtain the target detection information.

[0024] In some embodiments, performing a first filtering on all the first radar detection data to be clustered according to the preset clustering threshold to obtain second radar detection data to be clustered includes:

[0025] Calculating the deviation between each of the first radar detection data to be clustered and the cluster center;

[0026] The first radar detection data to be clustered whose deviation is less than or equal to the preset clustering threshold is used as the second radar detection data to be clustered.

[0027] In some embodiments, fusing the object of interest region image and the target detection information to obtain a multimodal information fusion result includes:

[0028] Performing data alignment on the image of the object of interest region and the target detection information;

[0029] Associating the aligned image of the region of object of interest with the target detection information, wherein the associating process is used to match the same obstacle;

[0030] Environmental perception information is generated as the multimodal information fusion result according to the association result, the three-dimensional object position information in the image of the object area of ​​interest, and the obstacle distance, obstacle movement speed and obstacle movement angle of the target detection information.

[0031] In some embodiments, generating bridge collision avoidance warning information according to the multimodal information fusion result and preset bridge collision avoidance safety rules includes:

[0032] Analyze the multimodal information fusion result to obtain key information;

[0033] Determine whether the preset bridge anti-collision safety rules are met based on the key information, and if not, generate the bridge anti-collision warning information.

[0034] To achieve the above object, another aspect of the embodiment of the present application provides a bridge collision avoidance detection system based on multi-modal fusion of object imaging, the system comprising:

[0035] The first module is used to obtain camera detection data and millimeter wave radar detection data, wherein the camera detection data includes ship image information and the millimeter wave radar detection data includes ship distance information;

[0036] The second module is used to process the camera detection data through an object imaging algorithm to obtain an image of the object area of ​​interest;

[0037] The third module is used to process the millimeter wave radar detection data through a spatiotemporal clustering algorithm to obtain target detection information;

[0038] The fourth module is used to fuse the image of the object of interest region and the target detection information to obtain a multimodal information fusion result;

[0039] The fifth module is used to generate bridge collision avoidance warning information according to the multimodal information fusion result and preset bridge collision avoidance safety rules.

[0040] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a bridge collision avoidance detection system based on multi-modal fusion of object imaging, comprising:

[0041] at least one processor;

[0042] at least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0044] The embodiments of the present application include at least the following beneficial effects: The present application provides a bridge collision avoidance detection method and system based on multimodal fusion of object imaging, which processes the camera detection data through an object imaging algorithm to obtain an image of the object area of ​​interest, and processes the millimeter-wave radar detection data through a spatiotemporal clustering algorithm to obtain target detection information, and then fuses the image of the object area of ​​interest and the target detection information to obtain a multimodal information fusion result, and then generates bridge collision avoidance warning information based on the multimodal information fusion result and preset bridge collision avoidance safety rules; the present embodiment uses the area of ​​interest to perform bridge collision avoidance detection, thereby eliminating the need to process irrelevant data, effectively reducing the amount of algorithm calculation, reducing data processing time, and thereby improving the real-time performance of ship detection data in bridge collision avoidance application scenarios, and simultaneously uses a combination of camera data and millimeter-wave radar data to perform bridge collision avoidance detection, effectively improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a bridge anti-collision detection method based on multi-modal fusion of object imaging provided in an embodiment of the present application;

[0046] Figure 2 is a flowchart of processing camera detection data by an object imaging algorithm provided in an embodiment of the present application;

[0047] Figure 3 It is a ship field of view diagram at different distances provided by an embodiment of the present application;

[0048] Figure 4 is a flowchart of processing millimeter wave radar detection data by the spatiotemporal clustering algorithm provided in an embodiment of the present application;

[0049] Figure 5 It is a framework diagram of a bridge anti-collision detection method based on multi-modal fusion of object imaging provided in an embodiment of the present application;

[0050] Figure 6 It is a complete flow chart of the bridge anti-collision detection method based on multi-modal fusion of object imaging provided in an embodiment of the present application;

[0051] Figure 7 It is a structural schematic diagram of a bridge anti-collision detection device based on multi-modal fusion of object imaging provided in an embodiment of the present application;

[0052] Figure 8 It is a schematic diagram of the hardware structure of a bridge anti-collision detection system based on multi-modal fusion of object imaging provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Bridges play an important role as hubs in transportation networks. However, bridge damage and even collapse due to various reasons (such as ship collisions, bad weather, etc.) occur from time to time, causing huge losses to public safety and economic development. Camera sensors have become one of the most commonly used sensors in target detection applications due to their low cost and rich semantic information. However, the performance of cameras may be limited in challenging environments such as rain, snow, fog, dust, and strong / weak lighting conditions. Although lidar can show certain advantages in the above environments, it is expensive. Millimeter-wave radar is not affected by weather and light and can work all day, but its detection accuracy is low. In related technologies, due to the high price of lidar, perception systems usually complement and fuse the information of cameras and millimeter-wave radars to achieve better accuracy and robustness when monitoring objects. However, combining camera data with millimeter-wave radar data increases the complexity and computational complexity of the algorithm, thereby increasing the data processing time, making it difficult to meet the real-time requirements in bridge collision avoidance application scenarios.

[0056] In view of this, an embodiment of the present application provides a bridge collision avoidance detection method and system based on multimodal fusion of object imaging. The present application processes the camera data through an object imaging algorithm to reduce the amount of later data processing, reduce the data processing time, and improve the real-time performance of ship detection data in bridge collision avoidance application scenarios. It also combines millimeter-wave radar data for bridge collision avoidance detection, thereby improving the accuracy of ship detection data in bridge collision avoidance application scenarios.

[0057] The bridge anti-collision detection method based on multimodal fusion of object imaging provided in the embodiment of the present application relates to the field of transportation technology. The bridge anti-collision detection method based on multimodal fusion of object imaging provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application of a bridge anti-collision detection method based on multimodal fusion of object imaging, etc., but is not limited to the above forms.

[0058] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer bridge collision avoidance detection systems based on multimodal fusion of object imaging, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0059] Figure 1 is an optional flow chart of a bridge anti-collision detection method based on multi-modal fusion of object imaging provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S110 to S160:

[0060] Step S110, acquiring camera detection data, wherein the camera detection data includes ship image information;

[0061] Step S120, acquiring millimeter wave radar detection data, wherein the millimeter wave radar detection data includes ship distance information;

[0062] Step S130, processing the camera detection data by an object imaging algorithm to obtain an image of the object area of ​​interest;

[0063] Step S140: Processing the millimeter wave radar detection data by using a spatiotemporal clustering algorithm to obtain target detection information;

[0064] Step S150, fusing the image of the object region of interest and the target detection information to obtain a multimodal information fusion result;

[0065] Step S160: Generate bridge anti-collision warning information according to the multimodal information fusion result and the preset bridge anti-collision safety rules.

[0066] In an embodiment of the present application, after acquiring the camera detection data and the millimeter-wave radar detection data, this embodiment performs time synchronization processing on the camera detection data and the millimeter-wave radar detection data, and performs spatial coordinate alignment on the camera detection data and the millimeter-wave radar detection data after the time synchronization processing, thereby improving the accuracy of subsequent data fusion results.

[0067] In the embodiment of the present application, when the camera detection data (image data to be processed) is processed by the object imaging algorithm, such as Figure 2 As shown, the preset neural network can be trained to obtain the convolution kernel parameters including the features of the object of interest, and after constructing the feature filter according to the convolution kernel parameters, the camera detection data is background filtered by the feature filter to obtain the area image of the object of interest including the ship position information. It can be understood that the preset neural network can be a neural network model capable of target recognition. This embodiment uses the feature filter to complete the screening and extraction of specific images in complex images, that is, the recognition of the object of interest, to avoid the image containing irrelevant information and background information during imaging, and uses the AI ​​object imaging technology algorithm to perform object detection, real-time identification and tracking of ships traveling on the water surface, and filter out complex background information, which not only realizes ship positioning and detection, but also reduces the amount of data for subsequent radar fusion processing. In addition, the center of mass coordinates of the ship can also be calculated in the obtained object imaging results, so as to locate the target ship in the radar data in the subsequent multimodal fusion part.

[0068] It is understandable that the detection range of millimeter-wave radar is usually presented as a fan-shaped area. Within this fan-shaped area, the millimeter-wave radar transmits electromagnetic wave signals and receives signals reflected by the target object. Point cloud data provides information such as the shape, size, and spatial position of the target object, laying the foundation for further processing and analysis. However, in the actual data collection process, millimeter-wave radar may be affected by various interference factors, such as empty targets (i.e., reflections of non-substantial objects) and false targets (non-real targets generated by signal interference or equipment errors). These interference signals may cause multiple non-target points to appear in the data, thereby increasing the complexity of target detection and recognition.

[0069] Therefore, in order to improve the target recognition accuracy of millimeter-wave radar detection data, the embodiment of the present application is processed by a spatiotemporal clustering algorithm. Specifically, the collected point cloud data (the first radar detection data to be clustered) is preprocessed by a filtering algorithm based on prior knowledge to remove empty targets and false target points that do not meet specific conditions, thereby purifying the data set and reducing noise interference in subsequent analysis. Then, the inter-frame clustering algorithm is used to analyze the remaining point cloud data (the second radar detection data to be clustered). The clustering algorithm can classify points that are close in space or similar in features into the same category, thereby achieving the aggregation of multiple reflection points of the same target object. This process not only helps to identify and distinguish different target objects, but also enhances the understanding of the shape and structure of the target object.

[0070] In the embodiment of the present application, before the millimeter wave radar detection data is processed by the spatiotemporal clustering algorithm, the millimeter wave radar detection data is preprocessed to obtain the first radar detection data to be clustered. It can be understood that this embodiment is used to detect whether the ship will hit the bridge pier according to the heading and speed of the ship when it passes under the bridge. The data of the millimeter wave radar detection point includes the longitudinal coordinates, transverse coordinates, longitudinal speed, transverse speed and cross-sectional reflection surface value of the ship. Since the transverse speed of the ship changes little and the characteristics are not obvious in the field of view of the millimeter wave radar during the process of moving from far to near, this embodiment removes the characteristic value of the transverse speed through preprocessing. For the longitudinal coordinate value, that is, the vertical distance of the ship relative to the bridge, since the set warning range is 0~1000 meters, the echo signals outside the range of 1000 meters are filtered out to reduce the amount of data processing. For the transverse coordinate, according to the width of the inland waterway, the echo signals of objects outside the river channel are filtered out to reduce other target interference signals and improve the ship target detection rate. For the longitudinal speed value, since there are some moored ships in the inland waterway, the static ship targets are eliminated according to the speed threshold. After filtering some data through the above preprocessing process, the data processing amount in the subsequent steps can be reduced and the data processing speed can be improved.

[0071] It is understandable that if Figure 3 As shown, the echo information of ships at close range and at long range is inconsistent. If the e and MinPts thresholds are set according to the spatiotemporal clustering algorithm, that is, if the MinPts threshold is set too large, multiple target ships at long range will be clustered into the same ship, and if the MinPts threshold is set too small, ships at close range will be clustered into multiple target ships. Therefore, the embodiment of the present application divides the target clustering algorithm into multiple spaces according to the spatial distance information of the ships. That is, when the distance between the ship and the bridge is close, a larger MinPts threshold is set to cluster the same target ship, and when the distance between the ships is far, a smaller MinPts threshold is set to distinguish the signals of multiple target ships at long range.

[0072] Moreover, for dynamic ship targets, according to the time continuity, the echo signals between multiple frames have the same characteristics, while the noise in the environment has no certain connection with time. Therefore, the embodiment of the present application further performs filtering processing according to the time continuity, that is, the time frame sequence characteristics of the millimeter wave radar, so as to filter out the noise points in the environment and the noise points caused by the radar multipath effect.

[0073] Specifically, the process of processing the millimeter wave radar detection data by the spatiotemporal clustering algorithm in this embodiment is as follows: Figure 4 As shown. Figure 4 It can be seen that after the preset clustering threshold (e, MinPts) is determined according to the spatial feature information, the first filtering of all the first radar detection data to be clustered (millimeter wave radar reflection signal data) is performed according to the preset clustering threshold to obtain the second radar detection data to be clustered, and then the second filtering of all the second radar detection data to be clustered is performed according to the time frame characteristics to obtain the target detection information. It can be understood that in the first filtering process, after calculating the deviation between each first radar detection data to be clustered and the clustering center, the first radar detection data to be clustered whose deviation is less than or equal to the preset clustering threshold is used as the second radar detection data to be clustered. That is, when the deviation is less than or equal to the preset clustering threshold, it is determined that these radar detection data belong to the reflection signal of the same ship; when the deviation is less than or equal to the preset clustering threshold, it is determined that these radar detection data belong to the reflection signal of different ships. Among them, the deviation can be the speed and distance deviation between each first radar detection data to be clustered and the clustering center. After traversing all the first radar detection data to be clustered, the second radar detection data to be clustered is filtered according to the time frame characteristics to obtain the target detection data.

[0074] In an embodiment of the present application, the data collected by the millimeter-wave radar is processed by the radar target detection algorithm to generate a target point containing distance, azimuth and speed, and then the effective target point is mapped to the image pixel coordinate system by means of coordinate system conversion. After the camera collects the image data, the target object is obtained by the object imaging processing algorithm, and then after the information is fused, a more accurate and complete environmental perception information can be generated. That is, after obtaining the image of the object area of ​​interest of the camera detection data and the target detection information of the millimeter-wave radar detection data, the two data are fused to generate a more accurate and complete environmental perception information as a multimodal information fusion result.

[0075] It can be understood that the fusion process of this embodiment includes data alignment, data association and data fusion processes. Specifically, data alignment is to align the image of the object area of ​​interest and the target detection information to ensure that the information obtained by both is in the same coordinate system, complete the synchronization of the millimeter-wave radar and the camera in the time dimension, set a unified sampling period point, and ensure that there is no time deviation in the data collected by different sensors. Data association is to associate the aligned image of the object area of ​​interest with the target detection information to match the objects detected by the monocular camera and the obstacles detected by the millimeter-wave radar. Data fusion is to generate environmental perception information as a multimodal information fusion result based on the association results, the three-dimensional object position information in the image of the object area of ​​interest, and the obstacle distance, obstacle movement speed and obstacle movement angle of the target detection information.

[0076] In an embodiment of the present application, after obtaining the multimodal information fusion result, bridge collision avoidance warning information is generated in combination with the preset bridge collision avoidance safety rules. In this embodiment, key information can be obtained by parsing the multimodal information fusion result, and whether the preset bridge collision avoidance safety rules are met based on the key information. If not, bridge collision avoidance warning information is generated. Specifically, the preset bridge collision avoidance safety rules include minimum safety distance rules and speed limit rules. Among them, the minimum safety distance rule refers to setting a safety distance based on the structure of the bridge and the location of the underwater facilities to ensure that the ship will not collide or be damaged when approaching the bridge pier. This safety distance is usually dynamically calculated or preset based on factors such as the height of the bridge, tidal changes, the size and speed of the ship. The speed limit rule refers to setting a maximum limit on the navigation speed based on the structure of the bridge and the conditions of the surrounding waters to ensure that the ship can stop or adjust the speed in time when approaching the bridge to avoid safety risks caused by too fast. In order to adapt to the rules in different situations, this embodiment sets a manually adjustable maximum threshold for the position and speed obtained after multimodal fusion as the minimum safety distance and speed limit rule. When this maximum threshold is exceeded, it can cooperate with relevant hardware to generate bridge collision avoidance warning information to notify the bridge monitoring center or ship driver.

[0077] In some embodiments, the complete process of the method provided in the embodiments of the present application is as follows Figure 5 As shown. First, the camera captures the image of the ship (camera detection data), and the radar captures the full-scene point cloud data (millimeter-wave radar detection data), and the data at both ends are aligned in time and space. Then, the camera detection data is preprocessed for object imaging, and at the same time, the millimeter-wave radar obtains the precise distance information of the ship and maps it to the image. Next, through multimodal data fusion processing, the ship's heading, speed, and real-time position data are integrated. Subsequently, the integrated data is transmitted to the decision center. In the decision center, the received data is deeply processed and evaluated using the preset anti-collision safety rules, and finally it is determined whether the ship has the potential risk of deviation or speeding.

[0078] In the embodiment of the present application, the spatiotemporal clustering algorithm is an algorithm for identifying and classifying data points, which are usually spatially and temporally related. Through this algorithm, useful information can be effectively extracted from complex data sets. Figure 6 In the figure, you can see a group of points of different colors, which represent different data categories after clustering. Each category of points corresponds to a different target. Next, the data after spatiotemporal clustering will be subjected to target detection to identify and locate the target. By analyzing the data of different categories, the location and state of the target can be accurately detected, and the target can be abstracted as a point (D1, D2, and D3).

[0079] At the same time, the image data (camera detection data) is processed using the object imaging processing algorithm. After the data has passed through the radar spatiotemporal mechanism processing and the image processing module and completed their tasks respectively, it enters the perception fusion stage. Perception fusion comprehensively processes information from different sources to obtain more accurate and comprehensive results. At this stage, the results of the spatiotemporal clustering processing algorithm and the object imaging processing algorithm are fused, and the corresponding target (D3) in the radar data is located through the target center of mass (A1) of the object imaging. By comprehensively analyzing this information, more accurate target detection and positioning are provided. Ultimately, the result of perception fusion will output detailed information about a target A1 (D3). Figure 6 In the figure, we can see an image processed by perceptual fusion, showing the deviation angle of target A1 (D3). =1.5°, distance D=75.3m and speed V=5m / s.

[0080] It can be seen that the method provided in the embodiment of the present application images the ship by applying an object imaging algorithm, filters out irrelevant background, retains only the ship information, and then fuses the imaged ship with the data of the millimeter-wave radar. When applied in the bridge collision avoidance scenario, it can effectively improve the real-time performance.

[0081] Reference Figure 7 The embodiment of the present application provides a bridge anti-collision detection system based on multi-modal fusion of object imaging, the system comprising:

[0082] The first module 710 is used to obtain camera detection data and millimeter wave radar detection data, wherein the camera detection data includes ship image information and the millimeter wave radar detection data includes ship distance information;

[0083] The second module 720 is used to process the camera detection data through the object imaging algorithm to obtain an image of the object area of ​​interest;

[0084] The third module 730 is used to process the millimeter wave radar detection data through a spatiotemporal clustering algorithm to obtain target detection information;

[0085] The fourth module 740 is used to fuse the image of the object region of interest and the target detection information to obtain a multimodal information fusion result;

[0086] The fifth module 750 is used to generate bridge collision avoidance warning information according to the multimodal information fusion result and the preset bridge collision avoidance safety rules.

[0087] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0088] The embodiment of the present application also provides a bridge anti-collision detection system based on multimodal fusion of object imaging, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned bridge anti-collision detection system based on multimodal fusion of object imaging when executing the computer program. The bridge anti-collision detection system based on multimodal fusion of object imaging can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0089] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0090] See also Figure 8 , Figure 8 The hardware structure of a bridge anti-collision detection system based on multi-modal fusion of object imaging in another embodiment is illustrated. The bridge anti-collision detection system based on multi-modal fusion of object imaging includes:

[0091] The processor 810 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0092] The memory 820 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 820 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 820, and the processor 810 calls and executes the bridge collision avoidance detection method based on multi-modal fusion of object imaging in the embodiment of this application;

[0093] Input / output interface 830, used to implement information input and output;

[0094] Communication interface 840, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0095] A bus 850 that transmits information between various components of the device (e.g., the processor 810, the memory 820, the input / output interface 830, and the communication interface 840);

[0096] The processor 810 , the memory 820 , the input / output interface 830 and the communication interface 840 are connected to each other in communication within the device via a bus 850 .

[0097] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0098] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0099] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0100] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0102] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0103] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0105] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0108] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A bridge anti-collision detection method based on multi-modal fusion of object imaging, characterized in that: The method comprises the following steps: Acquire camera detection data and millimeter wave radar detection data, wherein the camera detection data includes ship image information, and the millimeter wave radar detection data includes ship distance information; Processing the camera detection data by an object imaging algorithm to obtain an image of the object area of ​​interest; Processing the millimeter wave radar detection data by a spatiotemporal clustering algorithm to obtain target detection information; Fusing the object of interest region image and the target detection information to obtain a multimodal information fusion result; Generate bridge anti-collision warning information according to the multimodal information fusion result and preset bridge anti-collision safety rules; Before the step of processing the millimeter wave radar detection data by using a spatiotemporal clustering algorithm, the method further includes the following steps: Preprocessing the millimeter wave radar detection data according to the longitudinal coordinate value of the ship to obtain first radar detection data to be clustered; The processing of the millimeter wave radar detection data by a spatiotemporal clustering algorithm to obtain target detection information includes: Determine a preset clustering threshold according to the spatial feature information; Performing a first filtering on all the first radar detection data to be clustered according to the preset clustering threshold to obtain second radar detection data to be clustered; Performing a second filtering on all the second radar detection data to be clustered according to the time frame feature to obtain the target detection information; The step of fusing the image of the object of interest region and the target detection information to obtain a multimodal information fusion result includes: Performing data alignment on the image of the object of interest region and the target detection information; Associating the aligned image of the region of object of interest with the target detection information, wherein the associating process is used to match the same obstacle; Environmental perception information is generated as the multimodal information fusion result according to the association result, the three-dimensional object position information in the image of the object area of ​​interest, and the obstacle distance, obstacle movement speed and obstacle movement angle of the target detection information.

2. The method according to claim 1, characterized in that After the step of acquiring the camera detection data and the millimeter wave radar detection data, the method further includes the following steps: Performing time synchronization processing on the camera detection data and the millimeter wave radar detection data; The camera detection data and the millimeter wave radar detection data after time synchronization processing are aligned in spatial coordinates.

3. The method according to claim 1, characterized in that The step of processing the camera detection data by an object imaging algorithm to obtain an image of the object region of interest includes: Training a preset neural network to obtain convolution kernel parameters, wherein the convolution kernel parameters include features of an object of interest; Constructing a feature filter according to the convolution kernel parameters; The camera detection data is subjected to background filtering by the feature filter to obtain the object of interest region image, wherein the object of interest region image includes the ship position information.

4. The method according to claim 1, characterized in that: The first filtering of all the first radar detection data to be clustered according to the preset clustering threshold to obtain second radar detection data to be clustered includes: Calculating the deviation between each of the first radar detection data to be clustered and the cluster center; The first radar detection data to be clustered whose deviation is less than or equal to the preset clustering threshold is used as the second radar detection data to be clustered.

5. The method according to claim 1, characterized in that The generating bridge anti-collision warning information according to the multi-modal information fusion result and the preset bridge anti-collision safety rules includes: Analyze the multimodal information fusion result to obtain key information; Determine whether the preset bridge anti-collision safety rules are met based on the key information, and if not, generate the bridge anti-collision warning information.

6. A bridge anti-collision detection system based on multi-modal fusion of object imaging, characterized in that: The system comprises: The first module is used to obtain camera detection data and millimeter wave radar detection data, wherein the camera detection data includes ship image information and the millimeter wave radar detection data includes ship distance information; The second module is used to process the camera detection data through an object imaging algorithm to obtain an image of the object area of ​​interest; The third module is used to process the millimeter wave radar detection data through a spatiotemporal clustering algorithm to obtain target detection information; The fourth module is used to fuse the image of the object of interest region and the target detection information to obtain a multimodal information fusion result; A fifth module is used to generate bridge anti-collision warning information according to the multimodal information fusion result and preset bridge anti-collision safety rules; Before the step of processing the millimeter wave radar detection data by using the spatiotemporal clustering algorithm, the third module is further used to perform the following steps: Preprocessing the millimeter wave radar detection data according to the longitudinal coordinate value of the ship to obtain first radar detection data to be clustered; The processing of the millimeter wave radar detection data by a spatiotemporal clustering algorithm to obtain target detection information includes: Determine a preset clustering threshold according to the spatial feature information; Performing a first filtering on all the first radar detection data to be clustered according to the preset clustering threshold to obtain second radar detection data to be clustered; Performing a second filtering on all the second radar detection data to be clustered according to the time frame feature to obtain the target detection information; The step of fusing the image of the object of interest region and the target detection information to obtain a multimodal information fusion result includes: Performing data alignment on the image of the object of interest region and the target detection information; Associating the aligned image of the region of object of interest with the target detection information, wherein the associating process is used to match the same obstacle; Environmental perception information is generated as the multimodal information fusion result according to the association result, the three-dimensional object position information in the image of the object area of ​​interest, and the obstacle distance, obstacle movement speed and obstacle movement angle of the target detection information.

7. A bridge anti-collision detection system based on multi-modal fusion of object imaging, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 5.