All-time bridge anti-ship collision warning method with visible light and thermal imaging camera

Abstract

The application discloses a full-time bridge anti-ship collision warning method of visible light and thermal imaging camera, comprising a visible light and thermal imaging lens as a data acquisition device, a trajectory tracking and real-time data analysis and processing three parts. First, according to the real-time environmental conditions of the channel and the lens recognition accuracy, a double-lens adaptive decision system is used to switch the input source of the visible light and thermal imaging camera autonomously, then a multi-target ship identification and tracking algorithm based on deep learning is used to locate the ship in real time and display the trajectory, finally, according to a large number of historical trajectories of ships in the bridge area, a sound and light warning is given to the ship trajectory higher than the safety threshold. The application has the advantages of real-time convenience, all-weather monitoring, good robustness and non-contact, and overcomes the problem that the existing method is difficult to be applied to bridge anti-ship collision warning in poor illumination environment, and has good prospects for wide application in practical bridge safety monitoring.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of bridge health monitoring and intelligent shipping, and particularly relates to a full-period bridge self-adaptive anti-ship collision early warning method of visible light and thermal imaging cameras. BACKGROUND

[0002] Bridges provide convenience for traffic in the vicinity of water, and the navigation of waterway traffic is directly related to the safety of the bridge bottom. A large number of ships pass through the bridge every day, and the bridge piers at the bottom of the bridge reduce the passability of these ships in the river, divide the waterway, and are more likely to cause collisions between the ships when deviating and the bridge piers, which can seriously damage the bridge and even cause it to collapse. In order to prevent accidental collisions between ships and bridge piers during navigation, and to improve the reliability and safety of ship navigation in the vicinity of the bridge, some protective hard structures are usually built around the bridge piers to avoid collisions between ships and bridge piers. However, this passive anti-ship collision method cannot prevent collisions from occurring, and there is still a great risk of threats to the bridge. The active anti-ship collision early warning system of the bridge analyzes and identifies environmental conditions, ship moving trajectories, driving states and other factors through various sensors installed on the bridge, and long-term all-weather key technologies to ensure the safety of the bridge, which can actively warn dangerous ships in advance and greatly reduce the occurrence of ship-bridge collision accidents. However, most of the current active anti-ship collision methods are applied in good light conditions.

[0003] Most of the existing bridge anti-ship collision methods are based on traditional single sensors, including video, GPS and laser, etc. This kind of method generally has limited visibility, and has obvious effect on high-illumination conditions during the day, and limited early warning capability in poor visibility environments such as heavy fog and night when ship-bridge collision accidents are prone to occur. Ship GPS positioning is manually turned on by the ship driver, and accidents often occur because the driver forgets to turn it on. The distance between the ship and the bridge is relatively far, and the ship anti-ship collision warning needs to give a warning to the deviating ship in advance to provide enough time for the ship driver to correct the course. Laser has a relatively fast energy loss, and has a great influence on the early warning capability of large bridges for long-distance ships, so it is urgent to respond to the method to solve the above problems.

[0004] With the rapid development of computer vision technology, visual-based measurement methods have begun to be applied to structural deformation detection. Such methods are low in cost, high in visibility, and have been proven to achieve the required recognition accuracy in some complex environment safety monitoring operations. In order to solve the problem that traditional visible light cameras are difficult to provide effective visual monitoring at night or in low light, infrared thermal imager and visible light are combined as a breakthrough point. Infrared thermal imaging technology is to collect the temperature of all objects in nature whose temperature is higher than absolute zero, and the difference between the object and the background radiation is imaged after signal processing, which is independent of the lighting conditions of the monitored object. In order to solve the problem that traditional optical measurement methods are easily affected by light and shielding, some scholars believe that deep learning methods widely studied in recent years can be used to solve this problem. These methods effectively use various methods in deep learning to improve the stability of target recognition and tracking, and further research can be conducted on these methods in the positioning and tracking of ships in the near bridge area. On the other hand, in order to solve the problem that there are few reasonable positioning points in large water areas and it is difficult to determine the real-time driving state of the ship, the above trajectory detection helps to find ships with high bridge collision risk, but due to the wide waterway, the prediction of the dangerous behavior of the ship also faces challenges. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a visible light and thermal imaging camera full-time bridge adaptive anti-ship collision warning method, which uses a dual-lens adaptive decision system to automatically switch the input source of the visible light and thermal imaging camera according to the real-time environmental conditions of the waterway and the recognition accuracy of the lens, uses a deep learning-based multi-target ship recognition and tracking algorithm to locate the ship in real time and display the trajectory, and issues an audible and visual warning for the ship trajectory that is higher than the safety threshold based on a large number of historical trajectories of ships in the near bridge area. The present application can quickly and conveniently identify and track ships at all times, and can achieve all-weather efficient and accurate audible and visual warnings for off-course ships.

[0006] The present application adopts the following technical solutions to solve the above technical problems:

[0007] The visible light and thermal imaging camera full-time bridge adaptive anti-ship collision warning method proposed by the present application comprises the following steps:

[0008] S1, the visible light and thermal imaging dual-lens camera is erected on the bridge or under the bridge facing the waterway, so that the field of view of the dual-lens camera can cover the near-bridge waterway area to be monitored, the camera lens position is adjusted to focus on the target waterway, and the data collection of all ships approaching the bridge is ensured all day long, realizing real-time collection and saving of visible light and thermal imaging video image sequences of target ships.

[0009] S2, a visible light and thermal imaging ship recognition model is constructed based on a YOLO v5s network.

[0010] S3, an adaptive decision system of visible light and thermal imaging lenses is constructed by using the light intensity classification network to realize adaptive switching of visible light and thermal imaging lenses in different light environments and obtain more effective and more accurate ship data input.

[0011] S4, the real-time video collected by the visible light and thermal imaging cameras is introduced into the ship identification model in step S2, the target ship position is identified by using the visible light and thermal imaging ship feature extraction network, the center coordinates of the marking box of each frame of the real-time lens input are obtained, and the real-time ship position is recorded.

[0012] S5, according to the center coordinates of the marking box of the target ship in the current frame, the center points of the detection boxes in each frame of the video are matched and connected by using the multi-target tracking algorithm based on DeepSORT, and the actual sailing track of the ship is obtained.

[0013] S6, for the working condition of ship collision with bridge, it is necessary to set the dangerous ship sailing range. Since most of the ship sailing tracks are normal, the number of ship track data that can cause danger to the bridge is small, and it is difficult to form a judgment of the dangerous ship sailing range. Combined with a large amount of historical normal ship track data, the safe driving range of the ship in the near-bridge channel area is generated, the current ship track data is evaluated in real time, and when the predetermined safety threshold range is exceeded, an audible and visual alarm is issued to alert the deviated ship.

[0014] Further, in step S2, the process of constructing the ship identification model based on the YOLO v5s network includes the following steps:

[0015] S201, target ship feature extraction in the channel, the center of the identification box is taken as the real-time position point of the whole ship;

[0016] S202, for the working condition of all-period bridge anti-ship collision, a pair of dual-lens images containing different light intensities is collected, the points are manually marked, a visible light and thermal imaging ship dataset in the all-period near-bridge area is prepared in the format of PASCAL VOC, the ship dataset is introduced into the YOLO v5s network for training and verification, the training parameters are adjusted according to the identification effect under different light intensities, and the trained ship identification model is obtained.

[0017] Further, in step S3, the specific content of constructing the visible light and thermal imaging lens adaptive decision system is: taking the visible light camera as the light intensity input source, synchronously collecting the all-weather ship navigation images by the dual lens, and comparing the ship recognition accuracy of the dual lens under various light intensity transformations according to the visible light and thermal imaging ship recognition model experiment, establishing the image light classification dataset of the input source with higher recognition accuracy, loading the light classification model trained based on Inception-ResNet v2 every fixed frame number, introducing the ResNet structure into the Inception model, and simultaneously adopting the 1x1 convolution kernel for dimension reduction processing to reduce the calculation amount. The introduction of the ResNet structure can reduce the overfitting and gradient disappearance phenomenon caused by the increase of the number of layers, increase the adaptability of the light intensity sub-network scale, and complete the adaptive switching of the visible light and thermal imaging lens under different light environments.

[0018] Further, in step S5, the specific content of obtaining the actual ship navigation trajectory is: obtaining the target ship bounding box of the current frame according to the visible light and thermal imaging ship recognition model, taking the center point of the bounding box as the real-time position of the ship, using the multi-target tracking algorithm based on DeepSort to predict the correlation degree of the target ship and the target ship in the next frame bounding box, if the correlation degree reaches the predicted correlation degree threshold, it is determined that the ship recognition result of the next frame is correct; otherwise, the predicted trajectory and the re-detected target ship are matched by IoU. When the ship recognition result is correct, the center points of the detection boxes in each frame of the video are connected to obtain the actual navigation trajectory of the ship.

[0019] Sort combines Kalman filtering and Hungarian algorithm, and takes the intersection IoU of the detection result and the tracking result as the cost matrix of the Hungarian algorithm, to realize a simple and effective multi-target tracking framework. The multi-target tracking algorithm of DeepSort adds a deep correlation feature to the Sort algorithm, combines the motion information and the obvious features of the target in the image space as the matching standard of the same target, reduces the problem of multi-target ship matching confusion, and changes the matching mechanism of Sort from the original IoU cost matrix matching to the combination of cascade matching and IoU matching. The introduction of cascade matching reduces the error caused by the mutual occlusion of ships, and obtains the accurate displacement trajectory of the target ship.

[0020] Further, in step S5, for the correlation degree calculation of the ship target motion information, the specific steps are as follows:

[0021] S501, the correlation degree of the ship target motion information is described by the detection box and the tracking box, and the specific formula is as follows:

[0022]

[0023] wherein, (y i ,Si ) is the projection mapping of the i-th ship trajectory to the detection space, d j is the position of the j-th detection frame, y i is the target position predicted by the i-th tracking trajectory, S i is the observation space at the current time of the Kalman filter prediction of the i-th ship trajectory covariance matrix.

[0024] S502, thresholding by Mahalanobis distance to exclude association, t (1) is the distance metric indication value, and if the Mahalanobis distance is less than the threshold value, the association is successful, and the formula is as follows:

[0025]

[0026] S503, the existence of jitter under the influence of external factors such as strong wind will make the Mahalanobis method invalid, so it is necessary to introduce appearance model matching and cosine distance to find the feature vector r i of each detection target d j . For tracker i, the feature vectors of the 100 frames before and after the k-th trajectory are stored in the feature library R i , that is, L = 100. Finally, the minimum cosine distance between the i-th tracking trajectory r k (i) and the j-th detector r j T is calculated. When the distance between them is less than or equal to a certain threshold value, it means that they are related, and the formula is as follows:

[0027]

[0028] S504, determine whether to associate according to the threshold value:

[0029]

[0030] S505, combined with the association of target appearance information, linearly weighted two ways to calculate the matching degree between the final detection and tracking trajectory:

[0031] c i,j = λd (1) (i,j) + (1-λ)d (2) (i,j).

[0032] Further, in step S6, a set of double-lens adaptive decision system is used as a bridge for the blind area period, to determine the current environment best input source, and the visible light and thermal imaging ship identification model is used to output real-time ship identification frame, the center of the ship identification frame is used as the actual position of the whole ship, the multi-target tracking algorithm based on DeepSort is used to automatically judge the correlation degree of the target ship and the target ship in the next frame marking frame, when the correlation degree reaches the threshold, the ship center point is connected, and the ship navigation trajectory is output; meanwhile, a large number of visible light and thermal imaging lens camera position safety navigation ship trajectory samples are collected, a double-lens camera anti-ship collision early warning safety zone is established, and the ship navigation trajectory is processed in real time, and an audible and light warning is given to the ship whose above-mentioned ship navigation trajectory exceeds the safe navigation range.

[0033] Further, the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the full-period bridge adaptive anti-ship collision early warning method of the visible light and thermal imaging camera.

[0034] Further, the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the full-period bridge adaptive anti-ship collision early warning method of the visible light and thermal imaging camera.

[0035] Compared with the prior art, the present application has the following technical effects:

[0036] (1) By comprehensively analyzing real-time light environment and other factors, using the object recognition advantages of visible light cameras and thermal imaging cameras under different light, the effectiveness of lens switching in the ship channel camera decision system under different light periods is realized, the real-time optimal data input of the ship in the near-bridge area is realized, and the shortcomings of the traditional method for bridge anti-ship collision warning under poor light environment are overcome.

[0037] (2) The multi-target ship identification and tracking method fuses the latest target detection algorithm and target tracking algorithm, and uses the visible light and thermal imaging lens ship image sequence as an effective input source to identify and detect ships and make early warning for dangerous trajectory ships, which has the advantages of strong real-time performance, high robustness and all-weather, and is suitable for ship yaw monitoring to ensure the safety of the bridge substructure. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The principle diagram of the full-period bridge adaptive anti-ship collision early warning method of the visible light and thermal imaging camera of the embodiment of the present application.

[0039] Figure 2 The principle diagram of the light intensity database classification method.

[0040] Figure 3 A schematic diagram of visible light and thermal imaging lens classification data in various lighting environments.

[0041] Figure 4 A flowchart of a ship identification, tracking and early warning method based on deep learning multi-target tracking.

[0042] Figure 5 A schematic diagram of the training results of a visible light and thermal imaging ship identification model based on a YOLO v5s network.

[0043] Figure 6 A flowchart of processing trajectories based on a DeepSORT multi-target ship tracking algorithm.

[0044] Figure 7 A schematic diagram of a near-bridge area ship-bridge collision early warning system.

[0045] Figure 8 A schematic diagram of a monitoring range and alert line based on historical ship trajectories. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below by reference to the accompanying drawings are exemplary and are used only to explain the present application, and cannot be interpreted as a limitation of the present application.

[0047] Reference Figure 1 , a full-time bridge adaptive anti-ship collision early warning method for visible light and thermal imaging cameras, the yawing ship early warning method comprising the following steps:

[0048] S1, erecting a visible light and thermal imaging dual-lens camera on the bridge deck or under the bridge opposite the fairway, so that the field of view of the dual-lens camera can cover the required monitoring area near the bridge fairway, adjusting the camera lens position to focus on the target fairway, ensuring that each lens can collect data all-weather for all ships approaching the bridge, realizing real-time collection and saving of visible light and thermal imaging video image sequences of target ships.

[0049] S2, constructing a visible light and thermal imaging ship identification model based on a YOLO v5s network, the specific content being:

[0050] S201, extracting the features of target ships in the fairway, taking the center of the identification frame as the real-time position point of the whole ship;

[0051] S202, for the whole period of bridge anti-ship collision, collect double-lens image pairs under different light intensities, manually mark points, use the PASCAL VOC format to produce a full-period near-bridge visible light and thermal imaging ship dataset, import the ship dataset into the YOLO v5s network for training and verification, adjust the training parameters according to the recognition effect under different light intensities, and obtain a trained ship recognition model.

[0052] S3, using a visible light camera as a light intensity input source, synchronously collecting all-weather ship navigation images by double-lens and comparing the double-lens ship recognition accuracy under various light intensity transformations according to the visible light and thermal imaging ship recognition model experiment, establishing an image light classification dataset for the input source with higher recognition accuracy, loading a light classification model based on Inception-ResNetv2 training every fixed frame number, introducing ResNet structure into the Inception model, and using 1x1 convolution kernel for dimension reduction processing to reduce the amount of calculation, introducing ResNet structure can reduce overfitting and gradient disappearance caused by increasing the number of layers, increase the adaptability of the light intensity division network scale, and complete the adaptive switching of visible light and thermal imaging lenses under different light environments.

[0053] S4, importing the real-time video collected by the visible light and thermal imaging cameras into the ship recognition model in step S2, using the visible light and thermal imaging ship feature extraction network to recognize the target ship position, obtaining the center coordinates of the marked frame of each frame of real-time lens input measuring the ship, and recording the real-time ship position.

[0054] S5, according to the center coordinates of the target ship in the current frame, using a multi-target tracking algorithm based on DeepSORT to match and connect the center points of the detection frames in the video, and obtaining the actual navigation trajectory of the ship.

[0055] S6, for the ship collision with the bridge, it is necessary to set the dangerous ship navigation range, since most of the ship navigation trajectories are normal, the number of ship trajectory data that can cause danger to the bridge is very small, and it is difficult to form a judgment of the dangerous ship navigation range. Combined with a large amount of historical normal ship trajectory data, a safe driving range of the ship in the near-bridge channel area is generated, the current ship trajectory data is evaluated in real time, and when the predetermined safety threshold range is exceeded, an audible and visual alarm is issued to alert the deviated ship.

[0056] Embodiment 1:

[0057] The application will be further described in conjunction with the drawings and specific embodiments.

[0058] The application relates to a full-period bridge adaptive anti-ship collision warning method of a visible light and thermal imaging camera. The specific embodiment of the application includes the following contents:

[0059] (i) Adaptive switching of visible light and thermal imaging lenses

[0060] The river channel for experimental data collection is a canal in the main urban area of a city, and a large number of ships pass through the canal at all times. The bridge is a city highway bridge, with a main span of more than 80 meters, a bridge width of more than 30 meters, and a navigation height of 7 meters, and is a steel truss bridge. First, visible light and thermal imaging cameras are fixed on the bridge or under the bridge on the anti-collision side, and are arranged on the bridge or under the bridge opposite the navigation channel. The field of view of the dual-lens camera can cover the required monitoring area near the bridge navigation area, and is used to capture the water surface navigation area in real time. The thermal imaging camera uses a non-cooled focal plane micro-bolometer, with a resolution of 640x480, a video frame rate of 30Hz, a thermal sensitivity (NETD) of ≤50mK@25℃, a focal length of 25mm, and an effective field of view angle of 14.9°x11.2°, and an angular resolution of 0.68mrad. The visible light camera uses a 1 / 2.7"Progressive Scan CMOS fixed-focus 8mm lens, with an effective resolution of 2 million pixels. A real-time data processing terminal and an audible and light alarm device are installed on the bridge, and the dual-lens camera, audible and light alarm device, and data processing terminal are connected. The visible light camera is used to obtain rich environmental light information, and the visible light camera for collecting ship information is also used as a light intensity recognition source. As shown in Figure 2 , when the light conditions are good, the bridge anti-ship collision system input source is automatically switched to the visible light camera which has an advantage in light conditions; otherwise, in poor light conditions such as night, the input source is automatically switched to the thermal imaging camera. For the fuzzy area of environmental light intensity transformation, according to the ship recognition accuracy of different illuminance collected by experiments, as shown in Figure 3 , a light environment classification data set is established to train a light classification model to realize adaptive switching of the optimal input source.

[0061] (ii) Multi-target ship identification and tracking

[0062] As shown in Figure 4 , the overall process of the ship identification, tracking, and early warning method based on deep learning multi-target tracking. First, the YOLO v5s network can quickly identify the position and size of the target marking box. It is necessary to create a manual marking of 1000 visible light and thermal imaging ships in each lens advantage period of the channel, and divide the two kinds of images into training set and test set according to the ratio of 8:2 to establish a visible light and thermal imaging ship target detection algorithm. The training results are as follows Figure 5The center coordinates of the detected object are calculated, such as the coordinates of the four corners of the ship marker frame P1(x1, y1), P2(x2, y2), P3(x3, y3), P4(x4, y4), and the center point coordinates are the average of the four point coordinates. The position of the center point is recorded as the position point of the whole ship. The DeepSort algorithm process is as shown in the figure Figure 6 As shown in the figure, first, the detector identifies the target in each image, then initializes the tracker according to the detection result, uses Kalman filtering to predict the trajectory of the target, inputs the predicted trajectory into the ship target detection of the next frame, matches the trajectories between adjacent images according to the Hungarian algorithm, updates the prediction result, and repeatedly executes the detection, prediction and matching update process. Deep Sort uses the state vector X = [u, v, r, h, u', v', r', h'] as the direct observation model of the target, which represents the state of the ship trajectory at a certain time. Where u and v represent the center coordinates of the target detection frame, r and h represent the width-height ratio and height of the detection frame respectively, and (u, v', r', h') represents the predicted result in the next frame. Assign a tracking target to each detected ship target in each frame. When a new ship detection result appears in a certain frame, a new tracking target is created for that frame. If the prediction result of the tracker matches the detection result for three consecutive frames, it is considered that a new tracking trajectory has appeared, otherwise the trajectory is deleted from the tracker list, and the process is repeated.

[0063] (Three) Off-course ship early warning

[0064] Ship trajectories with high bridge collision risk are defined as abnormal ship trajectories, and a dangerous trajectory rule is developed for bridge anti-ship collision early warning. For ships passing through the channel, the gap between the bottom structures of the adjacent piers of the bridge is the safe navigation range of the ships in the channel, as shown in the figure Figure 7 When the ship shows a trend of deviating from this area, it is determined that the ship has a high risk of colliding with the bridge. Therefore, the primary goal is to establish the boundary of the safe navigation area for the relative angle and position of the dual-channel camera input source, i.e. the virtual warning line of the early warning system. In order to generate the boundary of the safe navigation area, a large amount of historical trajectory data of safe navigation in the channel is needed as the basis, and since the visible light and thermal imaging camera angles and positions are slightly different, ship trajectory samples need to be collected during the effective working period of each dual-channel camera. Use the ship identification and tracking model proposed above, and at the same time draw the safe navigation trajectory of the ship passing through the area collected later in the algorithm, and identify the boundary of the output batched ship navigation trajectory, establish the bridge anti-ship collision early warning warning line of the visible light and thermal imaging camera parallel to the channel direction, and the method is as shown in the figure Figure 8The shown. For the lens ship trajectory, if there is a sliding window corresponding to the ship to cross the bridge point more than the prompt line, do flash warning, such as more than the historical warning line, determine that the trajectory is abnormal, the ship light and sound alarm at the same time, until its corresponding to the bridge point back to the safe area stop alarm.

[0065] Embodiment 2:

[0066] The computer readable storage medium of the embodiment stores a computer program, which is executed by a processor to implement the steps in the full-time bridge adaptive anti-collision warning method of the visible light and thermal imaging camera of embodiment 1.

[0067] The computer readable storage medium of the embodiment can be an internal storage unit of a terminal, such as a hard disk or a memory of the terminal; the computer readable storage medium of the embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer readable storage medium can include both an internal storage unit and an external storage device of the terminal.

[0068] The computer readable storage medium of the embodiment is used to store a computer program and other programs and data required by the terminal, and can also be used to temporarily store data that has been output or will be output.

[0069] Embodiment 3:

[0070] The computer device of the embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the steps in the full-time bridge adaptive anti-collision warning method of the visible light and thermal imaging camera of embodiment 1.

[0071] In the embodiment, the processor can be a central processing unit, and can also be other general-purpose processors, digital signal processors, application-specific integrated circuits, ready programmable gate arrays, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The memory can include read-only memory and random access memory, and provide instructions and data to the processor, and a part of the memory can also include non-volatile random access memory, such as a memory that can also store device type information.

[0072] Those skilled in the art will appreciate that embodiments disclosed herein can be provided as methods, systems, or computer program products. Accordingly, embodiments can be provided in the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Also, embodiments can be provided in the form of computer program products embodied on one or more computer-usable storage media (including, but not limited to, disk memory and optical memory) having computer usable program code embodied thereon.

[0073] Embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0074] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0076] Those skilled in the art will appreciate that implementing all or part of the methods described above in the embodiments can be accomplished by way of computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams.

[0077] The examples described herein are merely illustrative of the preferred embodiments of the present application and are not intended to limit the scope of the present application. Any variation and modification of the present application, which do not depart from the spirit and scope of the present application, should be construed as falling within the scope of the present application.

Claims

1. A full-time bridge adaptive anti-ship collision warning method for visible light and thermal imaging cameras, characterized in that, The method comprises the following steps: S1, erecting a visible light and thermal imaging dual-lens camera on the bridge deck or under the bridge opposite the channel, so that the field of view of the dual-lens camera can cover the required monitoring area of the channel near the bridge, adjusting the lens position of the camera to focus on the target channel, and collecting and saving the visible light and thermal imaging video image sequences of the target ship in real time; S2, constructing a visible light and thermal imaging ship identification model based on a YOLO v5s network; S3, constructing a visible light and thermal imaging lens adaptive decision system using an illumination intensity classification network; specifically: The specific content of constructing the visible light and thermal imaging lens adaptive decision system is: taking the visible light camera as the illumination intensity input source, synchronously collecting all-weather ship navigation images by the dual-lens, and comparing the ship identification accuracy of the dual-lens under various illumination intensity transformations according to the visible light and thermal imaging ship identification model experiment, establishing an image illumination classification data set for the input source with higher identification accuracy, loading the illumination classification model trained based on Inception-ResNet v2 at fixed frame intervals, introducing the ResNet structure into the Inception model, and simultaneously using a 1x1 convolution kernel for dimension reduction processing; S4, importing the real-time video collected by the visible light and thermal imaging camera into the ship identification model in step S2, identifying the target ship position using the visible light and thermal imaging ship feature extraction network, obtaining the labeled box center coordinates of each frame of the measured ship of the real-time lens input, and recording the real-time ship position; S5, according to the labeled box center coordinates of the target ship of the current frame, matching and connecting the center points of the ship detection boxes of each frame in the video using a multi-target tracking algorithm based on DeepSort, and obtaining the actual navigation trajectory of the ship; S6, using a set of dual-lens adaptive decision system as a supplement to the bridge blind area period, determining the best input source in the current environment, using a multi-target tracking algorithm, outputting a large number of monitored bridge full-ship historical navigation trajectories, and generating a safe navigation range of the ship in the near-bridge area according to the collected historical safe navigation data, and when exceeding the range, issuing an audible and visual alarm to alert the off-course ship to correct the navigation direction in time.

2. The all-time bridge adaptive anti-ship collision warning method of visible light and thermal imaging camera according to claim 1, characterized in that, In step S2, the ship identification model comprises the following steps: S201, extracting the features of the target ship in the channel, and taking the center of the identification box as the real-time position point of the whole ship; S202, for the working condition of bridge ship collision prevention, collect dual-lens image pairs under different illumination intensities, manually mark the points, use the PASCAL VOC format to make a visible light and thermal imaging ship data set for the full-time near-bridge area, import the ship data set into the YOLO v5s network for training and verification, adjust the training parameters according to the identification effect under different illumination intensities, and obtain the trained ship identification model.

3. The all-time bridge adaptive anti-ship collision warning method of visible light and thermal imaging camera according to claim 1, characterized in that, In step S5, the specific content of obtaining the actual sailing track of the ship is: obtaining the target ship marking box of the current frame according to the visible light and thermal imaging ship identification model, taking the center point of the marking box as the real-time position of the ship, using a multi-target tracking algorithm based on DeepSort to predict the correlation degree of the target ship and the target ship in the next frame marking box, if the correlation degree reaches the predicted correlation degree threshold, it is determined that the ship identification result of the next frame is correct, otherwise, the predicted track and the re-detected target ship are matched by IoU; when the ship identification result is correct, the center points of the detection boxes of each frame in the video are connected to obtain the actual sailing track of the ship.

4. The all-time bridge adaptive anti-ship collision warning method of visible light and thermal imaging camera according to claim 3, characterized in that, In step S5, the correlation degree of the ship target motion information is calculated, and the specific steps are as follows: S501, the correlation degree of the ship target motion information is calculated by the detection box and the tracking box, and the specific formula is as follows: Among them, (y i ,S i ) is the projection mapping of the i-th ship trajectory to the detection space, d j It is the position of the j-th detection frame, y i S is the target position predicted by the i-th tracking trajectory. i It is the observation space at the current moment predicted by the Kalman filter of the covariance matrix of the i-th ship trajectory; S502, thresholding is performed by Mahalanobis distance to exclude correlation, that is, whether the Mahalanobis distance is less than the distance measurement indication value is judged, and the specific formula is as follows: wherein t (1) is a distance metric indication value, d (1) (i,j) is a Mahalanobis distance; the association is successful if the Mahalanobis distance is less than a threshold value; S503, introduce appearance model matching and cosine distance to find the feature vector r of each detection target d i j ; for tracker i, the feature vector of 100 frames before and after the kth track is stored in the feature library R i , that is, L = 100; calculate the minimum cosine distance between the i th tracking track r k (i) and the j th detector r j T , when the distance is less than or equal to a certain threshold, it indicates that it is related, the formula is as follows:​ S504, whether the correlation is determined according to the threshold value: S505, while combining the correlation of the target appearance information, the matching degree between the final detection and tracking track is calculated by linear weighting of the two ways: c i,j = λd (1) (i,j) + (1 - λ)d (2) (i,j).

5. The all-time bridge adaptive anti-ship collision warning method of visible and thermal imaging cameras according to claim 1, characterized in that, In step S6, the real-time ship identification box is output by the visible light and thermal imaging ship identification model, the center of the ship identification box is taken as the actual position of the whole ship, the correlation degree of the target ship and the target ship in the next frame marking box is automatically judged by using a multi-target tracking algorithm based on DeepSort, when the correlation degree reaches the threshold value, the ship center points are connected, and the sailing track of the ship in the channel is output; meanwhile, a large number of safety sailing ship track samples of visible light and thermal imaging lens camera positions are collected, a double-lens camera anti-ship collision warning safety zone is established, and the ship sailing track is processed in real time, and a sound and light warning is given to the ship whose sailing track exceeds the safe navigation range.

6. A computer readable storage medium having stored thereon a computer program, characterized in that: The program is executed by the processor to realize the steps in the all-time bridge adaptive anti-ship collision warning method of the visible light and thermal imaging camera in any one of claims 1-5.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps in the all-time bridge adaptive anti-ship collision warning method of the visible light and thermal imaging camera in any one of claims 1-5.

Images