An abnormal behavior detection system and method based on multimodal information fusion

Abstract

The present invention provides an abnormal behavior detection system and method based on multimodal information fusion, wherein the method comprises the following steps: S1 collects AIS message information of ships in the target sea area and radar monitoring information of the radar system; S2 performs a primary abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information to obtain a primary behavior detection result of the ship; S3 further collects video image information of the target ship according to the positioning information of the target ship for the target ship whose primary behavior detection result is abnormal; S4 performs a secondary behavior detection process according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship. The present invention is helpful to improve the reliability of the ship supervision system for the supervision of ships in the target sea area.

Description

Technical Field

[0001] The present invention relates to the technical field of sea area supervision, and in particular to an abnormal behavior detection system and method based on multimodal information fusion. Background Art

[0002] At present, most of the supervision systems for ships in the sea area rely on the AIS data transmitted by the ship's Automatic Identification System (AIS) to detect the behavior of ships in the sea area. However, in the technical solutions of the prior art that rely solely on AIS data for analyzing ships in the sea area, when the ship-borne AIS equipment fails to operate normally, the ship-borne AIS is manually turned off, the AIS data is inaccurate, or the ship is not equipped with AIS equipment, it is impossible to accurately analyze the behavior of ships in the sea area, affecting the reliability of the ship supervision system. Summary of the invention

[0003] In view of the technical problem that the above-mentioned technical solution of analyzing ships in sea areas only relying on AIS data has insufficient reliability, the present invention aims to provide an abnormal behavior detection system and method based on multimodal information fusion.

[0004] The purpose of the present invention is achieved by the following technical solutions:

[0005] In a first aspect, the present invention provides an abnormal behavior detection method based on multimodal information fusion, comprising:

[0006] S1 collects AIS message information of ships in the target sea area and radar monitoring information of radar system;

[0007] S2 performs an abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information, and obtains a behavior detection result of the ship;

[0008] S3, for a target ship whose behavior detection result is abnormal, further collects video image information of the target ship according to the positioning information of the target ship;

[0009] S4 performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship.

[0010] In one implementation, step S1 specifically includes:

[0011] Receive AIS message information transmitted by ships in the target sea area, where the AIS message information contains ship ID and ship positioning data;

[0012] The radar monitoring information transmitted by the radar system for monitoring the target sea area is received, wherein the radar monitoring information includes the positioning information of the ships monitored in the target sea area.

[0013] In one implementation, step S2 specifically includes:

[0014] S21 performs positioning consistency judgment based on the ship positioning data in the AIS message information at the specified time and the ship positioning information in the radar monitoring information to obtain a positioning consistency judgment result;

[0015] S22: acquiring first ship trajectory data according to the ship positioning data in the AIS message information within a set time period; acquiring second ship trajectory data according to the positioning data of the corresponding ship in the radar monitoring information within the set time period; performing trajectory consistency judgment according to the first ship trajectory data and the second ship trajectory data to obtain a trajectory consistency judgment result;

[0016] S23: When the positioning consistency judgment result or the trajectory consistency judgment result of the ship is abnormal, a behavior detection result of the ship is output as abnormal.

[0017] In one implementation, step S3 specifically includes:

[0018] S31, for a target ship whose behavior detection result is abnormal, obtaining positioning data of the target ship according to radar monitoring information;

[0019] S32 transmits the positioning data of the target ship to the UAV control system, and the UAV control system assigns the UAV to arrive at the corresponding positioning site to collect video image information of the target ship;

[0020] S33 receives the video image information of the target ship transmitted by the UAV.

[0021] In one implementation, step S4 specifically includes:

[0022] S41 performs abnormal behavior recognition processing based on the acquired video image information of the target ship and the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship.

[0023] In one implementation, step S41 includes:

[0024] According to the acquired video image information of the target ship, the type of the ship is identified using the trained ship classification and recognition model;

[0025] Based on the acquired ship type information of the target ship, combined with the positioning data of the target ship, it is analyzed whether the target ship is located in the warning area corresponding to the target ship type, and the secondary behavior detection result of the target ship is output.

[0026] In one implementation, step S41 includes:

[0027] Based on the acquired video image information of the target ship, the trained monitoring model is used to detect whether the ship has issued a distress sign, and the secondary behavior detection result of the target ship is output.

[0028] In a second aspect, the present invention provides an abnormal behavior detection system based on multimodal information fusion, comprising:

[0029] The first acquisition module is used to collect AIS message information of ships in the target sea area and radar monitoring information of the radar system;

[0030] The first analysis module performs an abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information to obtain a behavior detection result of the ship;

[0031] The second acquisition module is used to further acquire video image information of a target ship whose behavior detection result is abnormal according to the positioning information of the target ship;

[0032] The second analysis module performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain the secondary behavior detection result of the target ship.

[0033] The beneficial effects of the present invention are as follows: the present invention firstly judges the AIS data of the ship, and can quickly screen out the target ships with abnormal AIS data. For the target ships whose behavior cannot be detected according to the AIS data, the present invention further adopts the method based on video image data to perform behavior detection on the target ships, which can help to improve the reliability of the ship supervision system in supervising the ships in the target sea area. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0035] Figure 1 A flowchart of an exemplary embodiment of an abnormal behavior detection method based on multimodal information fusion according to the present invention;

[0036] Figure 2 This is a framework diagram of an exemplary embodiment of an abnormal behavior detection system based on multimodal information fusion of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in conjunction with the following application scenarios.

[0038] See also Figure 1 The present invention shows an abnormal behavior detection method based on multimodal information fusion, including:

[0039] S1 collects AIS message information of ships in the target sea area and radar monitoring information of radar system;

[0040] S2 performs an abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information, and obtains a behavior detection result of the ship;

[0041] S3, for a target ship whose behavior detection result is abnormal, further collects video image information of the target ship according to the positioning information of the target ship;

[0042] S4 performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship.

[0043] In the above implementation, the abnormal behavior detection system first obtains the AIS message information transmitted by the ship and the radar monitoring information transmitted by the radar system for monitoring the target sea area, and performs an abnormal behavior detection process on the ships in the target sea area according to the obtained AIS message information and radar monitoring information (for example, whether the ships detected according to the radar monitoring information have corresponding AIS message information, etc.), detects whether the AIS message information transmitted by the ship is accurate, and obtains the first behavior detection result of the ship; for the target ship with abnormal AIS message information, further obtains the positioning information of the target ship according to the radar monitoring information, and obtains the video image information of the target ship; based on the video image information of the target ship, the target ship is subjected to secondary behavior detection to obtain the behavior detection result of the target ship. In the above technical scheme, the AIS data of the ship is first judged, and the target ship with abnormal AIS data can be quickly screened out. For the target ship that cannot be detected according to the AIS data, the target ship is further detected based on the video image data, which can help improve the reliability of the ship supervision system in supervising the target sea area.

[0044] In one scenario, the abnormal behavior detection method and system based on multimodal information fusion proposed in the present invention can be applied to a ship supervision system for supervising ships in a target sea area.

[0045] In one implementation, step S1 specifically includes:

[0046] Receive AIS message information transmitted by ships in the target sea area, where the AIS message information contains ship ID and ship positioning data;

[0047] The radar monitoring information transmitted by the radar system for monitoring the target sea area is received, wherein the radar monitoring information includes the positioning information of the ships monitored in the target sea area.

[0048] In one implementation, step S2 specifically includes:

[0049] S21 performs positioning consistency judgment based on the ship positioning data in the AIS message information at the specified time and the ship positioning information in the radar monitoring information to obtain a positioning consistency judgment result;

[0050] In one scenario, the ship's positioning data is extracted from the AIS message information obtained at a certain moment, and at the same time, the positioning data is detected according to the radar monitoring information at the same moment. If so, the consistency judgment result is normal, and the ship is identified in the radar monitoring information; otherwise, when the radar monitoring information at a certain moment detects the appearance of a ship in a certain position, but the AIS message information corresponding to the position is not obtained at the same moment, the positioning consistency judgment result is judged to be abnormal, and the target ship with the abnormal consistency judgment result is marked according to the radar monitoring information.

[0051] In one scenario, when a ship appears for the first time in the radar monitoring information and the positioning consistency judgment result of the ship is normal, the process jumps to step S22 and the positioning consistency judgment of step S21 is no longer performed on the ship.

[0052] S22: acquiring first ship trajectory data according to the ship positioning data in the AIS message information within a set time period; acquiring second ship trajectory data according to the positioning data of the corresponding ship in the radar monitoring information within the set time period; performing trajectory consistency judgment according to the first ship trajectory data and the second ship trajectory data to obtain a trajectory consistency judgment result;

[0053] In one scenario, since the positioning consistency is judged by the AIS message information and radar monitoring information corresponding to the same time, it is easy to be affected by data transmission or other factors, resulting in low robustness. Therefore, after the ship is detected as normal by positioning consistency in step S21, the trajectory data of the ship is further tracked, that is, the trajectory consistency is judged based on the first ship trajectory data obtained from the AIS message information and the second ship trajectory data obtained from the radar monitoring information, which helps to improve the robustness of the primary behavior detection result.

[0054] In one implementation, trajectory consistency determination specifically includes:

[0055] The first ship track data GA is obtained according to the ship positioning data in the AIS message information. a(1),D a (2),…,D a (t),…D a (T)}, where GA represents the first ship trajectory within a set time period, D a (t) represents the positioning data corresponding to the tth moment in the first ship trajectory, where t = 1, 2, ..., T, and T represents the total number of the first ship trajectory positioning data within the time period; and the corresponding speed change data VA is obtained according to the first ship trajectory data = {V a (1),V a (2),…,V a (t),…V a (T)} and heading data θA={θ a (1),θ a (2),…,θ a (t),…θ a (T)}, where V a (t) and θ a (t) respectively represent the speed and heading corresponding to the tth moment in the first ship trajectory;

[0056] The second ship trajectory data is obtained according to the positioning data of the corresponding ship in the radar monitoring information, GL = {D l (1),D l (2),…,D l (t),…D l (T)}, where GL represents the second ship trajectory within the set time period, D l (t) represents the positioning data corresponding to the tth moment in the second ship trajectory; and obtains the corresponding speed change data VL={V l (1),V l (2),…,V l (t),…V l (T)} and heading data θL={θ l (1),θ l (2),…,θ l (t),…θ l (T)}, where V l (t) and θ l (t) respectively represent the speed and heading corresponding to the tth moment in the second ship trajectory;

[0057] Get the speed deviation between the first ship track and the second ship track respectively Distance Deviation and heading deviation

[0058] The trajectory consistency is judged based on the acquired speed deviation, distance deviation and heading deviation, and the judgment condition function used is: Where v′ represents the set speed deviation threshold, d′ represents the set distance deviation threshold, and θ′ represents the set heading deviation threshold;

[0059] If the above judgment condition function is met, the trajectory consistency judgment result is judged to be normal, otherwise, the trajectory consistency judgment result is judged to be abnormal.

[0060] S23 When the positioning consistency judgment result or the trajectory consistency judgment result of the ship is abnormal, the behavior detection result of the ship is output as abnormal; when the positioning consistency judgment result and the trajectory consistency judgment result are both normal, the behavior detection result is output as normal.

[0061] In the above implementation mode, the abnormal behavior detection system compares and performs consistency analysis based on the acquired AIS message information and the radar monitoring information monitored by the radar system, and makes a behavior judgment on the ships in the target sea area. Based on this, when there is a ship without corresponding AIS message data, or the AIS message data does not correspond to the ship positioning data monitored by the radar monitoring information (for example, the shipborne AIS equipment fails to operate normally, the shipborne AIS is manually turned off, the AIS data is inaccurate, the AIS data is tampered with, or the ship is not equipped with AIS equipment, etc.), the target ship is marked and further behavior detection processing is performed on the target ship.

[0062] When the behavior detection result is normal, the ship in the target sea area transmits AIS data to the abnormal behavior detection system normally. The abnormal behavior detection system can detect the behavior of the ship based on the acquired AIS data:

[0063] In one implementation, the AIS message information also includes ship destination path data, filing record data, speed data, heading data, etc.

[0064] Step S2 also includes:

[0065] S20 determines whether the ship has an abnormal speed (too high speed, too low speed, wandering, etc.) according to the speed data in the AIS message information; determines whether the ship has an abnormal route (illegally entering the warning area, not heading to the destination, etc.) according to the destination path data, direction data and ship positioning data in the AIS message information; determines whether the ship is a legal ship according to the record data in the AIS message information, etc.;

[0066] Step S23 also includes: when the speed of the ship is abnormal, the route is abnormal or the ship is illegal, outputting a behavior detection result of the ship as abnormal.

[0067] In one implementation, step S3 specifically includes:

[0068] S31, for a target ship whose behavior detection result is abnormal, obtaining positioning data of the target ship according to radar monitoring information;

[0069] S32 transmits the positioning data of the target ship to the UAV control system, and the UAV control system assigns the UAV to arrive at the corresponding positioning site to collect video image information of the target ship;

[0070] S33 receives the video image information of the target ship transmitted by the UAV.

[0071] Among them, the anomaly analysis system is connected to the UAV control system. The UAV control system assigns the UAV to the designated target location to collect video images for target positioning. It can further acquire video image information for the target ship whose anomaly detection result is abnormal, laying the foundation for the subsequent anomaly analysis system to conduct further abnormal behavior detection on the target ship based on the acquired video image data of the target ship.

[0072] The drone is equipped with a positioning device and an image acquisition device, which can go to a designated place to take pictures of the target ship based on the real-time positioning information of the target ship (obtained by the radar monitoring system).

[0073] In one implementation, step S4 specifically includes:

[0074] S41 performs abnormal behavior recognition processing based on the acquired video image information of the target ship and the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship.

[0075] In one implementation, step S41 includes:

[0076] According to the acquired video image information of the target ship, the type of the ship is identified using the trained ship classification and recognition model;

[0077] Based on the acquired ship type information of the target ship, combined with the positioning data of the target ship, it is analyzed whether the target ship is located in the warning area corresponding to the target ship type, and the secondary behavior detection result of the target ship is output.

[0078] In one implementation, step S41 includes:

[0079] Based on the acquired video image information of the target ship, the trained monitoring model is used to detect whether the ship has issued a distress sign, and the secondary behavior detection result of the target ship is output.

[0080] In one scenario, for a ship that sends out a distress signal due to a malfunction in the sea and the AIS system cannot operate normally, video images can be used to monitor whether a distress flag is hoisted on the ship, or whether a distress light signal is sent out at night, etc. This helps to promptly discover the ship trapped at sea through the abnormal behavior detection system and arrange rescue work in a timely manner.

[0081] In one implementation, step S4 further includes:

[0082] S40 preprocesses the acquired video image of the target ship to obtain a preprocessed video image;

[0083] S41 performs abnormal behavior recognition processing based on the preprocessed target image and the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship.

[0084] In one implementation, step S40 pre-processes the acquired video image of the target ship, specifically including:

[0085] Convert the acquired video image from the RGB color space to the LAB color space to obtain the brightness component L of the video image;

[0086] For the acquired brightness component L, brightness enhancement processing is performed on each pixel in the brightness component L, where:

[0087] When the brightness component value L(x,y) of the pixel point (x,y) is equal to the average brightness component value The absolute value of the difference is less than the set brightness interval L T When Then the following first brightness enhancement function is used to perform brightness enhancement processing on the pixel point (x, y):

[0088]

[0089] in, represents the brightness component value of the pixel (x, y) after brightness enhancement processing; α represents the set scale factor, where α∈(0,1), L′ represents the set standard brightness value, where L′∈[60,70];

[0090] When the brightness component value L(x,y) of the pixel point (x,y) is equal to the average brightness component value The absolute value of the difference is greater than or equal to the set brightness interval L T When Then the following second brightness enhancement function is used to perform brightness enhancement processing on the pixel point (x, y):

[0091]

[0092] In the formula, β represents the set proportional factor, where β∈(0,1), represents the set compensation factor, where

[0093] According to the brightness component value of each pixel after brightness enhancement processing, the brightness component after brightness enhancement processing is obtained.

[0094] Based on the brightness component after brightness enhancement processing The LAB color space is converted back to the RGB color space to obtain a video image with brightness enhanced as the preprocessed video image.

[0095] The above implementation mode aims at the situation that the sea environment is easily affected by factors such as direct sunlight and sea surface reflection, low brightness at night but water surface reflection, etc., which can easily cause uneven brightness in the collected video image of the target ship (for example, the appearance of reflective spots, or the overall brightness is high, etc.), thereby affecting the clarity of the ship on the sea surface. Therefore, the above-mentioned technical solution for brightness enhancement processing of the acquired brightness image is proposed, which can adapt to the situation of strong light reflection on the water surface during the day or at night, enhance and adjust the overall brightness level and local brightness level of the image, improve the clarity of the video image and the contrast of the target ship, and lay the foundation for further behavior detection of the target ship based on the video image.

[0096] Also, see Figure 2 , which shows an abnormal behavior detection system based on multimodal information fusion of the present invention, comprising:

[0097] The first acquisition module is used to collect AIS message information of ships in the target sea area and radar monitoring information of the radar system;

[0098] The first analysis module performs an abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information to obtain a behavior detection result of the ship;

[0099] The second acquisition module is used to further acquire video image information of a target ship whose behavior detection result is abnormal according to the positioning information of the target ship;

[0100] The second analysis module performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain the secondary behavior detection result of the target ship.

[0101] Among them, each module in the abnormal behavior detection system proposed above is also used to implement the method steps corresponding to each embodiment of the abnormal behavior detection method based on multimodal information fusion mentioned above, and the present invention will not repeat the description here.

[0102] It should be noted that each functional unit / module in each embodiment of the present invention may be integrated into one processing unit / module, or each unit / module may exist physically separately, or two or more units / modules may be integrated into one unit / module. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of software functional unit / module.

[0103] Through the description of the above implementation mode, it can be clearly understood by those skilled in the art that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the process of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a computer. Computer-readable media can include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should analyze that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. An abnormal behavior detection method based on multimodal information fusion, characterized in that: include: S1 collects AIS message information of ships in the target sea area and radar monitoring information of radar system; S2 performs an abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information, and obtains a behavior detection result of the ship, which specifically includes: S21 performs positioning consistency judgment based on the ship positioning data in the AIS message information at the specified time and the ship positioning information in the radar monitoring information to obtain a positioning consistency judgment result; S22: acquiring first ship trajectory data according to the ship positioning data in the AIS message information within a set time period; acquiring second ship trajectory data according to the positioning data of the corresponding ship in the radar monitoring information within the set time period; performing trajectory consistency judgment according to the first ship trajectory data and the second ship trajectory data to obtain a trajectory consistency judgment result; The trajectory consistency judgment specifically includes: The first ship track data GA is obtained according to the ship positioning data in the AIS message information. a (1),D a (2),…,D a (t),…D a (T)}, where GA represents the first ship trajectory within a set time period, D a (t) represents the positioning data corresponding to the tth moment in the first ship trajectory, where t = 1, 2, ..., T, and T represents the total number of the first ship trajectory positioning data within the time period; and the corresponding speed change data VA is obtained according to the first ship trajectory data = {V a (1),V a (2),…,V a (t),…V a (T)} and heading data θA={θ a (1),θ a (2),…,θ a (t),…θ a (T)}, where V a (t) and θ a (t) respectively represent the speed and heading corresponding to the tth moment in the first ship trajectory; The second ship trajectory data is obtained according to the positioning data of the corresponding ship in the radar monitoring information, GL = {D l (1),D l (2),…,D l (t),…D l (T)}, where GL represents the second ship trajectory within the set time period, D l (t) represents the positioning data corresponding to the tth moment in the second ship trajectory; and obtains the corresponding speed change data VL={V l (1),V l (2),…,V l (t),…V l (T)} and heading data θL={θ l (1),θ l (2),…,θ l (t),…θ l (T)}, where V l (t) and θ l (t) respectively represent the speed and heading corresponding to the tth moment in the second ship trajectory; Get the speed deviation between the first ship track and the second ship track respectively Distance Deviation and heading deviation The trajectory consistency is judged based on the acquired speed deviation, distance deviation and heading deviation, and the judgment condition function used is: Where v′ represents the set speed deviation threshold, d′ represents the set distance deviation threshold, and θ′ represents the set heading deviation threshold; If the above judgment condition function is met, the trajectory consistency judgment result is judged to be normal, otherwise, the trajectory consistency judgment result is judged to be abnormal; S23: when the positioning consistency judgment result or the trajectory consistency judgment result of the ship is abnormal, outputting a behavior detection result of the ship as abnormal; S3, for a target ship whose behavior detection result is abnormal, further collects video image information of the target ship according to the positioning information of the target ship; S4 performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain a secondary behavior detection result of the target ship; Wherein, step S4 comprises: S40 preprocesses the acquired video image of the target ship to obtain a preprocessed video image; S41 performs abnormal behavior recognition processing based on the preprocessed target image and the trained abnormal behavior detection model to obtain an abnormal behavior recognition result of the target ship; Wherein, step S40 pre-processes the acquired video image of the target ship, specifically including: Convert the acquired video image from the RGB color space to the LAB color space to obtain the brightness component L of the video image; For the acquired brightness component L, brightness enhancement processing is performed on each pixel in the brightness component L, where: When the brightness component value L(x,y) of the pixel point (x,y) is equal to the average brightness component value The absolute value of the difference is less than the set brightness interval L T When Then the following first brightness enhancement function is used to perform brightness enhancement processing on the pixel point (x, y): in, represents the brightness component value of the pixel (x, y) after brightness enhancement processing; α represents the set scale factor, where α∈(0,1), L′ represents the set standard brightness value, where L′∈[60,70]; When the brightness component value L(x,y) of the pixel point (x,y) is equal to the average brightness component value The absolute value of the difference is greater than or equal to the set brightness interval L T When Then the following second brightness enhancement function is used to perform brightness enhancement processing on the pixel point (x, y): In the formula, β represents the set proportional factor, where β∈(0,1), represents the set compensation factor, where According to the brightness component value of each pixel after brightness enhancement processing, the brightness component after brightness enhancement processing is obtained. Based on the brightness component after brightness enhancement processing The LAB color space is converted back to the RGB color space to obtain a video image with brightness enhanced as the preprocessed video image.

2. The abnormal behavior detection method based on multimodal information fusion according to claim 1 is characterized in that: Step S1 specifically includes: Receive AIS message information transmitted by ships in the target sea area, where the AIS message information contains ship ID and ship positioning data; The radar monitoring information transmitted by the radar system for monitoring the target sea area is received, wherein the radar monitoring information includes the positioning information of the ships monitored in the target sea area.

3. The abnormal behavior detection method based on multimodal information fusion according to claim 1 is characterized in that: Step S3 specifically includes: S31, for a target ship whose behavior detection result is abnormal, obtaining positioning data of the target ship according to radar monitoring information; S32 transmits the positioning data of the target ship to the UAV control system, and the UAV control system assigns the UAV to arrive at the corresponding positioning site to collect video image information of the target ship; S33 receives the video image information of the target ship transmitted by the UAV.

4. The abnormal behavior detection method based on multimodal information fusion according to claim 1 is characterized in that: Step S41 includes: According to the acquired video image information of the target ship, the type of the ship is identified using the trained ship classification and recognition model; Based on the acquired ship type information of the target ship, combined with the positioning data of the target ship, it is analyzed whether the target ship is located in the warning area corresponding to the target ship type, and the secondary behavior detection result of the target ship is output.

5. The abnormal behavior detection method based on multimodal information fusion according to claim 1 is characterized in that: Step S41 includes: Based on the acquired video image information of the target ship, the trained monitoring model is used to detect whether the ship has issued a distress sign, and the secondary behavior detection result of the target ship is output.

6. An abnormal behavior detection system based on multimodal information fusion, characterized in that: include: The first acquisition module is used to collect AIS message information of ships in the target sea area and radar monitoring information of the radar system; The first analysis module performs an abnormal behavior detection process on the ship based on the acquired AIS message information and radar monitoring information, and obtains a behavior detection result of the ship, which specifically includes: Performing a positioning consistency judgment based on the ship positioning data in the AIS message information at a specified time and the ship positioning information in the radar monitoring information to obtain a positioning consistency judgment result; Acquire first ship trajectory data according to ship positioning data in AIS message information within a set time period; acquire second ship trajectory data according to positioning data of the corresponding ship in radar monitoring information within a set time period; perform trajectory consistency judgment based on the first ship trajectory data and the second ship trajectory data to obtain a trajectory consistency judgment result; The trajectory consistency judgment specifically includes: The first ship track data GA is obtained according to the ship positioning data in the AIS message information. a (1),D a (2),…,D a (t),…D a (T)}, where GA represents the first ship trajectory within a set time period, D a (t) represents the positioning data corresponding to the tth moment in the first ship trajectory, where t = 1, 2, ..., T, and T represents the total number of the first ship trajectory positioning data within the time period; and the corresponding speed change data VA is obtained according to the first ship trajectory data = {V a (1),V a (2),…,V a (t),…V a (T)} and heading data θA={θ a (1),θ a (2),…,θ a (t),…θ a (T)}, where V a (t) and θ a (t) respectively represent the speed and heading corresponding to the tth moment in the first ship trajectory; The second ship trajectory data is obtained according to the positioning data of the corresponding ship in the radar monitoring information, GL = {D l (1),D l (2),…,D l (t),…D l (T)}, where GL represents the second ship trajectory within the set time period, D l (t) represents the positioning data corresponding to the tth moment in the second ship trajectory; and obtains the corresponding speed change data VL={V l (1),V l (2),…,V l (t),…V l (T)} and heading data θL={θ l (1),θ l (2),…,θ l (t),…θ l (T)}, where V l (t) and θ l (t) respectively represent the speed and heading corresponding to the tth moment in the second ship trajectory; Get the speed deviation between the first ship track and the second ship track respectively Distance Deviation and heading deviation The trajectory consistency is judged based on the acquired speed deviation, distance deviation and heading deviation, and the judgment condition function used is: Where v′ represents the set speed deviation threshold, d′ represents the set distance deviation threshold, and θ′ represents the set heading deviation threshold; If the above judgment condition function is met, the trajectory consistency judgment result is judged to be normal, otherwise, the trajectory consistency judgment result is judged to be abnormal; When the positioning consistency judgment result or the trajectory consistency judgment result of the ship is abnormal, the primary behavior detection result of the ship is output as abnormal; The second acquisition module is used to further acquire video image information of a target ship whose behavior detection result is abnormal according to the positioning information of the target ship; The second analysis module performs secondary behavior detection processing according to the acquired video image information of the target ship to obtain the secondary behavior detection result of the target ship, including: Preprocessing the acquired video image of the target ship to obtain a preprocessed video image; According to the preprocessed target image, abnormal behavior recognition processing is performed based on the trained abnormal behavior detection model to obtain the abnormal behavior recognition result of the target ship; The acquired video image of the target ship is preprocessed, specifically including: Convert the acquired video image from the RGB color space to the LAB color space to obtain the brightness component L of the video image; For the acquired brightness component L, brightness enhancement processing is performed on each pixel in the brightness component L, where: When the brightness component value L(x,y) of the pixel point (x,y) is equal to the average brightness component value The absolute value of the difference is less than the set brightness interval L T When Then the following first brightness enhancement function is used to perform brightness enhancement processing on the pixel point (x, y): in, represents the brightness component value of the pixel (x, y) after brightness enhancement processing; α represents the set scale factor, where α∈(0,1), L′ represents the set standard brightness value, where L′∈[60,70]; When the brightness component value L(x,y) of the pixel point (x,y) is equal to the average brightness component value The absolute value of the difference is greater than or equal to the set brightness interval L T When Then the following second brightness enhancement function is used to perform brightness enhancement processing on the pixel point (x, y): In the formula, β represents the set proportional factor, where β∈(0,1), represents the set compensation factor, where According to the brightness component value of each pixel after brightness enhancement processing, the brightness component after brightness enhancement processing is obtained. Based on the brightness component after brightness enhancement processing The LAB color space is converted back to the RGB color space to obtain a video image with brightness enhanced as the preprocessed video image.

Images