Ship intelligent detection method and system based on artificial intelligence

Abstract

The present invention discloses a method and system for intelligent detection of ships based on artificial intelligence, and the method includes: obtaining real-time video / image data of the target sea area; recognizing and calculating the pixel position of the ship based on the real-time video / image data; obtaining AIS signals, calculating the AIS pixel coordinates of the ship, and judging whether the AIS pixel coordinates match the pixel position, if so, determining the identity of the ship based on the AIS signal; if not, judging that the ship turns off AIS identity recognition, performing focus control of the real-time video / image data acquisition device according to the pixel position, obtaining new real-time video / image data, and performing ship recognition on the real-time video / image data based on a preset recognition model; wherein the recognition model is set to be trained based on an incremental sample database set, and the incremental sample database set is established according to a group update mechanism. The present application can be used to improve the recognition effect of ships without AIS equipment.

Description

Technical Field

[0001] The present application relates to the field of marine artificial intelligence technology, and in particular to an artificial intelligence-based ship intelligent detection method and system. Background Art

[0002] For ocean data, how to automatically build a deep learning network and explore the automatic identification and retrieval system based on artificial intelligence + ocean targets is the key to improving the transformation and service of ocean big data and knowledge, and is the key to improving the application efficiency of ocean big data.

[0003] The patent with publication number CN111523465A is a ship identification system based on camera calibration and deep learning algorithm, which includes the following steps:

[0004] Step 1, establish a ship detector;

[0005] Step 2, ship detection: Use the ship detector established in step 1 to perform ship detection on each frame image in the real-time video stream to determine whether the current video frame contains a ship and detect the pixel position of the ship in the video frame image.

[0006] Step 3, ship tracking: Design a KCF multi-target tracker to track the ship targets detected in step 2.

[0007] Step 4, obtain the AIS pixel coordinates of the ship: calibrate the internal and external parameters of the camera. Access the AIS signal, obtain the real-time longitude and latitude and identity information of the ship, and convert the longitude and latitude coordinates of the ship into the AIS pixel coordinates of the ship in the video frame image through the conversion relationship of the four coordinate systems and the combination of the internal and external parameters of the camera.

[0008] Step 5, matching the ship position: matching the ship AIS pixel coordinates obtained in step 4 with the pixel position of the ship detected in step 3 in the video frame image.

[0009] Step 6, ship identification: When the match in step 5 is successful, the ship identity information obtained in step 4 is directly used. If the match is not successful, the tracked ship is considered to be a non-AIS ship, and the deep learning method is used to identify the ship identity.

[0010] With respect to the above-mentioned related technologies, the inventors believe that the following defects exist: for ships without AIS equipment, the initiative of the basic database for ship identification is poor, and the identification is feature retrieval. If there is no image of the ship, it cannot be identified and the accuracy needs to be improved. Summary of the invention

[0011] In order to improve the recognition effect of ships without AIS equipment, the present application provides a ship intelligent detection method and system based on artificial intelligence.

[0012] In the first aspect, the present application provides a ship intelligent detection method based on artificial intelligence, which adopts the following technical solutions:

[0013] An artificial intelligence-based ship intelligent detection method, comprising:

[0014] Obtain real-time video / image data of the target sea area;

[0015] Identify and calculate the pixel position of the ship based on real-time video / image data;

[0016] Get the AIS signal, calculate the AIS pixel coordinates of the ship, and determine whether the AIS pixel coordinates match the pixel position.

[0017] If yes, determine the vessel’s identity based on the AIS signal;

[0018] If not, it is determined that the ship has turned off AIS identification, and the focus control of the real-time video / image data acquisition device is performed according to the pixel position to obtain new real-time video / image data, and the real-time video / image data is used to identify the ship based on the preset recognition model;

[0019] Wherein, the recognition model is set to be trained based on an incremental sample database set, and the incremental sample database set is established according to a group update mechanism.

[0020] Optionally, the establishing of a group update mechanism includes:

[0021] Obtain video / image data uploaded by users at a specified network address and record them as candidate samples; users include fishermen, sea anglers, port investigators, and maritime professionals;

[0022] Perform image processing on the candidate samples to obtain the standard candidate sample image;

[0023] Compare the candidate sample pictures with the sample picture database to identify and determine whether they meet the new sample picture standards. If so, add the candidate sample pictures to the sample picture database and execute the graded reward logic; if not, send a sample duplicate prompt message to the specified network address.

[0024] Optionally, the hierarchical reward logic includes:

[0025] Determine whether the user is a full-time authorized user. If not, execute the preset reward standard according to any one or more of whether the sample is a new ship type, whether the sample is a sample with a different posture, and whether the sample is a new sample with a different sight distance; and

[0026] The user who obtains the binding standard sample map uploads the sea condition information to determine whether it is a sample under a new environment. If so, another preset reward standard is implemented.

[0027] Optionally, the sea condition information uploaded by the user includes location, time and weather information, and correction and evaluation are performed on the sea condition information uploaded by the user, and the correction and evaluation includes:

[0028] Obtain online weather information based on the location and time in the sea condition information uploaded by the user;

[0029] Compare the meteorological information on the Internet with the meteorological information in the sea condition information uploaded by the user. If the difference meets the threshold, mark the sample map according to the location, time and meteorological information.

[0030] Optionally, the incremental sample database set is configured as follows: databases are established respectively according to the types of acquisition equipment of the standard sample selection diagram.

[0031] Optionally, the ship identification includes ship type identification and ship hull number character recognition; wherein, the ship hull number character recognition includes number detection and correction based on the WPOD.NET network, and the ship hull number is identified based on the OCR.NET network.

[0032] Optionally, the ship identification includes:

[0033] Determine target pixel parameters based on pixel position and identified ship type;

[0034] Get the target area image in real-time video / image data based on the target pixel parameters and AForge.Net framework;

[0035] The target area image is used for ship number character recognition.

[0036] In the second aspect, the present application provides an artificial intelligence-based ship intelligent detection system, which adopts the following technical solutions:

[0037] A ship intelligent detection system based on artificial intelligence comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the above-mentioned ship intelligent detection method based on artificial intelligence.

[0038] In summary, the present application includes at least one of the following beneficial technical effects: the identity of a ship at sea can be identified based on video / image data; because the recognition model is trained based on an incremental sample database set, the recognition accuracy is higher; at the same time, because the image area is determined based on the pixel position and the ship's hull number is identified, the recognition accuracy and recognition information of ships whose AIS pixel positions are not successfully matched, that is, ships whose AIS equipment is turned off, are higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic flow chart of the method of the present application;

[0040] Figure 2 It is a schematic diagram of the ship's hull number recognition process of this application. DETAILED DESCRIPTION

[0041] The following is combined with Figure 1-2 This application is described in further detail.

[0042] The embodiment of the present application discloses an artificial intelligence-based ship intelligent detection method. In terms of autonomous perception of the marine environment and intelligent detection of targets, the method mainly uses photoelectric image data (fixed-point camera data and mobile camera data), combines target search radar and AIS data, actively detects marine ship targets through radar, discovers and realizes dangerous targets appearing in the marine ship surface area, and monitors them; after the target is detected through photoelectric image data, after the invading ship target turns off the AIS and other identification devices, the ship is locked by adjusting the angle of the photoelectric platform, and the hull number characters of the target ship are focused and identified. In the detected target area, the algorithm proposes the area where the ship hull number characters are located, and then character recognition is performed on the detected character area to determine the specific information of the ship hull number characters.

[0043] Reference Figure 1 , the ship intelligent detection methods based on artificial intelligence include:

[0044] S101, obtaining real-time video / image data of the target sea area; such as collecting and uploading by the fixed camera and the mobile camera mentioned above to obtain.

[0045] S102, identifying and calculating the pixel position of the ship based on real-time video / image data;

[0046] S103, obtaining the AIS signal, calculating the AIS pixel coordinates of the ship, and determining whether the AIS pixel coordinates match the pixel position.

[0047] If yes, determine the vessel’s identity based on the AIS signal;

[0048] If not, it is determined that the ship has turned off AIS identification, and the focus control of the real-time video / image data acquisition device is performed according to the pixel position to obtain new real-time video / image data, and the real-time video / image data is used to identify the ship based on the preset recognition model.

[0049] Regarding the contents of the "yes" case in the above S102 and S103, this application is also implemented using technology based on the YOLO network, and is similar to the disclosed contents of the patent document described in the background, so it will not be described in detail, and only the contents of the "no" case will be expanded.

[0050] It can be understood that, in this embodiment, the video data needs to be extracted into image data for subsequent steps to be executed, so the subsequent description generally refers to image data.

[0051] Regarding ship identification in the above scenario of turning off AIS equipment, it includes ship type identification and ship hull number character recognition.

[0052] Reference Figure 2 , which is the general process, including: 1) input image - 2) ship detection and segmentation - 3) ship classification - 4) number detection and correction based on WPOD.NET network - 5) recognition of ship hull number based on OCR.NET network.

[0053] Among them, steps 1), 2) and 3) are implemented by using the above-mentioned YOLO network technology, and are disclosed in the literature and will not be repeated here.

[0054] The ship number technology based on WPOD.NET network and OCR.NET network refers to license plate recognition, which is an existing technology; or refers to another recognition method based on convolutional neural network, such as: using pytorch framework, importing related libraries (loading pictures); loading data sets from the specified path and converting them into Tensor format; defining and packaging data, with specifications such as 16, 32, 64, etc.; unifying picture pixels and latitudes; defining convolutional network layers, defining models, training and testing; saving models; importing the image to be recognized into the model to obtain results.

[0055] Regarding the above focus control, the reason for setting this step is that the image data used for AIS matching judgment is relatively broad, and may store multiple ships at the same time, and the clarity is slightly poor. If it is used directly for ship identification, only a rough identification of ship types can be done. Specifically: determine the target ship based on the unmatched AIS pixel coordinates and pixel positions, reversely calculate based on the pixel positions, determine the theoretical longitude and latitude to evaluate the ship distance; find the preset focus-distance relationship table, determine the focus parameters, and send them to the corresponding camera equipment for control.

[0056] Based on the above, a relatively clearer image can be obtained for ship identification.

[0057] It can be understood that the basis for ship number recognition is a target map area, and the marine vessel in the application scenario of this embodiment is not like driving through an electronic access control, and has to drive to a designated area to ensure map area sampling. For this purpose, this method includes:

[0058] S201, determining target pixel parameters according to pixel positions and recognized ship types.

[0059] It should be noted that the pixel position at this time is only used to determine the position of the ship relative to the image; the ship type is used to provide contour reference benchmark data, which is used to delineate the theoretical ship area in the image in conjunction with the error allowance, and even determine the position of the ship's hull number in the image based on the preset actual hull number relative to the ship's position data.

[0060] S202, acquiring a target area image in real-time video / image data based on target pixel parameters and AForge.Net framework.

[0061] Specifically, it includes: binarizing the grayscale image (image decolorization, after initial processing) to obtain a binary image; using the BlobCounter class of AForge.Net to filter out spots that are not within the target pixel parameters, that is, extracting the target area image whose width and height meet the target pixel parameters from the image.

[0062] S203: The target area image is used for ship hull number character recognition.

[0063] Based on the above content, it can be better ensured that the identification of ship hull numbers at sea can be as accurate as license plate recognition.

[0064] Regarding the recognition model, in addition to the above-mentioned ship number recognition, there is the ship type recognition part. For ship type recognition, it also includes: setting it to be trained based on an incremental sample database set, and the incremental sample database set is established according to a group update mechanism.

[0065] Among them, establishing a group update mechanism includes:

[0066] Obtain video / image data uploaded by users at a specified network address and record them as candidate samples; users include fishermen, sea anglers, port investigators, and maritime professionals;

[0067] Perform image processing on the candidate samples (extract frames, identify images with ships, and remove the initial processing of images) to obtain the standard candidate sample images;

[0068] Compare the candidate sample pictures with the sample picture database to identify and determine whether they meet the new sample picture standards. If so, add the candidate sample pictures to the sample picture database and execute the graded reward logic; if not, send a sample duplicate prompt message to the specified network address.

[0069] It is understandable that the acquisition of the above-mentioned network address is to provide an online public webpage / platform by the official to mobilize and utilize maritime-related personnel resources. It is no longer limited to full-time personnel to periodically update the database. After all, sample maps under different scenarios require various specified environments, etc. It is relatively time-consuming and labor-intensive to rely entirely on full-time personnel, and there is a lag.

[0070] The above new sample image standards include:

[0071] 1) Evaluate the image quality, i.e., calculate the clarity, and determine whether it meets the clarity standard, such as the quality evaluation method disclosed in CN108109147A. If it meets the standard, it is a new sample image;

[0072] 2) Identify whether the ship type is a known ship type. If not, it is a new sample image.

[0073] The above-mentioned hierarchical reward logic includes:

[0074] Determine whether the user is a full-time authorized user (identify the characteristic tag bound to the user identity). If not, proceed to the next step;

[0075] Determine whether it is a new ship type. If yes, execute the preset first-level reward standard; if no, execute the next step;

[0076] Determine whether it is a sample of different posture (different observation angle), if so, execute another preset reward standard; if not, execute the next step;

[0077] Determine whether it is a new sample with a different viewing distance. If yes, execute another preset reward standard; if no, execute the next step;

[0078] The user who obtains the bound standard sample map uploads the sea condition information to determine whether it is a sample under a new environment. If so, another preset reward standard is executed; if not, the process ends.

[0079] It is understandable that, except for the last one, the others are optional and are determined manually by the government; at the same time, the various reward standards are also determined by the government based on its own financial resources. The above settings are mainly to guide maritime-related personnel to supplement the samples of the database and improve the update rate of the database.

[0080] The sea condition information uploaded by the user includes the location, time and weather information. The method also includes performing correction and evaluation on the sea condition information uploaded by the user, and the correction and evaluation includes:

[0081] Obtain online weather information based on the location and time in the sea condition information uploaded by the user;

[0082] Compare the meteorological information on the Internet with the meteorological information in the sea condition information uploaded by the user. If the difference meets the threshold, mark the sample map according to the location, time and meteorological information.

[0083] The above correction, on the one hand, is used to verify the authenticity and accuracy of the uploaded data to reduce losses; on the other hand, through marking, the information of the standard sample drawing is more comprehensive; subsequently, in the ship type recognition model training and recognition stages, it can be carried out according to label classification to improve the recognition accuracy. After all, different lighting, weather, etc. will cause interference.

[0084] Furthermore, the incremental sample database set is set as follows: databases are established respectively according to the types of acquisition equipment of the standard sample selection diagram.

[0085] The reason for this setting is that different acquisition devices sample different images. The most intuitive ones are ordinary photoelectric images and remote sensing satellite images, which are quite different. Through database classification and classification model training, the accuracy of recognition can be improved.

[0086] For the above-mentioned ship type recognition, data enhancement is generally used to improve the accuracy of marine big data target detection. However, the previous data enhancement was manually designed. This method can also use a population-based training algorithm to enhance the policy scheduling, which can also be adjusted based on the current training epoch. This method is the opposite of the fixed enhancement strategy, which uses the same transformation for different epochs.

[0087] Cluster training strategy:

[0088] Strategy 1: Model parallelism. Model cluster training refers to deploying the model to run on multiple devices.

[0089] Strategy 2: Data parallelism. Place the same model on multiple devices, and use different training samples to train the model on each device.

[0090] Cluster training architecture:

[0091] In the PS architecture, the nodes in the cluster are divided into two categories: PS and worker. The PS stores the parameters of the model, while the worker is responsible for calculating the gradient of the parameters. In each iteration, the worker obtains the parameters from the PS and returns the calculated gradient to the PS. The PS aggregates the gradients returned from the worker, updates the parameters, and broadcasts the new parameters to the workers.

[0092] The embodiment of the present application also discloses an artificial intelligence-based ship intelligent detection system.

[0093] The ship intelligent detection system based on artificial intelligence includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned ship intelligent detection method based on artificial intelligence.

[0094] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A ship intelligent detection method based on artificial intelligence, characterized in that: include: Obtain real-time video / image data of the target sea area; Identify and calculate the pixel position of the ship based on real-time video / image data; Get the AIS signal, calculate the AIS pixel coordinates of the ship, and determine whether the AIS pixel coordinates match the pixel position. If yes, determine the vessel’s identity based on the AIS signal; If not, it is determined that the ship has turned off AIS identification, and the focus control of the real-time video / image data acquisition device is performed according to the pixel position to obtain new real-time video / image data, and the real-time video / image data is used to identify the ship based on the preset recognition model; Wherein, the recognition model is set to be trained based on an incremental sample database set, and the incremental sample database set is established according to a population update mechanism; Said ship identification includes ship type identification and ship number character recognition; wherein, the ship number character recognition includes number detection and correction based on WPOD.NET network, and the ship number is identified based on OCR.NET network; Said ship identification comprises: Determine target pixel parameters based on pixel position and identified ship type; Get the target area image in real-time video / image data based on the target pixel parameters and AForge.Net framework; The target area image is used for ship number character recognition.

2. The ship intelligent detection method based on artificial intelligence according to claim 1 is characterized in that: The establishment of a group update mechanism comprises: Obtain video / image data uploaded by users at a specified network address and record them as candidate samples; users include fishermen, sea anglers, port investigators, and maritime professionals; Perform image processing on the candidate samples to obtain the standard candidate sample image; Compare the candidate sample pictures with the sample picture database to identify and determine whether they meet the new sample picture standards. If so, add the candidate sample pictures to the sample picture database and execute the graded reward logic; if not, send a sample duplicate prompt message to the specified network address.

3. The ship intelligent detection method based on artificial intelligence according to claim 2 is characterized in that: The hierarchical reward logic includes: Determine whether the user is a full-time authorized user. If not, execute the preset reward standard according to any one or more of whether the sample is a new ship type, whether the sample is a sample with a different posture, and whether the sample is a new sample with a different sight distance; and The user who obtains the binding standard sample map uploads the sea condition information to determine whether it is a sample under a new environment. If so, another preset reward standard is implemented.

4. The ship intelligent detection method based on artificial intelligence according to claim 3 is characterized in that: The sea condition information uploaded by the user includes the location, time and weather information, and the sea condition information uploaded by the user is corrected and evaluated, and the correction and evaluation includes: Obtain online weather information based on the location and time in the sea condition information uploaded by the user; Compare the meteorological information on the Internet with the meteorological information in the sea condition information uploaded by the user. If the difference meets the threshold, mark the sample map according to the location, time and meteorological information.

5. The ship intelligent detection method based on artificial intelligence according to claim 1 is characterized in that: The incremental sample database set is set as follows: databases are established respectively according to the types of acquisition equipment of the standard sample selection diagram.

6. An artificial intelligence-based ship intelligent detection system, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes any one of the artificial intelligence-based ship intelligent detection methods as claimed in claims 1 to 5.

Images