Method, system, equipment, product and medium for generating multi-sea state detection data set

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a method, system, device, product and medium for generating a multi-sea condition detection data set, including deploying a drone and a camera in a target sea area; the camera tracks a passing ship in the target sea area, and collects and obtains the first ship automatic identification system information of the passing ship, selects a calibration key point set by an approximate grid selection method, and completes the calibration of the camera by the calibration key point set and the first ship automatic identification system information; the camera monitors the target sea area, and a monitoring module monitors and obtains the second ship automatic identification system information, thereby obtaining a multi-modal ship sample; the multi-modal ship sample and the sea condition description text are input into a text image generation model to obtain an initial multi-sea condition ship data set; the initial multi-sea condition ship data set is tested to obtain a target multi-sea condition ship data set. The present invention can effectively and automatically generate a multi-sea condition ship data set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, device, product and medium for generating a multi-sea condition detection data set. Background Art

[0002] The multi-sea condition detection dataset can provide the necessary data guarantee for the parameter tuning of the deep learning model for surface target detection, thereby improving the recognition accuracy of the deep learning model for surface target detection and providing support for the autonomous perception of the surface situation of marine equipment. At present, the deep learning model for surface target detection still has the problem of low recognition accuracy in the actual surface environment scenario. The bottleneck is that the environment at sea is relatively complex and there is a lack of surface target data in the real sea condition scenario. Especially in high sea conditions, it is difficult to deploy drones or other visible light cameras to obtain high-quality surface target data. Therefore, it is difficult for the current multi-sea condition detection dataset to provide sufficient high-quality surface target data under high sea conditions for the deep learning model for surface target detection, resulting in a low recognition accuracy of the deep learning model for surface targets under high sea conditions. Moreover, the existing multi-sea condition detection dataset is more dependent on actual image collection, and it is difficult to automatically generate a multi-sea condition detection dataset covering a variety of conditions and sea conditions. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a method, system, device, product and medium for generating a multi-sea state detection data set, which realizes the automatic generation of a multi-sea state detection data set, thereby improving the recognition accuracy of a deep learning model for surface target detection on a surface target.

[0004] The present invention provides a method for generating a multi-sea condition detection data set, comprising:

[0005] S1: Determine a target sea area, select a camera and a drone, and deploy the drone and the camera in the target sea area, wherein the camera is provided with a monitoring module for obtaining information of an automatic identification system of a ship;

[0006] S2: the camera tracks the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set from the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0007] S3: The camera monitors the target sea area, and the monitoring module monitors and obtains the second ship automatic identification system information. When a sample ship appears in the target sea area, the camera and the drone obtain a target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0008] S4: obtaining a sea condition description text and a text image generation model, inputting the multimodal ship sample and the sea condition description text into the text image generation model, and obtaining an initial multi-sea condition ship data set;

[0009] S5: Checking the initial multi-sea-state ship dataset to obtain a target multi-sea-state ship dataset.

[0010] According to the method for generating a multi-sea condition detection data set provided by the present invention, step S1 further comprises:

[0011] S11: Acquire sea area ship density information and sea area cleanliness information, and determine the target sea area according to the sea area ship density information and the sea area cleanliness information;

[0012] S12: Select the camera and the drone, determine the camera deployment position and the drone deployment position in the target sea area, and deploy the camera and the drone in the target sea area in sequence according to the camera deployment position and the drone deployment position.

[0013] According to the method for generating a multi-sea condition detection data set provided by the present invention, step S2 further comprises:

[0014] S21: When the passing ship appears in the camera, the passing ship is marked to obtain a marking tag, and the camera tracks the passing ship in the target sea area through the marking tag to obtain the calibration image; at the same time, the monitoring module collects and obtains the first ship automatic identification system information of the passing ship;

[0015] S22: extracting the ship track and the picture center point track from the calibration image, and selecting a calibration key point set according to the ship track and the picture center point track by using the approximate grid selection method;

[0016] S23: extracting world coordinate sampling points from the first ship automatic identification system information, establishing a mapping relationship between the world coordinate sampling points and the calibration key point set, and constructing a coordinate transformation matrix according to the mapping relationship between the world coordinate sampling points and the calibration key point set;

[0017] S24: Establishing a camera coordinate system, obtaining the camera coordinate position, and completing the calibration of the camera in the camera coordinate system through the coordinate conversion matrix and the camera coordinate position.

[0018] According to the method for generating a multi-sea condition detection data set provided by the present invention, step S22 further includes:

[0019] S221: The camera extracts the calibration image to obtain the ship track and the track of the center point of the picture, and uses the ship track and the track of the center point of the picture as the calibration track;

[0020] S222: Determine a grid size, generate a plurality of initial grids by using the grid size and a grid generation algorithm, extract grid intersection points in each of the initial grids as seed clustering points, and cluster the calibration tracks according to the seed clustering points to obtain a clustering point set; filter the clustering point set to obtain a key point set, calculate the average value of the distances between the key points in the key point set and the adjacent grid intersection points, and obtain the average distance of the key points of each of the initial grids;

[0021] S223: Selecting the initial grid with the lowest average distance of the key points as the target grid, and using the key point set in the target grid as the calibration key point set.

[0022] According to the method for generating a multi-sea condition detection data set provided by the present invention, step S3 further comprises:

[0023] S31: the camera monitors the target sea area and determines whether the sample ship appears in the target sea area, and the monitoring module obtains the second ship automatic identification system information through monitoring;

[0024] S32: when the camera determines that the sample ship appears in the target sea area, the drone photographs the sample ship to obtain an overhead image of the sample ship, the camera photographs the sample ship to obtain an overhead image of the sample ship, and the overhead image of the sample ship is merged with the overhead image of the sample ship to obtain a multi-angle image of the target;

[0025] S33: extracting the ship position information of the sample ship in the target multi-angle image, extracting the ship position notification information in the second ship automatic identification system information, matching the ship position information with the ship position notification information, and obtaining the target second ship automatic identification system information matching the sample ship;

[0026] S34: Fusing the target second automatic ship identification system information with the target multi-angle image and removing the background to obtain the multi-modal ship sample.

[0027] According to the method for generating a multi-sea condition detection data set provided by the present invention, step S5 further comprises:

[0028] S51: Obtain a convolutional neural network model, train the convolutional neural network model using a ship data set to obtain a ship inspection model; train the convolutional neural network model using a sea condition data set to obtain a sea condition inspection model;

[0029] S52: Verify the initial multi-sea-condition ship data set by using the ship verification model and the sea-condition verification model to obtain the target multi-sea-condition ship data set.

[0030] The present invention also provides a system for generating a multi-sea condition detection data set, comprising:

[0031] Camera deployment module: used to determine the target sea area, select cameras and drones, and deploy the drones and cameras in the target sea area, wherein the cameras are provided with a monitoring module for obtaining ship automatic identification system information;

[0032] A camera calibration module: used for the camera to track the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set in the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0033] Multimodal ship sample module: used for the camera to monitor the target sea area, the monitoring module monitors and obtains the second ship automatic identification system information, when a sample ship appears in the target sea area, the camera and the drone obtain the target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0034] Initial multi-sea-state ship dataset module: used to obtain sea-state description text and text-image generation model, input the multi-modal ship sample and the sea-state description text into the text-image generation model, and obtain the initial multi-sea-state ship dataset;

[0035] Target multi-sea-state ship data set module: used to test the initial multi-sea-state ship data set to obtain the target multi-sea-state ship data set.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the above-mentioned methods for generating a multi-sea condition detection data set are implemented.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for generating a multi-sea condition detection data set.

[0038] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of any of the above-mentioned methods for generating a multi-sea condition detection data set.

[0039] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0040] The method, system, device, product and medium for generating a multi-sea condition detection data set provided by the present invention realize the labeling of basic information of sample ships in a multi-sea condition detection data set by fusing the information of a ship automatic identification system with the visible light image information; and automatically generate a multi-sea condition ship data set under various sea conditions through text image generation technology, thereby effectively reducing the cost and difficulty of obtaining image training samples under high sea conditions, and providing an economical and convenient means for obtaining a multi-sea condition ship data set; finally, by introducing a ship inspection model and a sea condition inspection model, the generated initial multi-sea condition ship data set is verified from the two perspectives of ship inspection and sea condition inspection, thereby improving data availability.

[0041] In summary, the present invention provides training samples for parameter optimization of the deep learning model for surface target detection on the one hand, and reduces the cost of obtaining ship datasets in multiple sea conditions on the other hand. It is an economical and effective means to obtain ship datasets in multiple sea conditions.

[0042] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a flow chart of the method for generating a multi-sea condition detection data set provided by the present invention.

[0045] Figure 2 It is a flow chart of an approximate grid selection method of a method for generating a multi-sea condition detection data set provided by the present invention.

[0046] Figure 3 It is a structural schematic diagram of a device for generating a multi-sea condition detection data set provided by the present invention.

[0047] Figure 4 It is a structural schematic diagram of a multi-sea condition detection data set generating device provided by the present invention.

[0048] Reference numerals:

[0049] 100, camera arrangement module; 200, camera calibration module; 300, multimodal ship sample module; 400, initial multi-sea state ship dataset module; 500, target multi-sea state ship dataset module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0051] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0052] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0053] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0054] Combine the following Figures 1 to 4 Describing embodiments of the present invention:

[0055] Figure 1 A flow chart of the method for generating a multi-sea condition detection dataset provided by the present invention, wherein S1 is to deploy drones and cameras; S2 is to calibrate the camera through a passing ship; S3 is to photograph sample ships to obtain multimodal ship samples; S4 is to generate an initial multi-sea condition ship dataset through a text image generation model; and S5 is to verify and obtain a target multi-sea condition ship dataset.

[0056] The present invention provides a method for generating a multi-sea condition detection data set, comprising:

[0057] S1: Determine a target sea area, select a camera and a drone, and deploy the drone and the camera in the target sea area, wherein the camera is provided with a monitoring module for obtaining information of an automatic identification system of a ship;

[0058] Furthermore, the purpose of this stage is to deploy the camera and the drone at a suitable deployment position in the target sea area, so as to provide an environment for the acquisition process of multimodal ship samples; wherein step S1 further includes:

[0059] S11: Acquire sea area ship density information and sea area cleanliness information, and determine the target sea area according to the sea area ship density information and the sea area cleanliness information;

[0060] S12: Select the camera and the drone, determine the camera deployment position and the drone deployment position in the target sea area, and deploy the camera and the drone in the target sea area in sequence according to the camera deployment position and the drone deployment position.

[0061] With respect to the above steps, the specific implementation methods in this embodiment are as follows:

[0062] First of all, since this application has certain requirements for the cleanliness and ship density of the target sea area: garbage and floating objects in a too dirty sea area will interfere with the generation of multimodal ship samples, too high a ship density will cause oversaturation of cameras and drones, and too low a ship density will result in too few multimodal ship samples generated; therefore, it is necessary to obtain the sea area ship density information and sea area cleanliness information, and select a sea area with moderate ship density and relatively clean as the target sea area based on the sea area ship density information and sea area cleanliness information.

[0063] Then, suitable drones and cameras are selected to monitor the target sea area, wherein the camera can be a fixed camera, a pan / tilt camera, a panoramic camera, etc., and the drone can be a rotary drone, a wire-controlled drone, etc. Here, the camera is equipped with a monitoring module that can obtain the ship automatic identification system information sent by the ship. Then, the camera deployment position and the drone deployment position are determined according to the needs of the camera and the drone and the distribution of the ship's route, and the camera and the drone are deployed in the target sea area respectively according to the camera deployment position and the drone deployment position.

[0064] S2: the camera tracks the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set from the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0065] Furthermore, the purpose of this stage is to track the track of the passing ship, and select the calibration key point set for camera calibration by the approximate grid selection method, and complete the camera calibration by combining the calibration key point set and the first ship automatic identification system information, so as to correspond the image data containing the ship collected by the camera to the ship in the real environment; wherein, step S2 further includes:

[0066] S21: When the passing ship appears in the camera, the passing ship is marked to obtain a marking tag, and the camera tracks the passing ship in the target sea area through the marking tag to obtain the calibration image; at the same time, the monitoring module collects and obtains the first ship automatic identification system information of the passing ship;

[0067] S22: extracting the ship track and the picture center point track from the calibration image, and selecting a calibration key point set according to the ship track and the picture center point track by using the approximate grid selection method;

[0068] S23: extracting world coordinate sampling points from the first ship automatic identification system information, establishing a mapping relationship between the world coordinate sampling points and the calibration key point set, and constructing a coordinate transformation matrix according to the mapping relationship between the world coordinate sampling points and the calibration key point set;

[0069] S24: Establishing a camera coordinate system, obtaining the camera coordinate position, and completing the calibration of the camera in the camera coordinate system through the coordinate conversion matrix and the camera coordinate position.

[0070] Step S22 also includes:

[0071] S221: The camera extracts the calibration image to obtain the ship track and the track of the center point of the picture, and uses the ship track and the track of the center point of the picture as the calibration track;

[0072] S222: Determine a grid size, generate a plurality of initial grids by using the grid size and a grid generation algorithm, extract grid intersection points in each of the initial grids as seed clustering points, and cluster the calibration tracks according to the seed clustering points to obtain a clustering point set; filter the clustering point set to obtain a key point set, calculate the average value of the distances between the key points in the key point set and the adjacent grid intersection points, and obtain the average distance of the key points of each of the initial grids;

[0073] S223: Selecting the initial grid with the lowest average distance of the key points as the target grid, and using the key point set in the target grid as the calibration key point set.

[0074] With respect to the above steps, the specific implementation methods in this embodiment are as follows:

[0075] Here, when the camera is arranged, the camera calibration work begins. The calibration work needs to use the passing ship as a reference, and the first ship that appears in the camera screen is taken as the passing ship. When the passing ship appears in the camera, the passing ship is marked to obtain a marking label. In this embodiment, the passing ship can be captured and marked manually. Then the camera can track the passing ship in the target sea area according to the marking label to obtain a calibration image including the ship track.

[0076] In addition to the ship track, since the image captured by the camera will move with the movement of the passing ship, the calibration image also needs to include the track of the center point of the image, that is, the trajectory of the center point of the image when the image moves. At the same time, the monitoring module of the camera will collect the ship automatic identification system information of the target sea area, that is, the information contained in the ship automatic identification system, and obtain the first ship automatic identification system information of the passing ship through manual screening and matching. In addition, when the camera is a pan-tilt camera, it is necessary to first rotate the pan-tilt camera to collect a panoramic view of the surrounding environment, and then determine the position of the passing ship and the image captured by the camera in the panoramic view.

[0077] Then, the ship track and the track of the center point of the picture are extracted from the calibration image, and the calibration key point set is selected according to the ship track and the track of the center point of the picture through the approximate grid selection method. Figure 2 The flowchart of the approximate grid selection method is shown in the figure. First, the ship track and the track of the center point of the picture are used as the calibration track, and then the grid size is determined according to the median, average or manual experience of the ship track length in the target sea area obtained by existing statistics. The calibration track and the grid size are input into the grid generation algorithm to generate an initial grid including the calibration track. At the same time, the grid size is changed or the grid is moved, and multiple initial grids can be obtained through the grid generation algorithm. Then, the grid intersection points are extracted from each initial grid as seed clustering points. Then, the calibration track is clustered according to the seed clustering points and the DBSCAN algorithm is used to obtain a cluster point set. Each cluster point set is screened, and the points closest to the adjacent grid intersection points in the cluster point set are retained to form a key point set. The average distance between the key points in the key point set and the adjacent grid intersection points is calculated to calculate the average distance of the key points of each initial grid. After obtaining the average distance of the key points of each initial grid, the initial grid with the lowest average distance of the key points is selected as the target grid, and the key point set in the target grid is used as the calibration key point set, thereby obtaining the calibration key point set.

[0078] Next, the world coordinate sampling points are extracted from the first ship automatic identification system information, that is, the position points and historical track points of the passing ships contained in the first ship automatic identification system information, and a mapping relationship between the world coordinate sampling points and the calibration key point set is established, and a coordinate conversion matrix is ​​constructed according to the mapping relationship between the world coordinate sampling points and the calibration key point set. Among them, when the camera is a pan-tilt camera, the calibration key point set is located in the picture taken by the camera in the panoramic view, and the coordinate conversion matrix is ​​constructed through the mapping relationship between the world coordinate sampling points and the calibration key point set in the panoramic view.

[0079] Finally, select any point in the target sea area as the origin to establish a spatial coordinate system as the camera coordinate system, and obtain the camera coordinate position in the camera coordinate system according to the camera deployment position, and then multiply the camera coordinate position with the coordinate transformation matrix to complete the camera calibration.

[0080] S3: The camera monitors the target sea area, and the monitoring module monitors and obtains the second ship automatic identification system information. When a sample ship appears in the target sea area, the camera and the drone obtain a target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0081] Furthermore, the purpose of this stage is to monitor the target sea area, obtain the second ship automatic identification system information through monitoring, obtain the target multi-angle picture through the drone and the camera, and finally obtain the multimodal ship sample using the target multi-angle picture and the second ship automatic identification system information; wherein step S3 further includes:

[0082] S31: the camera monitors the target sea area and determines whether the sample ship appears in the target sea area, and the monitoring module obtains the second ship automatic identification system information through monitoring;

[0083] S32: when the camera determines that the sample ship appears in the target sea area, the drone photographs the sample ship to obtain an overhead image of the sample ship, the camera photographs the sample ship to obtain an overhead image of the sample ship, and the overhead image of the sample ship is merged with the overhead image of the sample ship to obtain a multi-angle image of the target;

[0084] S33: extracting the ship position information of the sample ship in the target multi-angle image, extracting the ship position notification information in the second ship automatic identification system information, matching the ship position information with the ship position notification information, and obtaining the target second ship automatic identification system information matching the sample ship;

[0085] S34: Fusing the target second automatic ship identification system information with the target multi-angle image and removing the background to obtain the multi-modal ship sample.

[0086] With respect to the above steps, the specific implementation methods in this embodiment are as follows:

[0087] First, the camera monitors the target sea area and determines whether a sample ship appears in the target sea area. In this embodiment, the identification of the sample ship is achieved by using the YOLOv5 image recognition algorithm, and the YOLOv5 image recognition algorithm is parameter-optimized for the identification of ship categories. At the same time, the monitoring module in the camera continues to monitor and obtain the second ship automatic identification system information.

[0088] When the camera determines that a sample ship appears in the target sea area, the drone and the camera work together to shoot the sample ship. The drone takes a bird's-eye view of the sample ship, and the camera takes a head-on view of the sample ship. By fusing the bird's-eye view of the sample ship with the head-on view of the sample ship, a multi-angle image of the target can be obtained.

[0089] Next, since the camera has been calibrated, the ship position information of the sample ship can be extracted from the target multi-angle image, and the ship position notification information can be extracted from the second ship automatic identification system information, that is, the information of the current ship position included in the second ship automatic identification system information, and the ship position information is matched with the ship position notification information, that is, the ship position notification information that is the same as the ship position information is selected, and the second ship automatic identification system information corresponding to the ship position notification information can be used as the target second ship automatic identification system information, and the target second ship automatic identification system information matches the sample ship.

[0090] Finally, the target second ship automatic identification system information is fused with the target multi-angle image and the background is removed to obtain a multimodal ship sample. Among them, background removal can leave only the image of the sample ship, and fusion can make the multimodal ship sample include not only the image of the sample ship, but also the length, width, call sign, etc. included in the target second ship automatic identification system information, so that the target second ship automatic identification system information includes more abundant information about the sample ship, further improving the data richness of the multi-sea state detection data set.

[0091] S4: obtaining a sea condition description text and a text image generation model, inputting the multimodal ship sample and the sea condition description text into the text image generation model, and obtaining an initial multi-sea condition ship data set;

[0092] Furthermore, the purpose of this stage is to generate an initial multi-sea state ship data set using a text image generation model. First, obtain a sea state description text and a text image generation model. Here, the sea state description text is a paragraph of explanatory text used to describe the sea state image that the user wants to generate, and the text image generation model can generate the sea state described by the sea state description text. In this embodiment, there are four types of sea state description texts, corresponding to four types of sea state images: When the sea condition is a breeze and there are basically no waves, the description of the sea state description text is: The water exhibits a gentle ripple, devoid of breaking waves, featuring a low swell of short to average length occasionally. When the sea condition is small waves, the description of the sea condition description text is: Slight waves breaking, with smooth waves on the water surface. When the sea condition is obvious wind and waves, the description of the sea condition description text is: Mildly increased waves, leading to some rock buoys and causing minor disturbances for small craft. When the sea condition is characterized by strong winds and rough waves, the description of the sea condition text is: Thesea takes on a furrowed appearance, characterized by moderate waves.

[0093] In addition, the sea condition description text may also include a description of weather phenomena related to the sea condition. The text image generation model selected in this embodiment is a Blended Latent Diffusion text image generation model. By inputting the multimodal ship sample and the sea condition description text into the text image generation model, an initial multi-sea condition ship data set of the multimodal ship sample in the sea condition described in the sea condition description text can be generated.

[0094] S5: Checking the initial multi-sea-state ship dataset to obtain a target multi-sea-state ship dataset.

[0095] Furthermore, the purpose of this stage is to obtain a qualified target multi-sea state ship data set through inspection; wherein step S5 further includes:

[0096] S51: Obtain a convolutional neural network model, train the convolutional neural network model using a ship data set to obtain a ship inspection model; train the convolutional neural network model using a sea condition data set to obtain a sea condition inspection model;

[0097] S52: Verify the initial multi-sea-condition ship data set by using the ship verification model and the sea-condition verification model to obtain the target multi-sea-condition ship data set.

[0098] With respect to the above steps, the specific implementation methods in this embodiment are as follows:

[0099] First, DenseNet is obtained as a convolutional neural network model, and then the convolutional neural network model is trained with a ship dataset including the ship category in the SeaDrone-see dataset and the non-ship category in the Boat-MINIST dataset to obtain a ship inspection model. Next, the convolutional neural network model is trained with a sea state dataset including the Manzoor Umair Sea State Image Dataset (MU-SSiD) dataset to obtain a sea state inspection model. The sea state inspection model can be inspected for the four sea state images described above in this embodiment.

[0100] Finally, the initial multi-sea-state ship dataset is inspected by the ship inspection model and the sea condition inspection model. If the initial multi-sea-state ship dataset does not meet the requirements and fails the inspection, a new initial multi-sea-state ship dataset is regenerated, and the initial multi-sea-state ship dataset that passes the inspection is used as the target multi-sea-state ship dataset. In addition, the reliability of the ship inspection model and the sea condition inspection model can be guaranteed by introducing manual sampling.

[0101] In summary, the present invention automatically generates a multi-sea-state ship dataset by integrating the ship automatic identification system information with the visible light image positioning information, which effectively reduces the cost and difficulty of obtaining image training samples under high sea conditions.

[0102] The system for generating a multi-sea condition detection data set provided by the present invention is described below. The system for generating a multi-sea condition detection data set described below and the method for generating a multi-sea condition detection data set described above can be referred to each other.

[0103] Figure 3 The schematic diagram of the structure of the system for generating a sample multi-sea state detection dataset is as follows: Figure 3 As shown, the method for generating a multi-sea state detection data set as described above includes:

[0104] Camera deployment module: used to determine the target sea area, select cameras and drones, and deploy the drones and cameras in the target sea area, wherein the cameras are provided with a monitoring module for obtaining ship automatic identification system information;

[0105] A camera calibration module: used for the camera to track the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set in the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0106] Multimodal ship sample module: used for the camera to monitor the target sea area, the monitoring module monitors and obtains the second ship automatic identification system information, when a sample ship appears in the target sea area, the camera and the drone obtain the target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0107] Initial multi-sea-state ship dataset module: used to obtain sea-state description text and text-image generation model, input the multi-modal ship sample and the sea-state description text into the text-image generation model, and obtain the initial multi-sea-state ship dataset;

[0108] Target multi-sea-state ship data set module: used to test the initial multi-sea-state ship data set to obtain the target multi-sea-state ship data set.

[0109] on the other hand, Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the method for generating a multi-sea state detection data set, and the method includes:

[0110] S1: Determine a target sea area, select a camera and a drone, and deploy the drone and the camera in the target sea area, wherein the camera is provided with a monitoring module for obtaining information of an automatic identification system of a ship;

[0111] S2: the camera tracks the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set from the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0112] S3: The camera monitors the target sea area, and the monitoring module monitors and obtains the second ship automatic identification system information. When a sample ship appears in the target sea area, the camera and the drone obtain a target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0113] S4: obtaining a sea condition description text and a text image generation model, inputting the multimodal ship sample and the sea condition description text into the text image generation model, and obtaining an initial multi-sea condition ship data set;

[0114] S5: Checking the initial multi-sea-state ship dataset to obtain a target multi-sea-state ship dataset.

[0115] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0116] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the method for generating a multi-sea state detection data set provided by the above methods, the method comprising:

[0117] S1: Determine a target sea area, select a camera and a drone, and deploy the drone and the camera in the target sea area, wherein the camera is provided with a monitoring module for obtaining information of an automatic identification system of a ship;

[0118] S2: the camera tracks the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set from the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0119] S3: The camera monitors the target sea area, and the monitoring module monitors and obtains the second ship automatic identification system information. When a sample ship appears in the target sea area, the camera and the drone obtain a target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0120] S4: obtaining a sea condition description text and a text image generation model, inputting the multimodal ship sample and the sea condition description text into the text image generation model, and obtaining an initial multi-sea condition ship data set;

[0121] S5: Checking the initial multi-sea-state ship dataset to obtain a target multi-sea-state ship dataset.

[0122] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the method for generating a multi-sea state detection data set provided by the above methods, the method comprising:

[0123] S1: Determine a target sea area, select a camera and a drone, and deploy the drone and the camera in the target sea area, wherein the camera is provided with a monitoring module for obtaining information of an automatic identification system of a ship;

[0124] S2: the camera tracks the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set from the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information;

[0125] S3: The camera monitors the target sea area, and the monitoring module monitors and obtains the second ship automatic identification system information. When a sample ship appears in the target sea area, the camera and the drone obtain a target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information;

[0126] S4: obtaining a sea condition description text and a text image generation model, inputting the multimodal ship sample and the sea condition description text into the text image generation model, and obtaining an initial multi-sea condition ship data set;

[0127] S5: Checking the initial multi-sea-state ship dataset to obtain a target multi-sea-state ship dataset.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a multi-sea state detection data set, characterized in that: include: S1: Determine a target sea area, select a camera and a drone, and deploy the drone and the camera in the target sea area, wherein the camera is provided with a monitoring module for obtaining information of an automatic identification system of a ship; S2: the camera tracks the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set from the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information; S3: The camera monitors the target sea area, and the monitoring module monitors and obtains the second ship automatic identification system information. When a sample ship appears in the target sea area, the camera and the drone obtain a target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information; S4: obtaining a sea condition description text and a text image generation model, inputting the multimodal ship sample and the sea condition description text into the text image generation model, and obtaining an initial multi-sea condition ship data set; S5: Checking the initial multi-sea-state ship dataset to obtain a target multi-sea-state ship dataset.

2. The method for generating a multi-sea condition detection data set according to claim 1, characterized in that: Step S1 further comprises: S11: Acquire sea area ship density information and sea area cleanliness information, and determine the target sea area according to the sea area ship density information and the sea area cleanliness information; S12: Select the camera and the drone, determine the camera deployment position and the drone deployment position in the target sea area, and deploy the camera and the drone in the target sea area in sequence according to the camera deployment position and the drone deployment position.

3. The method for generating a multi-sea condition detection data set according to claim 1, characterized in that: Step S2 further comprises: S21: When the passing ship appears in the camera, the passing ship is marked to obtain a marking tag, and the camera tracks the passing ship in the target sea area through the marking tag to obtain the calibration image; at the same time, the monitoring module collects and obtains the first ship automatic identification system information of the passing ship; S22: extracting the ship track and the picture center point track from the calibration image, and selecting a calibration key point set according to the ship track and the picture center point track by using the approximate grid selection method; S23: extracting world coordinate sampling points from the first ship automatic identification system information, establishing a mapping relationship between the world coordinate sampling points and the calibration key point set, and constructing a coordinate transformation matrix according to the mapping relationship between the world coordinate sampling points and the calibration key point set; S24: Establishing a camera coordinate system, obtaining the camera coordinate position, and completing the calibration of the camera in the camera coordinate system through the coordinate conversion matrix and the camera coordinate position.

4. The method for generating a multi-sea condition detection data set according to claim 3, characterized in that: Step S22 also includes: S221: The camera extracts the calibration image to obtain the ship track and the track of the center point of the picture, and uses the ship track and the track of the center point of the picture as the calibration track; S222: Determine a grid size, generate a plurality of initial grids by using the grid size and a grid generation algorithm, extract grid intersection points in each of the initial grids as seed clustering points, and cluster the calibration tracks according to the seed clustering points to obtain a clustering point set; filter the clustering point set to obtain a key point set, calculate the average value of the distances between the key points in the key point set and the adjacent grid intersection points, and obtain the average distance of the key points of each of the initial grids; S223: Selecting the initial grid with the lowest average distance of the key points as the target grid, and using the key point set in the target grid as the calibration key point set.

5. The method for generating a multi-sea condition detection data set according to claim 1, characterized in that: Step S3 further comprises: S31: the camera monitors the target sea area and determines whether the sample ship appears in the target sea area, and the monitoring module obtains the second ship automatic identification system information through monitoring; S32: when the camera determines that the sample ship appears in the target sea area, the drone photographs the sample ship to obtain an overhead image of the sample ship, the camera photographs the sample ship to obtain an overhead image of the sample ship, and the overhead image of the sample ship is merged with the overhead image of the sample ship to obtain a multi-angle image of the target; S33: extracting the ship position information of the sample ship in the target multi-angle image, extracting the ship position notification information in the second ship automatic identification system information, matching the ship position information with the ship position notification information, and obtaining the target second ship automatic identification system information matching the sample ship; S34: Fusing the target second automatic ship identification system information with the target multi-angle image and removing the background to obtain the multi-modal ship sample.

6. The method for generating a multi-sea condition detection data set according to claim 1, characterized in that: Step S5 further comprises: S51: Obtain a convolutional neural network model, train the convolutional neural network model using a ship data set to obtain a ship inspection model; train the convolutional neural network model using a sea condition data set to obtain a sea condition inspection model; S52: Verify the initial multi-sea-condition ship data set by using the ship verification model and the sea-condition verification model to obtain the target multi-sea-condition ship data set.

7. A system for generating a multi-sea condition detection data set, used to execute the method for generating a multi-sea condition detection data set according to any one of claims 1 to 6, characterized in that: include: Camera deployment module: used to determine the target sea area, select cameras and drones, and deploy the drones and cameras in the target sea area, wherein the cameras are provided with a monitoring module for obtaining ship automatic identification system information; A camera calibration module: used for the camera to track the passing ship in the target sea area to obtain a calibration image including the ship track, the monitoring module collects the first ship automatic identification system information of the passing ship, selects a calibration key point set in the calibration image by using an approximate grid selection method, and completes the calibration of the camera by using the calibration key point set and the first ship automatic identification system information; Multimodal ship sample module: used for the camera to monitor the target sea area, the monitoring module monitors and obtains the second ship automatic identification system information, when a sample ship appears in the target sea area, the camera and the drone obtain the target multi-angle picture of the sample ship; extract the ship position information in the target multi-angle picture, and obtain a multimodal ship sample by pairing the second ship automatic identification system information with the ship position information; Initial multi-sea-state ship dataset module: used to obtain sea-state description text and text-image generation model, input the multi-modal ship sample and the sea-state description text into the text-image generation model, and obtain the initial multi-sea-state ship dataset; Target multi-sea-state ship data set module: used to test the initial multi-sea-state ship data set to obtain the target multi-sea-state ship data set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for generating a multi-sea condition detection data set as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a multi-sea condition detection data set as described in any one of claims 1 to 6 are implemented.

10. A computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that: When the program instructions are executed by a computer, the computer can execute the steps of the method for generating multiple sea state detection data sets as described in any one of claims 1 to 6.

Images