Dnn fuzzy identification model training method and hydro-meteorological identification method and device

Abstract

The present application relates to a DNN fuzzy identification model training method, which extracts features of input sample objects through an object DNN fuzzy identification model, maps the features to obtain the classification probability distribution of the objects, maps the classification probability equivalence to the membership of the fuzzy set in the fuzzy sample model library, combines sample labeling to calculate the loss function value of the classification probability distribution to train the model, combines sample labeling to calculate the loss function value of the fuzzy set closeness to adjust the model after completing a generation of training, if the result calculated by any loss function does not reach the preset loss value requirement, randomly shuffles the samples in the fuzzy sample model library and reorders them, iteratively calculates, until the results calculated by the two loss functions both reach the preset loss value requirement. The present application also relates to a method and device for identifying hydro-meteorological elements by using the above-mentioned model trained by the method, which can simulate the automatic identification of hydro-meteorological elements by the crew, improve the accuracy of identification, and greatly reduce the workload of the crew.

Description

Technical Field

[0001] This invention relates to a DNN fuzzy recognition model training method and a hydrological and meteorological recognition method and apparatus, belonging to the fields of artificial intelligence machine learning and fuzzy recognition technology. Background Technology

[0002] In many fields requiring object identification, manual identification is often performed based on classification or grading standards. However, the ambiguity of these standards and the influence of human experience frequently lead to significant discrepancies between the identification results and the actual situation. Specifically, in the identification of hydrological and meteorological elements during ship navigation, the traditional method involves crew members visually observing the ship's surroundings and the sky, relying on their personal experience and national standards for visual observation. While these methods are simple and easy to implement, the standards—such as wave type, wave height, sea state, wind level corresponding to sea surface characteristics, clarity of the sea-sky boundary, weather phenomena, and cloud formations—are vaguely defined. The identification results are susceptible to human error, failing to meet many application requirements and demanding a considerable workload from crew members. For example, regarding wave height, national standards only require crew members to roughly estimate the wave level, without specifying the actual wave height. This level of precision is far from sufficient for purposes beyond simply filling out the logbook. Although GB / T12763.2-2007 also provides a method for measuring wave height using a shipborne buoy-type accelerometer, the equipment is expensive and difficult to deploy. Apart from professional oceanographic survey vessels, no ship uses this type of equipment to measure wave height. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, this invention provides a DNN fuzzy recognition model training method and a hydrological and meteorological recognition method and device, which can simulate human fuzzy recognition ability, automatically identify and record hydrological and meteorological data, avoid the influence of human factors, and effectively improve the accuracy of recognition results.

[0004] The technical solution of this invention to achieve the above objectives is: a DNN (Deep Neural Network) fuzzy recognition model training method (or object DNN fuzzy recognition model training method). This method extracts features from input sample objects using an object DNN fuzzy recognition model, maps these features to obtain the object's classification probability distribution, maps the classification probabilities equally to the membership degrees of fuzzy sets in a fuzzy sample model library, and trains the model by calculating the loss function value of the classification probability distribution in conjunction with sample annotations. After completing one generation of training, the model is adjusted by calculating the loss function value of the fuzzy set proximity in conjunction with sample annotations. If the result of either loss function calculation does not meet the preset loss value requirement, the samples in the fuzzy sample model library are randomly shuffled and reordered. This iterative calculation is repeated until the results of both loss function calculations meet the preset loss value requirement, at which point model training is complete.

[0005] Preferably, based on the number of tasks sharing the extracted features, the trained model is divided into a multi-task object DNN fuzzy recognition model and a single-task object DNN fuzzy recognition model; for the single-task object DNN fuzzy recognition model, based on the different object recognition granularity, the trained model is divided into a DNN classification fuzzy recognition model and a DNN semantic fuzzy segmentation model.

[0006] Preferably, the object DNN fuzzy recognition model includes a DNN fuzzy recognition learning model and a DNN fuzzy recognition implementation model. A feature applicator is added to the back end of the DNN fuzzy recognition learning model obtained after training to form a DNN fuzzy recognition implementation model (or application model).

[0007] Preferably, the loss function includes:

[0008] Object classification probability distribution similarity (or difference) loss function

[0009]

[0010] Where H(P,Q) is the cross-entropy; P(i) is the classification probability distribution of the labeled samples in the training set; Q(i) is the classification probability distribution of the output; and i is the sample number. Let k be the weight of task k, where k = 1, 2, ..., l, k is the task number, and l is the number of tasks. When the number of tasks l = 1, it is a single task. When l>1, it is multitasking;

[0011] and,

[0012] Object fuzzy set proximity loss function

[0013]

[0014] in, Let represent the weight of the i-th fuzzy set for the k-th task, where i is the fuzzy set number, k = 1, 2, ..., l, k is the task number, and l is the number of tasks. For the j-th input sample (element) x in the i-th fuzzy set of the k-th task j The membership degree of the label, where j is the sample number; For the j-th input sample (element) x of the i-th fuzzy set of the k-th task j The output membership degree; m k The number of fuzzy sets for the k-th task; Let l represent the number of samples (elements) in the i-th fuzzy set of the k-th task; when the number of tasks l = 1, it is a single task.

[0015] Preferably, the method for establishing the fuzzy sample model library is as follows: collect a large number of samples covering all recognition tasks, traverse all recognition tasks of all samples, identify the object classification of each sample through manual recognition means, calculate the membership degree of each sample object to all fuzzy sets of each recognition task, generate membership degree vector and classification probability distribution vector, and perform vector annotation on each sample object to construct the fuzzy sample model library. The fuzzy sample model library includes a sample set that divides the samples according to deep learning conventions and a fuzzy set that divides the same samples into corresponding recognition tasks.

[0016] A team of professionals can be organized to identify the object classification of each sample under different identification tasks.

[0017] The samples in the fuzzy sample model library can be divided into training set, test set and validation set, and the same samples can be further divided into fuzzy sets corresponding to all recognition tasks.

[0018] The hydrometeorological identification method (or hydrometeorological element identification method) uses a corresponding hydrometeorological element DNN fuzzy recognition model to identify objects. The objects include the background of the water image, targets around the ship, sea state, wave type, wave height, wave direction, horizontal visibility, cloud type and amount, and weather phenomena in the captured images. The method obtains the membership degree of the corresponding elements to the fuzzy sets in the fuzzy sample model library, and determines the classification or level of the corresponding elements according to the maximum membership principle and its improvement, thereby identifying hydrometeorological elements. The hydrometeorological element DNN fuzzy recognition model is trained using any of the DNN fuzzy recognition model training methods of this invention.

[0019] Preferably, the hydro-meteorological identification method includes the following steps: (1) reading the collected information, including images of the area around the ship taken by the water, sky images, radar sea surface point cloud data, ship navigation data and auxiliary information; (2) preprocessing the collected information, including deleting or marking images without identification value, extracting or organizing valuable information, and adding timestamps and location stamps to the retained information to be identified; (3) segmenting the background of the water image and detecting targets around the ship. If the segmentation and detection results are measurable, then proceed to step (4). If the segmentation and detection results are unmeasurable, then proceed to step (6); (4) Defogging and capturing the wave water area image and identifying the horizontal visibility. Defogging and capturing the wave water area image includes defogging the image and capturing the target water area image; (5) If step (4) is executed, then the machine vision identifies the sea state, wave type and wave height data; (6) The radar identifies the wave type and calculates the wave direction angle, wave height, wavelength and wave period; (7) Select the final wave type and wave height data; (8) Identify the cloud shape and calculate the cloud cover; (9) If step (8) is executed, then identify the weather phenomenon.

[0020] Preferably, in step (3), the method for segmenting the background of the water image is panoramic segmentation or semantic segmentation.

[0021] In step (3), the measurable segmentation and detection results are during the day and without dense fog, while the unmeasurable segmentation and detection results are at night or with dense fog.

[0022] Step (6) includes a method for identifying wave types by inputting a sea surface point cloud image, and using the DBSCAN density clustering method to obtain wave crest and trough information from the sea surface point cloud information, thereby calculating wave height, wave direction, wavelength, and wave period.

[0023] Preferably, the method for selecting the final waveform and wave height data in step (7) is as follows: read the waveform and wave height data identified in steps (5) and (6); if the waveform and wave height data identified in step (5) are empty, then select the waveform and wave height data identified in step (6); if the waveform and wave height data identified in steps (5) and (6) can be read at the same time, then select the waveform data identified in step (5) and the wave height data identified in step (6); if the waveform and wave height data identified in step (6) are empty, then select the waveform data identified in step (5) and determine the wave height according to the waveform category identified in step (5).

[0024] Preferably, in step (8), after outputting the cloud species of each pixel in the sky image, a sky pixel cloud vector is constructed, and the sky pixel cloud vector is mapped to the full cloud vector through mapping and transformation, so as to finally obtain cloud data that conforms to national standards.

[0025] Preferably, in step (1), the information collected at the time point is read from the navigation data storage medium at a preset time point.

[0026] Hydrometeorological identification device (or hydrometeorological element identification device) includes:

[0027] Navigation data storage medium, used to store acquired video images and pictures, sea surface radar point cloud data, ship navigation data and identified hydro-meteorological element data;

[0028] The hydro-meteorological identification unit is used to read video images, pictures, sea surface radar point cloud data, and ship navigation data from the navigation data storage medium at preset time points, identify hydro-meteorological pixels based on the read data, and send the identification results to the navigation data storage medium for storage.

[0029] and,

[0030] The human-computer interaction unit is used to display and view the collected ship navigation data and identified hydrological and meteorological element data information, and is equipped with a human-computer interaction interface.

[0031] The DNN fuzzy recognition models of each hydrological and meteorological element can be stored in the hydrological and meteorological recognition unit or in a storage medium connected to the hydrological and meteorological recognition unit.

[0032] Preferably, the hydrological and meteorological identification unit includes:

[0033] An information preprocessing module is used to preprocess the information read from the navigation data storage medium and use it as input for other modules;

[0034] The water image background segmentation and target detection module is used to perform panoramic segmentation or semantic segmentation on water images to identify the environmental background of the ship and the target objects on the water.

[0035] The dehazing and hydrological element identification module is used to dehaze the water surface screenshots segmented by the water image background segmentation and target detection module, and select the water area image as the hydrological element identification area.

[0036] The sea state, wave type, and wave height recognition module is used to extract sea surface features from images of areas with identified hydrological elements and to identify sea state, wave type, wind level, and wave height.

[0037] The radar point cloud hydrological element identification module is used to extract wave pattern features from the read radar point cloud information, identify wave patterns, and calculate wave direction angle, wave height, wavelength, and wave period.

[0038] The horizontal visibility recognition module is used to extract the water-sky boundary, land background and / or target clarity features from the screenshots segmented by the water image background segmentation and target detection module to identify horizontal visibility.

[0039] The cloud feature recognition and cloud volume calculation module is used to extract cloud features from sky images to identify cloud types and perform semantic segmentation on sky images to calculate cloud volume.

[0040] and,

[0041] The weather phenomenon recognition module is used to extract weather phenomenon features from water images to identify weather phenomena.

[0042] The beneficial effects of this invention are:

[0043] (1) The DNN fuzzy recognition model training method of the present invention trains the fuzzy recognition learning model from two perspectives: the similarity between the output object classification probability distribution and the sample label distribution and the similarity between the output fuzzy set and the sample label fuzzy set. This enables the fuzzy recognition implementation model to approach the fuzzy recognition ability of humans. Compared with the manual recognition method, it has a more unified and accurate fuzzy recognition standard for classifying each object and can automatically and accurately complete the classification of each object.

[0044] (2) The hydro-meteorological identification method of the present invention and the corresponding device can simulate the crew’s visual observation and cognitive judgment ability. Based on the information data collected by the corresponding device, the corresponding fuzzy recognition model is used to automatically identify the hydro-meteorological elements in the ship’s navigation and save the records. There is no need for the crew to perform manual observation, identification and writing input. This can improve the accuracy of hydro-meteorological element identification, avoid the influence of human factors, and greatly reduce the workload of crew members’ observation and recording, and reduce equipment cost investment.

[0045] (3) The hydro-meteorological identification method of the present invention and the corresponding device can realize the integrated identification of all hydro-meteorological elements during ship navigation, which is more suitable for the special application field of ships. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the hydrological and meteorological identification system of the present invention;

[0047] Figure 2 This is a schematic diagram of the hydrological and meteorological identification unit of the present invention;

[0048] Figure 3 This is a schematic diagram of the general multi-task DNN fuzzy recognition learning model of the present invention.

[0049] Figure 4 This is a schematic diagram of the general multi-task DNN fuzzy recognition implementation model of the present invention;

[0050] Figure 5 This is a flowchart of the DNN fuzzy recognition model training method of the present invention;

[0051] Figure 6 This is a schematic diagram of the general DNN semantic fuzzy segmentation learning model of the present invention.

[0052] Figure 7 This is a schematic diagram of the general DNN semantic fuzzy segmentation implementation model of the present invention.

[0053] Figure 8 This is a flowchart of the hydrological and meteorological identification method of the present invention;

[0054] Figure 9 This is a flowchart of the panoramic segmentation of water images according to the present invention;

[0055] Figure 10 This is a schematic diagram of the structural principle of the DNN panoramic segmentation learning model for water images of the present invention;

[0056] Figure 11 This is a schematic diagram of the structural principle of the DNN panoramic segmentation implementation model for water images according to the present invention;

[0057] Figure 12 This is a flowchart of the semantic segmentation and target detection of water images according to the present invention;

[0058] Figure 13 This is a schematic diagram of the structural principle of the DNN semantic segmentation learning model for water images of the present invention;

[0059] Figure 14 This is a schematic diagram of the structural principle of the DNN semantic segmentation implementation model for water images according to the present invention;

[0060] Figure 15 This is a flowchart of the machine vision recognition of sea state, wind level, wave type and wave height according to the present invention;

[0061] Figure 16 This is a schematic diagram of the structure of the multi-task DNN learning model for machine vision recognition of wave type, sea state, wind level and sea surface wave height of the present invention;

[0062] Figure 17 This is a schematic diagram of the structure and principle of a multi-task DNN implementation model for machine vision recognition of wave type, sea state, wind level, and sea surface wave height.

[0063] Figure 18 This is a flowchart of the radar identification waveform, wave height, and wave direction of the present invention;

[0064] Figure 19 This is a flowchart of the process for selecting waveform and wave height data in this invention;

[0065] Figure 20 This is a flowchart of the observation horizontal visibility of the present invention;

[0066] Figure 21 This is a schematic diagram of the structural principle of the object clarity DNN fuzzy recognition learning model of the present invention;

[0067] Figure 22 This is a schematic diagram of the structural principle of the object clarity DNN fuzzy recognition implementation model of the present invention;

[0068] Figure 23 This is a flowchart of the weather phenomenon identification process of the present invention;

[0069] Figure 24 This is a schematic diagram of the structural principle of the weather phenomenon DNN fuzzy recognition learning model of the present invention;

[0070] Figure 25 This is a schematic diagram of the structural principle of the weather phenomenon DNN fuzzy recognition implementation model of the present invention. Detailed Implementation

[0071] See Figure 1This invention discloses a hydro-meteorological identification system based on multiple deep neural networks (DNNs) for collaborative identification of hydro-meteorological elements. It is primarily used for ships to automatically identify hydro-meteorological element data of their maritime environment during navigation, and can also be used for fixed marine or terrestrial hydro-meteorological monitoring. The system is a subsystem of the navigation brain system, including a hydro-meteorological identification device. This device comprises a hydro-meteorological identification unit 1012, a navigation data storage medium 1011, a navigation data human-computer interaction interface 1013, and a navigation brain server display (or human-computer interaction unit) 1014. These components are associated with other subsystem devices of the navigation brain system and are collectively associated with system components other than the navigation brain terminal 101. The hydro-meteorological identification device is a component of the navigation brain server.

[0072] The navigation data storage medium 1011 is a shared storage medium component of multiple devices in the navigation brain system. It stores the collected videos and images, the 4D radar point cloud map of the sea surface collected by the sensors and processed by the navigation data processing device 1015, the original navigation data, and the identified hydrological and meteorological element data.

[0073] The hydro-meteorological identification unit 1012 is responsible for tasks such as reading data from the navigation data storage medium 1011 according to a specified time (or set time), identifying hydro-meteorological elements, and saving data to the navigation data storage medium 1011.

[0074] The navigation brain server display 1014 is used to display the identified hydrological and meteorological element data, navigation data and other auxiliary data information, and is equipped with the navigation data human-computer interaction interface 1013 for human-computer interaction operation.

[0075] The system also includes a hydro-meteorological outdoor unit 102, a ship IoT hub device 103, a data cable 104, and a navigation data processing device 1015. The hydro-meteorological outdoor unit 102 is a sensor used to collect raw hydro-meteorological data of the ship's environment. It includes binocular cameras pointing towards the bow and the water surface on the port and starboard sides, a fisheye camera for capturing the sky, and an anemometer. It also has embedded satellite positioning chips and temperature and pressure sensors. The collected data is directly connected to and transmitted to the navigation brain server 101 via the data cable 104. Other sensors and instruments are connected to the ship IoT hub device 103, either wired or wirelessly. The ship IoT hub device 103 transmits data to the navigation brain server 101 via the data cable 104. The collected data is processed by the navigation data processing device 1015 and stored in the navigation data storage medium 1011.

[0076] See Figure 2The hydrological and meteorological identification unit 1012 is a device for identifying hydrological and meteorological elements based on multi-DNN collaboration, including: an information preprocessing module 200, a water image background segmentation and target detection module 201, a defogging and hydrological element region identification module 202, a sea state, wave type, and wave height identification module 203, a radar point cloud hydrological element identification module 204, a horizontal visibility identification module 205, a cloud shape identification and cloud cover calculation module 206, and a weather phenomenon identification module 207. Each module consists of at least one processor, running its own code to complete its own task.

[0077] The information preprocessing module 200 is used to preprocess the read information as input to other modules;

[0078] The water image background segmentation and target detection module 201 is used to perform panoramic segmentation or semantic segmentation on the water image to identify the environmental background of the ship and the target objects on the water.

[0079] The defogging and hydrological element identification module 202 is used to defog the water surface screenshots segmented by the water image background segmentation and target detection module 201, and select specific water area images for hydrological element identification.

[0080] The sea state, wave type and wave height recognition module 203 is used to extract sea surface appearance features from selected specific water area images and recognize sea state, wave type, wind level and wave height;

[0081] The radar point cloud hydrological element identification module 204 is used to extract wave pattern features from the read radar point cloud information, identify wave patterns, and calculate wave direction angle, wave height, wavelength, and wave period information.

[0082] The horizontal visibility recognition module 205 is used to recognize horizontal visibility.

[0083] The cloud recognition and cloud amount calculation module 206 is used to extract cloud features from the sky image, identify cloud types, perform semantic segmentation on the sky image, and calculate cloud amount.

[0084] The weather phenomenon recognition module 207 is used to extract weather phenomenon features from water images and recognize weather phenomena.

[0085] This invention also discloses a hydrological and meteorological identification method based on multi-DNN collaborative identification of hydrological and meteorological elements. The DNN models for identifying different hydrological and meteorological elements in this invention share a common architectural pattern (model and method), i.e., a general DNN architecture pattern. For specific tasks, the general DNN architecture pattern is adjusted to meet the requirements of those specific tasks.

[0086] The general DNN architecture can be divided into four parts: a preprocessor, a feature extractor, a feature predictor, and a feature applicator. The feature extractor is decoupled from the other structural parts, making it easy to replace with a suitable feature extractor. This invention does not limit the specific selection of the feature extractor, but only uses technically mature or self-evolving, differentiable feature extractors. Typically, the learning model consists of the preprocessor, feature extractor, and feature predictor. After training, the learning model is combined with the feature applicator to form the implementation model, used to complete the specific prediction task. Therefore, the structures of the learning model and the implementation model are not entirely the same. The general DNN architecture includes:

[0087] A general multi-task DNN fuzzy recognition mode, suitable for multi-task (single task is a special case of multi-task) sparse prediction tasks that apply the principles of fuzzy mathematics.

[0088] A general DNN semantic fuzzy segmentation pattern, suitable for dense prediction tasks applying fuzzy mathematics principles.

[0089] The general multi-task DNN fuzzy recognition mode and the general DNN semantic fuzzy segmentation mode have general object DNN fuzzy recognition model training methods and fuzzy recognition methods, as well as methods for building a fuzzy sample model library for training models, but their learning models and implementation models are different.

[0090] 1. A general method for training object-oriented DNN fuzzy recognition models and a fuzzy recognition method, as well as a method for building a fuzzy sample model library for training models.

[0091] One objective of this invention is to enable machine learning to leverage the experience and ability of human senses to perform fuzzy recognition and classification of a given object for at least one task. The general object DNN fuzzy recognition model training method and fuzzy recognition method, along with the method for building a fuzzy sample model library for training the model, are designed using fuzzy mathematics and deep learning principles combined with the features of this invention. The general method is adjusted and adapted to specific recognition tasks to suit the requirements of those tasks, making it applicable to fuzzy perception by all human senses.

[0092] (1) Fuzzy mathematical model and method for object DNN fuzzy recognition

[0093] (1.1) Fuzzy concepts, fuzzy sets, and membership functions

[0094] There is a vague concept If x is an element (object), then the corresponding fuzzy set Represented as In the formula: k = 1, 2, ..., l, where k and l are the task number and quantity of fuzzy recognition, respectively; i = 1, 2, ..., m k i and m kThe fuzzy concept number and quantity of the k-th task are respectively defined; when l = 1, it is a single task, and when l > 1, it is a multi-task task. For fuzzy sets Membership function When x = x j Time-based fuzzy sets membership degree j and These are the element (object) number and quantity for the k-th task, respectively.

[0095] (1.2) Membership vector and classification probability distribution vector

[0096] For the same object x0 and the same task k, x0 has a membership vector.

[0097]

[0098] If x0 serves multiple tasks, then x0 has multiple membership vectors.

[0099] At the same time, because the identification results of the participants are affected by many random factors, the membership value... The image x0 is mapped to an image by a random variable chosen by the person being identified. The probability value of "

[0100]

[0101]

[0102] For task k, x0 has a probability distribution vector for selective classification.

[0103]

[0104] If x0 serves multiple tasks, then there will be multiple classification probability distribution vectors. It should be noted that because the objects to be identified are fuzzy concepts, the results of manual identification are also fuzzy; the classification probability distribution vectors of these objects cannot use the One-Hot vectors used in traditional deep learning manual annotation methods.

[0105] (1.3) Loss function of the DNN fuzzy recognition model for training objects

[0106] The core of training a DNN fuzzy recognition model using maximum likelihood estimation and stochastic gradient descent is to construct a loss function.

[0107] (a) Loss function based on the similarity (or dissimilarity) of object classification probability distributions.

[0108] Choosing the cross-entropy function as the loss function

[0109]

[0110] In the formula, H(P,Q) is the cross-entropy; P(i) is the classification probability distribution of the labeled samples in the training set; Q(i) is the classification probability distribution of the output; and i is the sample number. Let k be the weight of task k, where k = 1, 2, ..., l, k is the task number, and l is the number of tasks. When the number of tasks l = 1, it is a single task. When l>1, it is multitasking.

[0111] (b) Loss function for proximity of object fuzzy sets

[0112] The object DNN fuzzy recognition model uses fuzzy sets As input, the output recognition result is also a fuzzy set, which is... when and When the closest match is found, the corresponding DNN model best approximates human fuzzy recognition ability and can be selected as the optimal DNN fuzzy recognition model. This is achieved by calculating the fuzzy set. For sample set The degree of closeness, or "closeness" to the approximation. We select the optimal DNN fuzzy recognition model for the object. To ensure the differentiability of the DNN fuzzy recognition model, we choose Euclidean proximity and construct the following loss function, Loss2.

[0113]

[0114] In the formula, Let represent the weight of the i-th fuzzy set for the k-th task, where i is the fuzzy set number, k = 1, 2, ..., l, k is the task number, and l is the number of tasks. When the number of tasks l = 1, it is a single task. For the j-th input sample (element) x in the i-th fuzzy set of the k-th task j The membership degree of the label, where j is the sample number; For the j-th input sample (element) x of the i-th fuzzy set of the k-th task j The output membership degree; m k The number of fuzzy sets for the k-th task; The number of samples (elements) in the i-th fuzzy set for the k-th task.

[0115] (2) Training method of object DNN fuzzy recognition learning model

[0116] See Figure 5The training steps of the object DNN fuzzy recognition learning model are as follows: (2.1) Input samples into the DNN fuzzy recognition learning model; (2.2) The DNN fuzzy recognition learning model outputs the classification probability distribution results; (2.3) Map the classification probability to the membership degree: map the classification probability equally to the membership degree of the fuzzy set. membership degree (2.4) Iterative Model Training: The DNN model is trained using backpropagation with cross-entropy as the loss function (Loss 1), and the network parameters are iteratively updated. When the loss stabilizes and is less than the preset loss value, the model can be considered converged. Samples with labeled membership degrees are then added to the corresponding fuzzy sets based on the final non-zero membership values. In the database; (2.5) Iteratively adjust the model: After completing the first generation training (Epoch) of the model in step (2.4), adjust the model using fuzzy sets from the fuzzy sample model library. And predicting fuzzy sets The Loss2 function, which measures the proximity between nodes, is used as the objective function to adjust the DNN model. Backpropagation is performed based on the loss calculation to iteratively update the network parameters, and the model is updated according to the membership degrees mapped by the latest DNN model output. When the loss tends to stabilize and is less than the preset loss value, the model can be considered to have converged; (2.6) Sample reordering training: If Loss1 and Loss2 do not meet the requirements at the same time, the samples in the model library are randomly shuffled and reordered, and training is carried out in a loop. (2.7) Save the determined model.

[0117] (3) General object DNN fuzzy recognition method

[0118] The method for using the object DNN fuzzy recognition implementation model to identify the classification or level of objects is as follows: (3.1) Input the object to be identified or the set of objects into the object DNN fuzzy recognition implementation model; (3.2) The feature predictor of the implementation model outputs the classification probability of the object. Fuzzy set membership degree (3.3) The implementation model determines the classification or level of the object to be identified based on the maximum membership principle and its improvement principle; (3.4) The implementation model auxiliary module adjusts the results of step (3.3).

[0119] (4) General methods for building fuzzy sample model libraries

[0120] According to the fuzzy mathematical model of object recognition of the present invention, a fuzzy sample model library is established to train the object DNN fuzzy recognition learning model. The general method for establishing a fuzzy sample model library is as follows: (4.1) Collect a large number of object sets covering all categories and states and continuously enrich them; (4.2) Traverse all objects: x j,j=1,2,…,n, where n is the number of objects; (4.3) Traverse all fuzzy recognition tasks: the task number is k=1,2,…,l, where l is the number of tasks; (4.4) Recognize object x j Classification: Organize a sufficiently large team of professionals to identify objects x for task k. j Classification i = 1, 2, ..., m k m k Let x be the number of categories for the k-th task; (4.5) Calculate object x j Membership degree to a fuzzy set. x j right When the number of trials is large enough, x j right (4.6) Generate membership vector X and classification probability distribution vector P: Generate object x j For all fuzzy set membership vectors X of task k k and classification probability distribution vector P k (4.7) After traversal, generate the membership vector and classification probability distribution vector of all objects to all fuzzy sets of all tasks. (4.8) Label the data of each object and build a fuzzy sample model library: (a) Label the membership vector of all tasks served by the object. (b) Label the classification probability distribution vector of all tasks served by the object. (4.9) Divide the fuzzy sample model library dataset: (a) Divide the dataset of the fuzzy sample model library into training set, test set and validation set. (b) Divide the same dataset into fuzzy sets of task k. The object described in this invention can be an image or image pixels, or it can be other objects such as voice.

[0121] 2. General object DNN fuzzy recognition learning model and implementation model

[0122] Based on the number of tasks sharing extracted features, the model is divided into a multi-task object DNN fuzzy recognition model and a single-task object DNN fuzzy recognition model. For the single-task object DNN fuzzy recognition model, based on the different object recognition granularities, the model is further divided into a DNN classification fuzzy recognition model and a DNN semantic fuzzy segmentation model. Semantic fuzzy segmentation processes image pixels.

[0123] (2.1) General Multi-Task DNN Fuzzy Recognition Learning Model

[0124] like Figure 3As shown, the general multi-task DNN fuzzy recognition learning model consists of three parts: a preprocessor, a feature extractor, and a feature predictor. The feature predictor includes multi-task network heads, with the number of network heads equal to the number of tasks, l. Each task network head k shares a shared feature map output by the feature extractor, and the mapping module maps the shared feature map to neurons. There are m tasks k. k There are m neurons, corresponding to m k A fuzzy classification algorithm. Each neuron uses the softmax activation function and outputs a fuzzy set given an input object x0. membership degree Classification probability

[0125] (2.2) General Multi-Task DNN Fuzzy Recognition Implementation Model

[0126] like Figure 4 As shown, the general multi-task DNN fuzzy recognition implementation model consists of four parts: a preprocessor, a feature extractor, a feature predictor, and a feature applicator. Compared with the learning model, the implementation model adds a feature applicator, which embeds a fuzzy recognition classification module of the multi-task DNN fuzzy recognition classification method.

[0127] (2.3) General DNN Semantic Fuzzy Segmentation Learning Model

[0128] like Figure 6 As shown, a general DNN semantic fuzzy segmentation learning model consists of three parts: a preprocessor, a feature extractor, and a feature predictor. (a) Preprocessor: Composed of pooling layers to achieve downsampling and dimensionality reduction. The pool size, number of pooling steps, and number of pooling layers are determined based on the input image size of the specific task and the optimal input size required by the feature extractor; (b) Feature extractor: Preferably, a mature and computationally efficient semantic segmentation model (such as Segmenter, FCN variants, but not limited to these models) is adopted; (c) Feature predictor: Composed of a dense prediction layer and an output layer. The output layer is a network head composed of neurons connected to the softmax activation function, predicting the output tensor, which represents the membership vector X of each pixel belonging to the fuzzy classification / the classification probability distribution vector P.

[0129] (2.4) General DNN Semantic Fuzzy Segmentation Implementation Model

[0130] like Figure 7As shown, the general DNN semantic fuzzy segmentation implementation model consists of four parts: a preprocessor, a feature extractor, a feature predictor, and a feature applicator. Compared with the learning model, the implementation model adds a feature applicator. (a) Feature predictor: Compared with the learning model, it adds a fuzzy recognition classification module and an upsampling layer. The fuzzy recognition classification module is responsible for determining the classification of the pixels to be segmented based on the maximum membership principle and its improvement principle; the upsampling layer is responsible for restoring the input image size and performing bilinear interpolation to calculate the added pixel recognition results; (b) Feature applicator: Based on the prior knowledge of the specific semantic segmentation task, it determines the required method and adjusts and corrects the output of the feature predictor.

[0131] See Figure 8 The hydrological and meteorological identification method based on multi-DNN collaboration includes the following steps:

[0132] (a) Reading the collected information;

[0133] (ii) Preprocessing of some information;

[0134] (III) Segment the background of the water image and detect targets around the ship. If the segmentation and detection results are measurable (daytime and no dense fog), proceed to step (IV). If the segmentation and detection results are not measurable (night or dense fog), proceed to step (VI).

[0135] (iv) Defogging, extracting and identifying wave-filled water areas, and identifying horizontal visibility;

[0136] (v) Machine vision recognition of sea state, wave pattern and wave height data;

[0137] (vi) Radar identification of wave element data;

[0138] (vii) Select the final waveform and wave height data;

[0139] (viii) Identify cloud formations and calculate cloud cover. Perform this step after completing step (ii).

[0140] (ix) Identify weather phenomena.

[0141] The process method of step (one) is as follows:

[0142] For the purposes of this invention, it is necessary to set a time point (time t). *The following data were read from the navigation data medium 1011: (1) ship length, ship position latitude and longitude, ship heading angle, drift angle, ship speed, wind speed, and wind direction data; (2) frame images and sky images extracted from the images around the ship; (3) altitude of the radar motion coordinate system origin, sea surface point cloud information vector group, and point cloud image information. The point cloud information has been filtered and transformed into a spatial rectangular coordinate system from the original polar coordinate information. Based on the motion coordinate system calibrated by the equipment (x-axis is positive at the bow, y-axis is positive at the port side, and z-axis is positive upward), each sea surface point cloud point data is a vector p = (x r ,y r ,z r ,v r ), where (x r ,y r ,z r ) represents the spatial coordinates of that point, v r Let be the radial velocity at that point.

[0143] The process method of step (II) includes the following steps:

[0144] 1. Preprocessing of water images

[0145] The image shows a water surface at the bottom, with colors ranging from blue, green, to yellow. The RGB values ​​of the pixels corresponding to the water surface in the image are determined based on statistical data, representing the blue, green, and yellow water components, respectively. Values ​​exceeding the upper limit represent water reflections, which are close to white.

[0146] (1) Remove images without water: Remove images where the percentage of water-colored pixels is less than a threshold λ, i.e., images with no water or very little water. (2) Mark images with water reflections: Mark images where the median pixel color value is greater than the upper limit of the water color component range. (3) Add timestamps and location stamps to the original images and save them.

[0147] 2. 4D radar sea surface point cloud information preprocessing

[0148] (1) Extracting sea surface point cloud data: Only extract 4D radar point cloud data beyond the distance of 30 meters from the ship's hull or half the ship's length (whichever is greater), as required by national standards; (2) Point cloud information coordinate transformation: Transform the radar motion coordinate system into a still water surface motion coordinate system (the xoy plane coincides with the water surface at an altitude of 0 meters), and the vertical coordinate component in the read sea surface point cloud point vector is transformed accordingly to: z s =z r +h, where: z s Let z be the vertical coordinate of the sea surface point based on the coordinate system of still water surface motion. r Let p be the vertical coordinate of the sea surface point based on the radar motion coordinate system, and h be the altitude of the origin of the 4D radar motion coordinate system. The transformed point cloud vector is p. i =(xsi ,y si ,z si ,v si ), i is the point cloud point number. (3) Use a filtering algorithm to perform smooth filtering preprocessing on the 4D radar point cloud data to remove noise points and non-sea surface points (such as spray) in the point cloud. The specific steps are as follows: (3.1) Calculate the average distance from each point to all its neighboring points in the point cloud data, and remove points whose average distance or radial velocity exceeds the standard range. (3.2) Set the filtering radius and threshold, calculate the number N of other points within the radius range of each point in the point cloud data. When N is less than the set threshold, remove the point from the point cloud data; (4) The total number of 4D radar point cloud points after preprocessing is m. Add timestamps and location stamps together with the read sea surface point cloud image and save it.

[0149] 3. Preprocessing of the sky image: (1) Convert the fisheye lens image into a rectangular image; use latitude and longitude to unfold the read sky fisheye image into a rectangular image. (2) Add timestamp and location stamp to the sky rectangular image and save it.

[0150] The process, model, and training method of step (iii) are as follows:

[0151] To identify water surface visibility, hydrological elements, and the need for intelligent collision avoidance, it is necessary to identify and distinguish the ship's environmental background and surrounding objects.

[0152] (1) Ship environmental background (uncountable objects stuff): Ship environmental background includes 5 types of uncountable objects: Water, Sky, WSline, Fog, and Land. The WSline and Fog, and the WSline and Land cannot coexist.

[0153] (2) Targets around the ship (countable objects): Targets around the ship include five types of countable objects: ship, platform, reef, buoy, and UFO.

[0154] To improve the technical robustness of the present invention, the water image background segmentation and target detection model and training method of the present invention can select either a panoramic segmentation method or a semantic segmentation method according to the maturity and capability of the backbone network selected by the feature extractor.

[0155] 1. Workflow and Method for Panoramic Segmentation of Water Images

[0156] See Figure 9The process includes the following steps: (1) Reading water images: Reading water images at time point t*, including images marked as reflective. (2) Panoramic segmentation: (2.1) Inputting the read images into the water DNN panoramic segmentation implementation model; (2.2) Outputting the panoramic segmentation feature map of the water image: that is, the category of each pixel in the image, the bounding box of each region of interest (RoI) and its number; (2.3) Correcting the panoramic segmentation feature map: Generating the corrected panoramic segmentation feature map and adding label information. (3) Extracting object images: (3.1) If there are countable objects, extract the rectangular image of the countable objects and add numbering labels; (3.2) If there is a region where the pixel category = WSline, that is, the water-sky boundary line, extract the rectangular image of the water-sky boundary line and add labels; (3.3) If the original image is not marked as a reflective image, extract the rectangular image of the water surface and add labels. (4) Storing data: Storing the panoramic segmentation feature map of the water image and the extracted images for use in subsequent tasks.

[0157] 2. DNN panoramic segmentation learning model and implementation model for water images

[0158] (1) DNN panoramic segmentation learning model for water images

[0159] See Figure 10 The DNN panoramic segmentation learning model for water images consists of three parts: a preprocessor, a feature extractor, and a feature predictor.

[0160] (1.1) Preprocessor: Composed of pooling layers to achieve downsampling dimensionality reduction. The pool size, number of pooling steps, and number of pooling layers are determined based on the input image size of the specific task and the optimal input size required by the panoramic segmentation model selected by the feature extractor; (1.2) Feature extractor: Selects a mature, high-recognition-ability, and low-cost panoramic segmentation model under existing technology and decouples it from other parts of the model; (1.3) Feature predictor: Composed of a prediction layer and an upsampling layer: (a) Prediction layer: Composed of a semantic segmentation network head, an instance segmentation network head, and a semantic segmentation and instance segmentation fusion network head, outputting the initial feature map of panoramic segmentation; (b) Upsampling layer: Restores the input image size and performs bilinear interpolation to calculate the added pixel type prediction value, outputting the second feature map of panoramic segmentation with the same size as the input image.

[0161] (2) Implementing a model for DNN panoramic segmentation of water images

[0162] See Figure 11A model for panoramic segmentation of water images using DNN was implemented. Based on the learning model for panoramic segmentation of water images using DNN, a feature applicator was added, which includes four parts: preprocessor, feature extractor, feature predictor, and feature applicator. The feature applicator consists of a panoramic segmentation map correction module and a post-segmentation processing module. (2.1) Panoramic segmentation map correction module: Based on the prior knowledge rule that "the boundary between water and sky and fog, and the boundary between water and sky and land cannot coexist", the second feature map of panoramic segmentation is checked and a third feature map of panoramic segmentation is generated. (2.2) Post-segmentation processing module: The rectangular image of the object is cropped and all data is stored.

[0163] 3. Workflow and methods for semantic segmentation and target detection of water images

[0164] See Figure 12 The process includes the following steps: (1) Reading water images: reading time point t * For water images, including images with reflective markings. (2) Image semantic segmentation: (2.1) Input the read images into the water DNN semantic segmentation implementation model; (2.2) Output the water image semantic segmentation feature map, i.e., the category of each pixel in the image. (3) Distinguish countable object instances: (3.1) Object density clustering: Traverse each pixel of type countable object {Ship, Platform, Reef, Buoy, UFO} and cluster it using the DBSCAN density clustering method: A i ={x|Category=K,x∈R} H×W In the formula, A i For an object instance (cluster), i = 1, 2, ..., n, x is a pixel, R H×W For an image of size H×W, K is the object category code, K={Ship,Platform,Reef,Buoy,UFO}; (3.2) Modify clustering: i≠j, that is, the intersection between any clusters is an empty set, otherwise they are merged; (3.3) Object instance number: Add object instance number to the above image and save it. (4) Capture object image: (4.1) If there are countable objects, capture the countable object rectangle and add number mark; (4.2) If there is a region with pixel classification Category=WSline, that is, the water-sky boundary line, capture the water-sky boundary line rectangle and add mark; (4.3) If the original image is not marked as a reflective image, capture the water surface rectangle and add mark. (5) Store data.

[0165] 4. DNN semantic segmentation learning model and implementation model for water images

[0166] (1) Learning model for DNN semantic segmentation of water images

[0167] See Figure 13 The DNN semantic segmentation learning model for water images consists of three parts: a preprocessor, a feature extractor, and a feature predictor. (1.1) Preprocessor: It consists of pooling layers to achieve downsampling and dimensionality reduction. The size of the pool, the number of pooling steps, and the number of pooling layers are determined according to the input image size of the specific task and the optimal input size required by the semantic segmentation model selected by the feature extractor. (1.2) Feature extractor: It selects a mature and powerful semantic segmentation model with strong recognition capabilities under the existing technology and decouples it from other parts of the model. (1.3) Feature predictor: It consists of a dense prediction layer, an upsampling layer, and a module for distinguishing countable object instances. (a) Dense prediction layer: It consists of a mapping module and a network head composed of neurons connected to the softmax activation function, predicting the probability of each pixel belonging to a category; (b) Upsampling layer: It restores the input image size and performs bilinear interpolation to calculate the increased pixel type prediction value, outputting a second feature map of semantic segmentation with the same size as the input image; (c) Module for distinguishing countable object instances: It performs density clustering on each countable object. (1.4) Loss function: Cross-entropy.

[0168] (2) Implementing a model for DNN semantic segmentation of water images

[0169] See Figure 14 A model for DNN semantic segmentation of water images is implemented. Based on the DNN semantic segmentation learning model for water images, a feature applicator is added, which includes four parts: preprocessor, feature extractor, feature predictor, and feature applicator. The feature applicator consists of a semantic segmentation map correction module and a segmentation post-processing module. (2.1) Semantic segmentation map correction module: According to the prior knowledge rule that "the boundary between water and sky and fog, and the boundary between water and sky and land cannot coexist", the second feature map of semantic segmentation is checked and the third feature map of semantic segmentation is generated. (2.2) Segmentation post-processing module: The rectangular image of the object is cropped and all data is stored.

[0170] The process and method for dehazing and identifying wave and water area images in step (iv) are as follows:

[0171] The task of dehazing and identifying wave-prone water areas in images involves two parts: image dehazing and cropping the target water area image. The aim is to ensure the comparability of the extracted wave features with those in the training set, maintaining a relatively constant distance between the camera's optical center and the target water area. Existing mature DNN image dehazing techniques can be used to remove haze and noise from the images.

[0172] The detailed steps for the task of selecting and identifying wave-covered water areas after defogging are as follows: (1) Read the captured water surface image. (2) Input the water surface image into the image defogging DNN model (using a mature DNN model under existing technology). (3) The DNN defogging model outputs the defogging image. (4) Extract the wave-covered water area image according to the principle of binocular ranging: (4.1) Determine the positions of the nearest and farthest points in the image: Determine the positions of the nearest and farthest points on the center line of the water surface image and the positions of the nearest and farthest points mapped in the water surface image according to the principle of binocular ranging. The nearest point is the greater of 30 meters from the hull or half the length of the ship, according to the national standard. The farthest point is the farthest distance that can be measured by binocular ranging. (4.2) Extract the wave-covered water area image: Extract the wave-covered water area image according to the positions of the nearest and farthest points on the center line of the image. (5) Add annotation information to the wave-covered water area image and save it: The annotation information includes vector information such as purpose, timestamp and location stamp.

[0173] The process, model, and training method for identifying horizontal visibility in step (iv) are as follows:

[0174] According to GB18093-2000, visibility is divided into 10 levels, ranging from "less than 50m" to "greater than 50,000m". The four levels above 4,000m are the same as in GB17838-1999, meaning crew members judge visibility by visually observing the clarity of the sea-sky boundary. The level below 4,000m indicates the sea-sky boundary is completely obscured, requiring crew members to visually observe the clarity of surrounding objects and use radar or other equipment to determine visibility. This invention uses machine vision to simulate crew skills, identifying the clarity level of the sea-sky boundary, land background, or surrounding objects, and combining this with radar, AIS, binocular ranging, and other equipment to ultimately determine visibility. The clarity of the sea-sky boundary or objects is artificially defined without a clear boundary, thus possessing ambiguity. Therefore, based on a general multi-task DNN fuzzy recognition model, this invention adjusts the single-task (k=1) method for identifying water surface visibility to construct a process, model, and training method for observing water surface visibility.

[0175] 1. Procedures for observing water surface visibility

[0176] See Figure 20 A process for identifying water surface visibility was developed by referring to the general multi-task DNN fuzzy recognition method, including the following steps:

[0177] (1) Read the image to be identified in terms of clarity: based on time point t * Read the image x, which is a cropped image of the water-sky boundary, from the water image x0. WS Or land image x land or object instance image x ll = 1, 2, ..., L, where L is the number of target objects. If no image is read (the background is foggy and there are no target objects), then the captured image of the undefog-removed water surface, x, is read. water (2) If the background is foggy and there is no target object, use binocular ranging to measure the sharpness distance: use the binocular ranging method to calculate the x-axis of the captured water surface image. water The distance of the furthest wave instance from the ship's camera is used as the sharpness distance data. (3) If the sharpness image to be identified is read, the object sharpness level is determined: (3.1) Input the image into the applicable object sharpness DNN fuzzy recognition implementation model: water-sky boundary image x WS Input to the DNN fuzzy recognition model for the water-sky boundary clarity; land image x land or object instance image x l Then input into the target object clarity DNN fuzzy recognition implementation model; (3.2) The target object clarity DNN fuzzy recognition implementation model outputs the corresponding target clarity level: The fuzzy recognition classification module of the feature applicator determines the target clarity level i according to the maximum membership principle or its improved principle. (4) If the clarity level or distance of all cases has been determined, then determine the visibility level: (4.1) If the clarity level belongs to the water-sky boundary, then the clarity level is converted into the visibility level and saved: Convert the water-sky boundary clarity level i into the visibility level VisibilityClass=i+5. (4.2) If the clarity level belongs to land or target object, then: (a) Determine the target object instance with a clarity level of "2-barely visible": Traverse the clarity data of land and all target objects, and take the target object instance with a clarity level of "2-barely visible"; (b) Determine the target object visibility distance: Read the target object instance at time point t * (a) Determine the visibility class: Based on the target visibility distance and GB18093-2000, determine the visibility class VisibilityClass ∈ {0,1,2,3,4,5}. (b) If the background is foggy and there are no targets, then: (a) Determine the visibility class VisibilityClass: Based on the wave visibility distance and GB18093-2000, determine the visibility class VisibilityClass ∈ {0,1,2,3,4,5}; (c) Determine the visibility class VisibilityClass: Based on the wave visibility distance and GB18093-2000, determine the visibility class VisibilityClass ∈ {0,1,2,3,4,5}; (d) Determine the visibility distance and visibility class VisibilityClass.

[0178] 2. Object Clarity DNN Fuzzy Recognition Learning Model

[0179] According to the national standard GB17838-1999, water surface visibility is classified into five levels based on the clarity of the water-sky boundary: completely invisible, faintly visible, barely visible, clear, and very clear. The image background for "completely invisible" is fog. The further subdivisions of visibility levels are specified by the national standard GB18093-2000, which determines water surface visibility based on the visibility of the target object. Referring to the water-sky boundary clarity classification standard, target object clarity is divided into four levels: faintly visible, barely visible, clear, and very clear.

[0180] Based on a general multi-task DNN fuzzy recognition model, a single-task adjustment (number of tasks k=1) is performed to construct an object sharpness DNN fuzzy recognition learning model, which includes two parts: a feature extractor and a feature predictor. Figure 21 As shown, this invention divides the recognition of the water-sky boundary line clarity and the recognition of the target object clarity into two tasks, respectively training a "DNN fuzzy recognition learning model for object clarity". After training, these models serve as the basis for the "DNN fuzzy recognition implementation model for water-sky boundary line clarity" and the "DNN fuzzy recognition implementation model for target object clarity".

[0181] The feature predictor, the sharpness recognition network head, adapts and predicts object features for sharpness recognition, outputting the fuzzy set membership degree of the object sharpness. There are four fuzzy classification concepts for sharpness, denoted as a. i The numbers i = 1, 2, 3, 4. The universe of discourse U is the entire set of images, and there are 4 fuzzy sets, A. i The feature predictor's output layer has four neurons corresponding to four sharpness levels. Each neuron outputs the probability / fuzzy set membership degree of sharpness level i.

[0182] 3. Implementation Model of DNN Fuzzy Recognition for Object Clarity

[0183] Based on the object sharpness DNN fuzzy recognition learning models trained separately on the horizon line and target object training sets, feature applicators are added to form two independent implementation models: the horizon line sharpness DNN fuzzy recognition implementation model and the target object sharpness DNN fuzzy recognition implementation model. The two models have identical structures, as shown below. Figure 22 As shown. The feature applicator embeds a fuzzy recognition classification module that performs a multi-task DNN fuzzy recognition classification method.

[0184] 4. Training method for DNN fuzzy recognition learning model for object sharpness

[0185] Based on a general multi-task DNN fuzzy recognition learning model training method, the number of classification / fuzzy sets m=4, with explicit loss functions Loss1 and Loss2. During training, two independent models are trained using datasets of the water-sky boundary line and land images / object instance images respectively.

[0186] The process, model, and training method of step (v):

[0187] Traditionally, seafarers identified sea states, wave types, and wave heights by visually observing the sea surface features according to national standards GB 18093-2000, GB17838-1999, and GB12763.2-2007, based on their experience. They then selected several waves for each wave type to estimate their wave height. However, since most ships are now equipped with wind speed and direction sensors, the purpose of seafarers determining wind level by visually observing sea surface features is no longer simply to determine the wind level itself. Instead, it's about identifying wave height based on the close correlation between wave height and wind level (wind speed) using sea surface features that correspond to wind level standards. Although national standards do not specify methods for directly identifying wave height through sea surface features, in practice, many seafarers directly observe sea surface features to estimate wave levels.

[0188] This invention uses machine vision to simulate the work of crew members visually observing and identifying sea conditions, wave patterns, wind speeds, and wave heights according to the aforementioned national standards. Among these, wave height is a widely used and crucial element. To improve the accuracy and robustness of wave height identification, this invention identifies wave height and obtains the optimal value through three methods: inferring wave height based on wind speed identification from sea surface features, directly identifying wave height based on sea surface features, and calculating wave height based on 4D radar point cloud information.

[0189] Since sea state, wave type, and wind level classifications and grades are artificially defined by humans and lack clear extensions, they are inherently ambiguous. Therefore, based on the universal multi-task DNN fuzzy recognition mode of this invention, this paper adjusts and constructs a multi-task fuzzy recognition process method and DNN model structure for recognizing sea state, wave type, wind level, and wave height data for multiple tasks.

[0190] 1. Procedures for identifying sea state, wave type, and wave height

[0191] See Figure 15 Referring to the general DNN multi-task fuzzy recognition method, a process method for recognizing sea state, wave type and wave height is formulated, including the following steps: (1) Read the image of the water area to be identified: based on the time point t * Read the wave-shaped water area image x extracted from the water image x0. wave (2) Input x waveImplement the model to a multi-task DNN. (3) The feature extractor extracts the overall appearance features of the sea surface. (4) The wave pattern feature applicator outputs and saves the wave pattern classification: (4.1) After completing step (3), the overall appearance feature map of the sea surface is passed into the wave pattern feature predictor; (4.2) The wave pattern feature predictor realizes the fuzzy recognition of wave pattern classification and outputs the wave pattern classification membership degree. i = 1, 2, 3, 4, 5 are wave type classification numbers; (4.3) The fuzzy recognition classification module of the wave type feature app determines the wave type according to the maximum membership principle or its improved principle; (4.4) Save the determined wave type classification data. (5) The sea state feature app outputs and saves the sea state: (5.1) This is done simultaneously with step (4), and the overall appearance feature map of the sea surface is transmitted to the sea state feature predictor; (5.2) The sea state feature predictor realizes the fuzzy recognition of the sea state level and outputs the sea state level membership degree. i = 1, 2, ..., 10 are the sea state level numbers; (5.3) The fuzzy recognition classification module of the sea state feature app determines the sea state level according to the maximum membership principle or its improved principle; (5.4) Save the determined sea state level data. (6) The wind level feature app outputs the wind level and wind level wave height and saves them: (6.1) This is done simultaneously with step (4), and the overall appearance feature map of the sea surface is transmitted to the wind level feature predictor; (6.2) The wind level feature predictor realizes the fuzzy recognition of the wind force level and outputs the wind force level membership. i = 1, 2, ... , 13 is the wind force level number; (6.3) The fuzzy recognition classification module of the wind force feature app determines the wind force level based on the maximum membership principle or its improved principle; (6.4) The wind force wave height interpolation block of the wind force feature app reads the same time point t * The wind speed value measured by the anemometer. If the wind speed value read at the same time is within the wind speed range corresponding to the wind force level output in (6.3), the median wind speed is used to interpolate the effective wave height value of the wind waves by interpolating the wave height-wind speed relationship curve obtained according to national standards GB 18093-2000 and GB17838-1999. If the wind speed value is not within the range corresponding to the output wind force level, then abandon the estimation of wave height by wind force level; (6.5) Save the obtained wind force level and significant wave height values. (7) Identify sea surface wave height values ​​based on sea surface features and wave patterns: After completing step (4), (7.1) read the wave pattern value. If there is no wave pattern value, take the wind wave height. and swell wave height h UW =0. (7.2) After the overall sea surface appearance feature map is adapted by the adapter, it is transmitted to the wave height regressor to output the wave height value. (7.3) If the wave type is U or U / F, the wave height value is the swell wave height h. UW If the wave type is F or F / U, then the wave height value is the wind wave height. If the waveform is FU, then the wave height is FU. and h UW (7.4) Preserve wind and wave height values. and surge wave height h UF .

[0192] 2. Multi-task DNN learning model for identifying sea state, wave type, wind level, and sea surface wave height.

[0193] The four tasks of identifying sea state, wave type, wind force, and sea surface wave height are all accomplished by identifying the overall appearance features of the sea surface. Therefore, to improve the application of sample data and model efficiency, the DNN model for these four identification tasks is modified and adjusted based on the general DNN multi-task fuzzy recognition learning model, so that the four task network heads of the DNN feature predictor share the overall sea surface appearance features extracted by the DNN feature extractor. The multi-task DNN learning model for identifying sea state, wave type, wind force, and sea surface wave height consists of two parts: a feature extractor and a feature predictor. The model structure is as follows: Figure 16 As shown.

[0194] (1) Waveform Classification Fuzzy Recognition Network Head: Adapts and predicts sea surface features for waveform classification fuzzy recognition, outputting waveform classification probability / fuzzy set membership degree. According to the national standard GB12763.2-2007, sea surface waveforms are classified into five categories based on wave appearance characteristics: wind waves (F), swells (U), mixed wave 1 (FU), mixed wave 2 (F / U), and mixed wave 3 (U / F). Therefore, there are 5 artificially defined fuzzy classification concepts, denoted as... i = 1, 2, 3, 4, 5, where k = 1 represents the wave type classification task. The universe of discourse U is all sea surface images, with 5 fuzzy sets, denoted as... i = 1, 2, 3, 4, 5. For the waveform classification task, the feature predictor of the general DNN fuzzy recognition learning model of this invention is adjusted. (1.1) Output layer: 5 neurons in total, each neuron outputs the probability / fuzzy set membership degree of waveform classification i, for a total of 5 waveform classification probabilities / fuzzy set membership degrees. i = 1, 2, 3, 4, 5. (1.2) Take the number of classification / fuzzy sets m1 = 5 and the explicit loss function Loss1. wave and Loss2 wave

[0195] (2) Sea State Classification Fuzzy Recognition Network Head: The sea state classification fuzzy recognition network head adapts and predicts sea surface features for fuzzy recognition of sea state classes, outputting the sea state class probability / fuzzy set membership degree. According to the national standard GB12763.2-2007, sea surface classes are divided into 0 to 9 levels based on sea surface characteristics. Therefore, there are 10 artificially defined fuzzy classification concepts, denoted as... i = 1, 2, ..., 10, where k = 2 represents the sea state classification task. The universe of discourse U is all sea surface images, and there are 10 fuzzy sets. i = 1……10. For the sea state level identification task, the feature predictor of the general DNN fuzzy recognition learning model of this invention is adjusted. (2.1) Output layer: A total of 10 neurons correspond to 10 sea state levels. Each neuron outputs the probability / fuzzy set membership degree of sea state level i, for a total of 10 sea state level probabilities / fuzzy set membership degrees. i = 1, 2, ..., 10. (2.2) Take the number of classification / fuzzy sets m2 = 10 and the explicit loss function Loss1 SeaCond and Loss2 SeaCond .

[0196] (3) Wind force level fuzzy recognition network head

[0197] The wind force level fuzzy recognition network head adapts and predicts wind force levels by using sea surface features as fuzzy recognition methods, outputting the wind force level probability / fuzzy set membership degree. According to the national standard GB 17838-1999, sea surface wind force levels are divided into 0-12 levels based on sea surface characteristics. Therefore, there are 13 artificially defined fuzzy classification concepts, denoted as... i = 1, 2, ..., 13, where k = 3 represents the wave type classification task. The universe of discourse U is all sea surface images, and there are 13 fuzzy sets. i = 1……13. For the wind force level identification task, the feature predictor of the general DNN fuzzy recognition learning model of this invention is adjusted. (3.1) Output layer: A total of 13 neurons correspond to 13 wind levels. Each neuron outputs the probability / fuzzy set membership degree of wind level i, for a total of 13 wind level probabilities / fuzzy set membership degrees. i = 1, 2, ..., 13. (3.2) Take the number of classification / fuzzy sets m313 and the explicit loss function Loss1 Wind and Loss2 Wind .

[0198] (4) Seascape wave height regression network head: The seascape wave height regression network head is adapted and predicted according to the general DNN model and seascape wave height regression task of this invention, and outputs the seascape wave height regression result. (4.1) Output layer: 1 neuron, outputs wave height value. (4.2) Loss function: Euclidean distance is selected as the loss function. in, For label wave height, To predict wave height.

[0199] 3. Multi-task DNN implementation model for identifying sea state, wave type, wind force, and sea surface wave height.

[0200] See Figure 17Referring to the general DNN multi-task fuzzy recognition implementation model, the multi-task DNN implementation model for recognizing sea state, wave type, wind level, and sea surface wave height is based on the trained multi-task DNN learning model for recognizing sea state, wave type, wind level, and sea surface wave height, with the addition of feature applicators. (1) Wave type classification recognition feature applicator and sea state level recognition feature applicator: including fuzzy recognition classification module; (2) Wind level recognition feature applicator: including fuzzy recognition classification module and wind wave height interpolation module. Among them, the wind wave height interpolation module realizes the interpolation of the effective wave height value of wind and waves in the wave height-wind speed relationship regression curve according to the median wind speed corresponding to the wind level. The function.

[0201] 4. Training method for DNN fuzzy recognition learning model of wave type, sea state, wind level, and seascape wave height.

[0202] Based on the training method of the general multi-task DNN fuzzy recognition learning model, the following adjustments and adaptations are made according to the four tasks of wave type, sea state, wind level and sea surface wave height.

[0203] The wave pattern, sea state, wind level, and sea surface wave height DNN fuzzy recognition learning model is a multi-task learning model. Therefore, the loss function Loss1 used for iterative training of the model should be composed of the loss functions of the four tasks, and the loss function Loss2 used for iterative adjustment of the model should be composed of the loss functions of the three fuzzy recognition tasks.

[0204]

[0205] The weights of the training loss function for each task.

[0206]

[0207] The weights of the adjusted loss function for each fuzzy set of each task.

[0208] 5. Methods for establishing wave height training, testing, and validation sets.

[0209] The purpose of this invention is to enable machine learning to teach seafarers the skill and experience of inferring wave heights from the surface features of the sea with the naked eye. This invention uses wave heights measured by contact wave height meters in the same water area as training, testing, and validation datasets to train a DNN wave height regression learning model. The steps for establishing the dataset are as follows: (1) Read the wave height values ​​measured by contact wave height meters in the same water area as the "wave identification water area". (2) Identify and match wave water area images in chronological order. (3) Label the wave height values ​​on the corresponding images. (4) The division of the training, testing, and validation sets is the same as that for wave type classification datasets.

[0210] The process, model, and training method of step (vi) are as follows:

[0211] While machine vision can identify wave shape and wave height, the identified wave height value has a large error and it cannot identify wave height at night or in dense fog. 4D millimeter-wave radar solves these problems. At night or in dense fog, the marine 4D millimeter-wave radar can detect wave height at a range of 500-800 meters, meeting the observation range requirements of the national standard GB12763.2-2007. To improve the robustness of this invention, during daytime conditions, both machine vision and 4D millimeter-wave radar are used to simultaneously detect wave height, with priority given to the 4D millimeter-wave radar detection results.

[0212] National Standard GB12763.2-2007 specifies methods for visually observing wind and waves using automatic wave measuring instruments. GB17838-1999 specifies methods for seafarers to visually estimate wave height. Machine vision struggles to identify wave instances amidst numerous, chaotic, and randomly changing waves, but 4D millimeter-wave radar can collect large-scale point cloud data of sea surface water quality, including height, velocity, and direction data. This invention only considers waves whose actual statistical characteristics do not change over time, i.e., wave changes are a stationary random process. Based on the ergodicity of this stationary random process, wave records at a specific moment in the observed water area can replace wave records collected by automatic wave measuring instruments over a set duration. Therefore, this invention can calculate wave elements according to the methods specified in National Standard GB12763.2-2007.

[0213] This invention employs the DBSCAN density clustering method to cluster 4D millimeter-wave radar point cloud vector information according to wave troughs and crests, thereby determining wave troughs and crests and calculating wave elements such as wave height, wave direction, wavelength, and wave period according to national standards. The k-nearest neighbor method is used to determine the parameters Eps and MinPts for clustering wave troughs and crests using the DBSCAN method, where Eps represents the height and radial velocity components z in the point cloud vector. s ,v s The sum of squares of the neighborhood radius ( This invention does not limit the radar device to 4D millimeter-wave radar; the method of this invention is also applicable to other radars such as 4D lidar.

[0214] The crest and trough lines of swell waves are relatively long; the crest and trough lines of wind waves are short. The height and velocity of water particles on the crest and trough lines are almost equal, and the shape of the crest and trough lines is a straight line or a curve that is close to a straight line. Therefore, each point in each wave trough or crest cluster satisfies three conditions: (1) height z s and velocity v s The values ​​are close; (2) There exists an approximate straight line equation for the crest line y. s =kx s+b; (3) If the wave pattern is U or U / F (swell), the crest-trough length is greater than the threshold L. If the wave pattern is F or F / U or FU (wind wave), the crest-trough length is less than or equal to the threshold L.

[0215] 1. Process and Method for Radar Identification of Wave Element Data

[0216] See Figure 18 The method for radar to identify wave element data includes the following steps: (1) Reading time point t * The navigation and radar data: The data read includes the waveforms identified by machine vision, wind direction, heading angle, drift angle, ship speed, point cloud vector data of sea surface appearance collected by the wind speed and wind direction instrument, pi, i = 1, 2, ..., m, and point cloud image data of sea surface appearance. (2) If the waveform data identified by machine vision cannot be read (due to night or dense fog), the point cloud image of the sea surface is input into the point cloud image recognition waveform DNN model; if it can be read, proceed to step (5). (3) The feature extractor extracts the overall appearance features of the sea surface. (4) The waveform feature applicator outputs the waveform and saves it. (4.1) The sea surface appearance features are adapted by the adapter and then transferred to the waveform feature applicator; (4.2) The waveform feature applicator realizes the waveform classification fuzzy recognition and outputs the waveform classification probability / membership. i = 1, 2, 3, 4, 5 are wave type classification numbers; (4.3) The wave type feature application "fuzzy recognition classification module" determines the wave type according to the maximum membership principle or its improved principle; (4.4) Save the output wave type classification data. (5) DBSCAN method calculates wave trough clusters and number: (5.1) Determine the possible set of trough points: a trough is the lowest point (minimum value point) of a single wave. Traverse all sea surface point cloud vector height components p i (z s For point cloud vector information where ) < 0, apply the algorithms of "multivariate function extrema and their methods" and "numerical differentiation" to find possible valley point cloud vector sets that satisfy the necessary and sufficient conditions for the extrema of a bivariate function. j = 1, 2, ..., n t n t The number of possible wave troughs. To avoid interference from sea surface tension waves, the difference step size of the numerical differentiation should not be less than the maximum wavelength of the tension wave. (5.2) From the possible wave trough point cloud point vector group In (γ is a constant less than 1) randomly selects the core object seed for trough density clustering. (The distance between any two sub-subs is greater than the estimated wavelength) λ is a constant), k = 1, 2, ..., K, where K is the number of seeds for the core object, and K > 100. (5.3) Use the DBSCAN method and trough clustering parameters to find the set of points with the largest density connection. The point vectors selected for density accessibility, density reachability, and density connection must meet the following conditions: their height and velocity are close to those of the core object, i.e. If two If the clusters are density-connected, discard one of them and return to step (5.2) to select a new core object seed and repeat the calculation. For surges, if the cluster diameter... If the value is less than the threshold L for the peak line length, then continue searching for density-connected point vectors. (5.4) The number of clusters N obtained after completing the clustering calculation is the number of troughs. l = 1, 2, ..., N, where the average height is the average height of the point cloud points in the same cluster. (6) DBSCAN method to calculate wave crest clusters: (6.1) Determine possible set of crest points: Traverse the cloud of trough points Using the center of the circle to estimate the wavelength p represents the height component of the vector of all sea surface point clouds within the same π radian radius. i (z s For point cloud vector information where )>0, apply the algorithms of "multivariate function extrema and their methods" and "numerical differentiation" to find possible peak point cloud vector sets that satisfy the necessary and sufficient conditions for the extrema of a bivariate function. j = 1, 2, ..., n p ,n p The number of possible wave crests. To avoid interference from sea surface tension waves, the difference step size of the numerical differentiation should not be less than the maximum wavelength of the tension wave. (6.2) Seed the wave crest point cloud point vector group near the possible wave trough core object. In the process, select the core object seed for the corresponding peak density clustering. (6.3) Use the DBSCAN method and peak clustering parameters to find the set of points with the highest density connectivity. The selected point vectors that are density-reachable, density-accessible, and density-connected must meet the following conditions: their height and velocity must be close to the core object. For surges, if the cluster diameter If the value is less than the threshold L for the peak line length, then continue searching for point vectors that are density-connected. (6.4) The clusters obtained after completing the clustering calculation are peaks. For points l = 1, 2, ..., N, their average height is the average height of the points in the same cluster. (7) Determine wave height: (7.1) Traverse N wave crests and troughs to determine wave height. (7.2) If the wave type is a swell (U or U / F), then take h. lThe average of the top five wave height values ​​arranged in order of magnitude is the swell wave height; if it is a wind wave (F, F / U, or FU), then h is taken as the average. l The average value of the first third of the wave heights arranged by size is the effective wave height. (8) Determine the wave direction to the ground: use the least squares method to solve the clustering of N wave crest lines. The equation of the straight line y s l =k l x s l +b l The slope k in l and intercept b l l = 1, 2, N. Average slope of the crest line. For wind and waves, if the wave crest line is too short to regress into a straight line equation, it is recorded as none, and the wind direction angle is taken as the wave direction angle. The wave direction is perpendicular to the wave crest line, that is, the slope k' of the wave direction line is -k. In the moving coordinate system, according to the relationship between the motion vectors, if v s If <0, then the wave direction angle ψ ws =tan -1 k'-180°; if v s If ≥0, then the wave direction angle ψ ws =tan -1 k'. If the heading angle is ψ, then after transforming the coordinate system to a fixed coordinate system, the wave heading angle ψ relative to the ground wave is... wo For: ψ wo =ψ ws +ψ. (9) Determine the wave wavelength and period: (9.1) Rotate and transform the motion coordinate system into the wave coordinate system (x w (9.2) Traverse all wave crest line equations used to calculate wave height, and take the distance between each wave crest line and the adjacent wave crest line as the wave wavelength. The average wavelength of all waves used to calculate wave height is the wave wavelength. (9.3) Calculate the wave period according to the wave dispersion relation. (10) Save the wave element data to the navigation data storage medium 1011.

[0217] 2. Methods for verifying the accuracy of wave element data from radar observations

[0218] The radar observation results are compared with wave element data observed by a contact wave meter in the same sea area at the same time. If the error meets the national standard, the radar observation results are considered to meet the requirements. If the error exceeds the specified value, the radar calibration parameters are adjusted and observations are repeated until the accuracy meets the requirements.

[0219] The process method of step (vii) is as follows:

[0220] See Figure 19The method for selecting wave type and wave height data includes the following steps: (1) Read the wave type and wave height data at time point t*: Read the wave type and wave height data identified by machine vision and 4D radar respectively. (2) If the machine vision identification data is empty (i.e., at night or in foggy weather), then take the wave type and wave height data identified by 4D radar and save it. (3) If the wave type and wave height data identified by machine vision and 4D radar can be read at the same time, then take the wave type data identified by machine vision and the wave height data identified by 4D radar and save them. (4) If the 4D radar identification data is empty (radar equipment failure or no equipment), then: (4.1) If the wave type is U or U / F, then select the wave height h of the sea swell. UW And save. (4.2) If the wave type is F, F / U, or FU, then calculate the wind wave height. And save. (4.3) If there is no waveform, the wave height is zero and save.

[0221] The process, model, and training method of step (viii) are as follows:

[0222] Traditional methods for identifying cloud types and calculating cloud cover involve seafarers visually observing and calculating cloud types (cloud area ratios) according to national standards GB 18093-2000 and GB17838-1999, based on their experience. Clouds are then sorted by family, and within the same family, they are arranged and recorded according to their cloud cover size. This invention uses machine vision to simulate this skill. Since cloud types are artificially defined fuzzy classification concepts, and DNN semantic fuzzy segmentation methods are suitable for cloud cover calculation, the process, model, and training method of this invention for identifying cloud types and calculating cloud cover are adjusted or have parameters specified based on national standards, building upon the general DNN semantic fuzzy segmentation model.

[0223] 1. Procedures for identifying cloud types and calculating cloud cover

[0224] Methods for identifying cloud formations and calculating cloud cover include the following steps:

[0225] (1) Read the sky rectangle image: Read the time point t * A rectangular image of the sky, x0.

[0226] (2) Determine the cloud cover and cloud type:

[0227] (2.1) Input image to sky image DNN semantic fuzzy segmentation implementation model: Input x0 (total number of pixels is m) into sky image DNN semantic fuzzy segmentation implementation model for semantic segmentation, that is, identify the cloud-like species probability / membership degree of each pixel in the image.

[0228] (2.2) Output the cloud species for each pixel: Sky image DNN semantic fuzzy segmentation model outputs the cloud species for each pixel in the sky rectangle image. i i = 1, 2, ..., m:

[0229] cf i ∈{Bk,Cu,Cb,Sc,St,Ns,As,Ac,Ci,Cs,Cc,Lf,Lp,Lh,Ni,Oo}.

[0230] Among them, blue sky (Bk), low cloud-cumulus (Cu), low cloud-cumulonimbus (Cb), low cloud-stratocumulus (Sc), low cloud-stratus (St), low cloud-nimbostratus (Ns), mid cloud-alpine cloud (As), mid cloud-altocumulus (Ac), high cloud-cirrus (Ci), high cloud-cirrostratus (Cs), high cloud-cirrocumulus (Cc), low cloud full fog (Lf), low cloud partial fog (Lp), low cloud haze (Lh), cannot be determined (Ni), other objects (Oo); 2-11 are cloud types specified by the national standard, and 12-14 are special cases of low clouds specified by the national standard.

[0231] (2.3) Constructing the cloud-like vector of sky pixels

[0232] CF==(cf1,cf2,…,cf m ),

[0233] (2.4) Mapping to generate a full vector of cloud-like cloud cover

[0234] After several mapping calculations, the sky pixel cloud vector CF is mapped to the sky cloud amount full vector FCAF.

[0235] Mapping f1:CF→CAF

[0236] (a) Calculate the total number of sky pixels in the sky rectangle image: PIXEL total =m - number of pixels with cloud-like shape Oo;

[0237] (b) Calculate the number of pixels for each cloud shape except for the cloud shape that is Oo. PIXEL j j = 1, 2, ..., 15;

[0238] (c) Calculate the cloud cover for each cloud type j.

[0239] (d) Merge elements of the same cloud shape in the sky pixel cloud shape vector CF into one element, and map the sky pixel cloud shape vector CF to the sky cloud amount cloud shape vector CAF = (r1cf1, r2cf2, ..., r 15 cf 15 ),0 <r j <1.

[0240] Mapping f2: CAF → FCAF

[0241] According to national standards: 2-6 low cloud elements are arranged in descending order of cloud cover; 7-8 medium cloud elements are arranged in descending order of cloud cover; 9-11 high cloud elements are arranged in descending order of cloud cover. The cloud cover vector CAF is mapped to the total cloud cover vector FCAF = (r1Bk,…,r k Xx,…,r 12 Lf,r 13 Lp,r 14 Lh,r 15 Ni),

[0242] Xx represents sorted cloud formations, r k For the corresponding cloud cover, 0≤r k ≤1,k=1,2,…,15,k is the sorting number of cloud type and amount.

[0243] (3) Calculate low cloud cover

[0244]

[0245] (4) Save cloud cover data

[0246] Save the total vector of cloud cover and cloud shape (FCAF) and low cloud cover data to the navigation data storage medium 1011.

[0247] 2. DNN learning model, training method, and implementation model for semantic fuzzy segmentation of sky images

[0248] According to national standards GB 18093-2000 and GB 17838-1999, there are 16 categories for cloud formations in the sky. This means there are m = 16 artificially defined fuzzy classification concepts, denoted as a. i The domain U is the set of all "cloud-like" objects, i = 1, 2, ..., 16. There are 16 fuzzy sets, denoted as A. i ,i=1……16.

[0249] The DNN learning model, training method, and implementation model for semantic fuzzy segmentation of sky images are general DNN semantic fuzzy learning models, training methods, and implementation models with a fuzzy set size of m=16.

[0250] The process, model, and training method of step (ix) are as follows:

[0251] According to GB18093-2000, weather phenomena are classified into 20 types, of which 5 are related to cloud cover, and the remaining 15 require crew members to identify them by visual observation. According to GB / T17838-1999, weather phenomena are classified into 14 types, all of which require crew members to identify by visual observation. This invention selects the type with the most types in the national standard as the type to be identified (1—Sunny, 2—Partly Sunny, 3—Cloudy, 4—Overcast, 5—Bad Weather, 6—Rain, 7—Drizzle, 8—Snow, 9—Sleet, 10—Hail, 11—Fog, 12—Light Fog, 13—Haze, 14—Tornado, 15—Thunderstorm, 16—Thunderstorm, 17—Temporary Storm, 18—Showers, 19—Snow Showers, 20—Sleet Showers).

[0252] To determine the types of rain, snow, and sleet, it is necessary to compare the intensity of rain and snow at different times and combine this with the pattern of wind speed changes.

[0253] This invention uses machine simulation of crew skills, or determines the type of weather phenomenon based on cloud cover calculated according to the invention, or identifies the type of weather phenomenon based on DNN fuzzy recognition. Therefore, based on the general multi-task DNN fuzzy recognition mode, adjustments are made for the single task (k=1) of weather phenomenon identification to construct a process method, model, and training method for weather phenomenon identification.

[0254] 1. Procedures and methods for identifying weather phenomena

[0255] The process method was developed by referring to the general multi-task DNN fuzzy recognition method. See [link / reference]. Figure 23 The process includes the following steps: (1) Reading water images: based on time point t * Read the water image x0, which is the original image without past fog processing; (2) Determine the type of weather phenomenon: (2.1) Input x0 into the weather phenomenon DNN fuzzy recognition implementation model; (2.2) The weather phenomenon DNN fuzzy recognition implementation model outputs the type of weather phenomenon; (3) If the determined type of weather phenomenon i does not exist, proceed to step (4). If i equals any one of 6, 8, 9, 15, 16, determine the type of intermittent weather phenomenon: (3.1) Read the wind speed at the previous time point t*-Δt and the already identified weather phenomenon; (3.2) If i = 6 or 15 or 16, the wind speed is greater than or equal to the storm threshold and t * If i≠6 at -Δt, then i=17; (3.3) If i=6 or 15 or 16, the wind speed is less than the storm threshold and t * When i ≠ 6 at -Δt, then i = 18; (3.4) If i = 8, and t *If i ≠ 8 at -Δt, then i = 19; (3.5) If i = 9, and i ≠ 9 at t*-Δt, then i = 20. (4) Determine the types of weather phenomena related to cloud cover: (4.1) Read cloud cover data: based on time point t * Read cloud cover data from navigation data storage medium 1011; (4.2) Determine the type of cloud cover weather phenomenon: Determine the type of weather phenomenon i based on the cloud cover data, i∈{1,2,3,4,5}; (5) Save the type of weather phenomenon.

[0256] 2. DNN Fuzzy Recognition Learning Model and Training Method for Weather Phenomena

[0257] According to the national standard GB 18093-2000, weather phenomena are classified into 20 types, of which 11 require real-time visual observation by crew members. Therefore, there are m = 11 fuzzy classification concepts for artificially defined weather phenomena, denoted as a. i The i = 1, 2, ..., 11. The universe of discourse U is all images of water, and there exist 11 fuzzy sets, A. i i = 1, 2, ..., 11. See also Figure 24 The weather phenomenon DNN fuzzy recognition learning model is built upon a general multi-task DNN fuzzy recognition model, with single-task adjustments (number of tasks k=1). It consists of two parts: a feature extractor and a feature predictor. The feature predictor, the weather phenomenon recognition network head, adapts and predicts object features for weather phenomenon recognition. Each neuron outputs the probability / fuzzy set membership degree of weather phenomenon category i. i = 1, 2, ..., 11. The model training method is the same as the general object DNN fuzzy recognition model training method.

[0258] 3. Implementation Model of DNN Fuzzy Recognition for Weather Phenomenon Classification

[0259] See Figure 25 The weather phenomenon classification DNN fuzzy recognition implementation model is based on a pre-trained weather phenomenon classification DNN fuzzy recognition learning model, with the addition of a feature applicator containing a fuzzy recognition classification module. The fuzzy recognition classification module executes the multi-task DNN fuzzy recognition classification method of this invention.

[0260] The hydrological and meteorological data identified by this invention can be used for tasks such as automatically recording ship logs, multivariate regression machine learning of ship navigation and maneuvering performance, improving ocean wind and wave statistics, and ocean hydrological and meteorological monitoring and forecasting.

[0261] The general method for establishing a fuzzy model sample library, the general DNN fuzzy recognition model, the training method and the membership function expression form of the present invention, and the general DNN fuzzy recognition method can also be used in other fields that need to complete fuzzy recognition tasks. For example, the visibility fuzzy recognition DNN model and method of the present invention can be used for visibility recognition in car driving scenarios.

Claims

1. A method for training a DNN fuzzy recognition model, characterized in that The object DNN fuzzy recognition model extracts features from the input sample object, maps the features to obtain the classification probability distribution of the object, maps the classification probability to the membership degree of the fuzzy set in the fuzzy sample model library, and combines the sample label to calculate the loss function value of the classification probability distribution to train the model. After completing one generation of training, the model is adjusted by combining the sample label to calculate the loss function value of the fuzzy set closeness. If the result calculated by any loss function does not meet the preset loss value requirement, the samples in the fuzzy sample model library are randomly shuffled and reordered, and the iterative calculation is repeated until the results calculated by the two loss functions both meet the preset loss value requirement, and the model training is completed. The loss function includes: Object classification probability distribution closeness loss function ； wherein, is a cross-entropy; is a classification probability distribution of a sample label in the training set; is an output classification probability distribution; is a sample number; is a weight of a task , , is a task number, is a task number, , ; Also includes object fuzzy set closeness loss function ； wherein, is the weight of the th fuzzy set of the th task, is the fuzzy set number, , is the task number, is the number of tasks, , ; is the labeled membership of the th input sample of the th fuzzy set of the th task, is the sample number; is the output membership of the th input sample of the th fuzzy set of the th task; is the number of fuzzy sets of the th task; is the number of samples of the th fuzzy set of the th task. 2.The DNN blur identification model training method of claim 1, wherein The method for establishing the fuzzy sample model library is: collecting a large number of samples covering all recognition tasks, traversing all recognition tasks of all samples, identifying the object classification of each sample by manual recognition means, calculating the membership degree of each sample object to each recognition task, generating a membership degree vector and a classification probability distribution vector, and vector labeling each sample object to construct the fuzzy sample model library. The fuzzy sample model library includes sample sets divided according to deep learning conventions and fuzzy sets divided from the same sample for all recognition tasks.

3. A hydrometeor identification method characterized by The corresponding hydro-meteorological element DNN fuzzy recognition model is used to recognize the object, which includes the water image background, the target object around the ship, the sea state, the wave type, the wave height, the wave direction, the horizontal visibility, the cloud shape, the cloud amount, and the weather phenomenon element in the photographed picture. The membership degree of the corresponding element to the fuzzy set in the fuzzy sample model library is obtained, and the classification or grade of the corresponding element is determined according to the maximum membership degree principle, so as to recognize the hydro-meteorological element. The hydro-meteorological element DNN fuzzy recognition model is trained by the DNN fuzzy recognition model training method of any one of claims 1-2.

4. The hydrometeor discrimination method of claim 3, wherein The method includes the following steps: (1) reading the collected information, including the ship surrounding image picture, the sky picture, the radar sea surface point cloud data, and the ship navigation data; (2) preprocessing the collected information, including deleting or labeling the picture without recognition value, intercepting or arranging the valuable information, and adding time stamp and position stamp to the retained to-be-recognized information; (3) segmenting and detecting the water image background and the target object around the ship. If the segmentation and detection result is measurable, step (4) is executed, and if the segmentation and detection result is not measurable, step (6) is executed; (4) fog removal and target water area image recognition and horizontal visibility recognition, including fog removal processing and target water area picture interception; (5) if step (4) is executed, machine vision recognizes the sea state, wave type, and wave height data; (6) radar recognizes the wave type, calculates the wave direction angle, wave height, wave length, and wave period; (7) selects the final wave type and wave height data; (8) recognizes the cloud shape and calculates the cloud amount; (9) if step (8) is executed, the weather phenomenon is recognized.

5. The hydrometeor discrimination method of claim 4, wherein The method for selecting the final wave type and wave height data in step (7) is: reading the wave type and wave height data identified in steps (5) and (6), if the wave type and wave height data identified in step (5) are empty, selecting the wave type and wave height data identified in step (6); if the wave type and wave height data identified in steps (5) and (6) can be read at the same time, selecting the wave type data identified in step (5) and the wave height data identified in step (6); if the wave type and wave height data identified in step (6) are empty, selecting the wave type data identified in step (5) and determining the wave height according to the wave type category identified in step (5).

6. The hydrometeor discrimination method of claim 4, wherein In step (8), after outputting the cloud type of each pixel of the sky image, the sky pixel cloud vector is constructed, the sky pixel cloud vector is mapped to the cloud cloud amount vector through mapping and transformation, and finally the cloud cloud amount data conforming to the national standard is obtained.

7. The hydrometeor discrimination method of claim 4, wherein In step (6), the wave crest line and wave trough line information are obtained from the sea surface point cloud information by using the DBSCAN density clustering method, so as to calculate the wave height, wave direction angle, wave length and wave period.

Images