System and method for detecting and predicting harmful marine organism based on sensor fusion and deep learning

Abstract

One embodiment of the present invention provides a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system comprising: a sensor unit having one or more sensors for collecting marine environment data; a preprocessing unit that converts marine environment data collected by at least one sensor into a form suitable for analysis through normalization, filtering, and dimensionality reduction; a detection unit that detects and classifies harmful marine organisms from marine environment data converted by the preprocessing unit using multimodal data and a deep learning-based model; a state analysis unit that analyzes the state of harmful marine organisms by fusing marine environment data detected and classified by the detection unit; and a prediction unit that predicts the location and diffusion path of harmful marine organisms by analyzing temporal and spatial data from marine environment data using ConvLSTM and a Transformer model.

Description

Technology Field

[0001] The present invention relates to a sensor fusion and deep learning-based system for detecting and predicting the spread of harmful marine organisms and a prediction method thereof. More specifically, the invention relates to a sensor fusion and deep learning-based system for detecting and predicting the spread of harmful marine organisms and a prediction method thereof, which can simultaneously perform real-time detection and prediction of the spread of harmful marine organisms by fusing data from various sensors and using deep learning. Background Technology

[0002] Technical methods for the application of acoustic detection of marine organisms were established as early as the 1980s, and various acoustic surveys of economically important or ecosystem-important species are currently being conducted in the ocean. In most cases, acoustic detection in the ocean is carried out while the survey vessel moves, utilizing acoustic sensors installed on the ship. With the advancement of electronics in the late 1990s, acoustic detection methods for marine biological exploration also saw significant progress. Through the lightweighting of systems, increased data transmission capacity, and improved precision of acoustic sensors, the overall acoustic detection system became smaller, leading to enhanced accuracy.

[0003] Although the development of such acoustic systems has led to their application in various forms for the exploration of marine life in coastal and open waters, all surveys have been limited to one-time field surveys conducted by ships and periodic surveys conducted at regular intervals. Consequently, acoustic exploration for long-term and continuous detection of marine life at specific locations has been attempted only very restrictively by a few advanced maritime nations.

[0004] In Korea, marine life exploration in coastal and open waters has been conducted using acoustic sensors installed on ships since the early 1990s; however, there have been no instances of applying technology for continuous and long-term marine life detection using fixed acoustic systems. In particular, there are no cases of applying real-time, continuous detection technology for marine organisms that irregularly flow into national key industrial facilities along the coast, and such technology is currently needed.

[0005] Recently, jellyfish have been rapidly increasing due to factors such as global warming, a decline in natural predators, and an increase in coastal structures. As they wash ashore, they are causing significant damage. In particular, nuclear, thermal, and gas power plants built along the coast that operate using seawater as a coolant are suffering various forms of damage, including power plant shutdowns and reduced output, due to the unpredictable mass influx of not only jellyfish but also shrimp and anchovies. Despite being critical national infrastructure facilities where stability is paramount, these facilities are experiencing this damage. According to 2008 data from the Ministry of Land, Transport and Maritime Affairs, jellyfish alone cause tangible and intangible damages amounting to 58 billion won annually. Furthermore, beyond this direct economic damage, there is a significant psychological impact on the stable operation of these national infrastructure facilities. Prior art literature

[0006] Korean Registered Patent Publication No. 10-1046405 (Real-time marine organism inflow detection system and method of operation thereof, published July 5, 2011) The problem to be solved

[0007] The technical problem that the concept of the present invention aims to solve is to provide a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system and a prediction method thereof, which can fuse various sensor data to detect harmful marine organisms in real time, predict diffusion paths through deep learning algorithms, and provide effective countermeasures.

[0008] In addition, the present invention provides a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system and a prediction method that utilize sensor fusion and deep learning models to monitor harmful marine organisms occurring in the marine environment in real time and quickly solve problems through automatic alarm and response functions.

[0009] In addition, the present invention provides a sensor fusion and deep learning-based system for detecting and predicting the spread of harmful marine organisms, and a prediction method thereof, which maintains the balance of the marine ecosystem, reduces human or economic losses, and conserves the diversity of marine life. means of solving the problem

[0010] To achieve the aforementioned objective, a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system according to one embodiment of the present invention comprises: a sensor unit having one or more sensors that collect marine environment data; a preprocessing unit that converts the marine environment data collected by the at least one sensor into a form suitable for analysis through normalization, filtering, and dimensionality reduction; a detection unit that detects and classifies harmful marine organisms from the marine environment data converted by the preprocessing unit using multimodal data and a deep learning-based model; a state analysis unit that fuses the marine environment data detected and classified by the detection unit to analyze the state of the harmful marine organisms; and a prediction unit that analyzes temporal and spatial data from the marine environment data using ConvLSTM and a Transformer model to predict the location and diffusion path of the harmful marine organisms.

[0011] Here, the system may further include a visualization unit that visualizes the location and diffusion path of the harmful marine organism predicted by the prediction unit and the degree of impact caused by diffusion, and transmits them to a user terminal.

[0012] Here, the diffusion path of the aforementioned harmful marine organisms and the degree of impact caused by diffusion can be visualized in a map format.

[0013] Additionally, it may further include an appearance determination unit that uses an anomaly detection model to detect abnormal data where the location and diffusion path predicted by the prediction unit deviate from a preset location and preset pattern, and determines the presence and severity of the harmful marine organism, and a real-time notification unit that issues a real-time notification to a user terminal via alarm data according to the abnormal data determined by the appearance determination unit.

[0014] Here, a recommendation unit that recommends a response strategy based on the spread path and warning data of the harmful marine organism may be further included.

[0015] In addition, the above marine environment data may be one or more of ocean currents, water temperature, and salinity.

[0016] Meanwhile, a sensor fusion and deep learning-based method for detecting and predicting the spread of harmful marine organisms according to one embodiment of the present invention comprises: a step in which one or more sensors collect marine environment data; a step in which a preprocessing unit converts the collected marine environment data into a form suitable for analysis through normalization, filtering, and dimensionality reduction; a step in which a detection unit detects and classifies harmful marine organisms from the marine environment data converted by the preprocessing unit using multimodal data and a deep learning-based model; a step in which a state analysis unit fuses the detected and classified marine environment data to analyze the state of the harmful marine organisms; and a step in which a prediction unit analyzes temporal and spatial data from the marine environment data using ConvLSTM and a Transformer model to predict the location and spread path of the harmful marine organisms.

[0017] Here, the visualization unit may further include the step of visualizing the predicted location and diffusion path of the harmful marine organism and the degree of impact caused by diffusion and transmitting them to a user terminal.

[0018] Additionally, the method may include a step in which an appearance determination unit uses an anomaly detection model to detect abnormal data where the location and diffusion path predicted by the prediction unit deviate from a preset location and preset pattern, thereby determining the presence and severity of the harmful marine organism, and a step in which a real-time notification unit issues a real-time notification to a user terminal via alarm data according to the abnormal data.

[0019] Here, the recommendation unit may further include a step of recommending a response strategy based on the predicted spread path of the harmful marine organism and the warning data. Effects of the invention

[0020] According to the present invention, the purpose is to detect harmful marine organisms in real time by fusing various sensor data, predict the diffusion path through a deep learning algorithm, and provide effective countermeasures.

[0021] According to the present invention, by utilizing sensor fusion and deep learning models, harmful marine organisms occurring in the marine environment can be monitored in real time, and problems can be quickly resolved through automatic alarm and response functions.

[0022] According to the present invention, the balance of the marine ecosystem is maintained, human or economic losses are reduced, and marine biodiversity is preserved. Brief explanation of the drawing

[0023] FIG. 1 is a block diagram of a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating the construction and learning process of an object detection algorithm according to one embodiment of the present invention. Figure 3 is a graph showing the relationship between metric values ​​for an epoch according to one embodiment of the present invention. Figure 4 is a graph showing the relationship of Validation Loss for Epochs according to one embodiment of the present invention. FIG. 5 is a flowchart of a sensor fusion and deep learning-based method for detecting and predicting the spread of harmful marine organisms according to an embodiment of the present invention. Figure 6 is a performance evaluation table for fish, jellyfish, and salps for model learning and model verification of the present invention. Specific details for implementing the invention

[0024] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. The embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0025] Hereinafter, a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system and a prediction method according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0026] FIG. 1 is a block diagram of a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system according to one embodiment of the present invention.

[0027] Referring to FIG. 1, a sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system (1000) may include a sensor unit (100), a preprocessing unit (200), a detection unit (300), a state analysis unit (400), a prediction unit (500), a visualization unit (600), an appearance determination unit (700), a real-time notification unit (800), and a recommendation unit (900).

[0028] The sensor unit (100) collects marine environment data and may be at least one. The sensor unit (100) may be equipped with at least one sensor (100a, 100b, 100c). Here, the marine environment data may be at least one of ocean currents, water temperature, salinity, pH, dissolved oxygen, and nutrients, and may also be collected from a satellite. The at least one sensor (100a, 100b, 100c) may be an environmental sensor, an image sensor, etc. The image sensor may collect biological data. By utilizing satellite, aerial, and underwater images, abnormal phenomena in the marine monitoring area can be detected and fluctuations in biological communities can be identified. By fusing this data, the condition of harmful marine organisms can be comprehensively evaluated, and changes in the marine ecosystem can be monitored in real time from the collected data.

[0029] The preprocessing unit (200) converts marine environment data collected by at least one sensor into a form suitable for analysis through normalization, filtering, and dimensionality reduction. Through normalization and standardization, the data range for each sensor is consistently adjusted, and the data quality is improved through noise removal and filtering. Additionally, by utilizing dimensionality reduction techniques such as PCA (Principal Component Analysis) or t-SNE, the complexity of the data can be reduced and it can be converted into a form suitable for analysis. In this process, meaningful features of the data are extracted to increase the efficiency of model training and to enable a clear understanding of the correlations between the data.

[0030] The detection unit (300) detects and classifies harmful marine organisms from marine environment data converted by the preprocessing unit using multimodal data and a deep learning-based model. Specifically, the detection unit (300) classifies harmful marine organisms using a Convolutional Neural Network (CNN) model such as ResNet, Inception, or EfficientNet, and detects harmful marine organisms within an image and specifies the detection location using an object detection algorithm such as YOLOv5 or Faster R-CNN. Preferably, the deep learning-based object detection model may be YOLOv5 (hereinafter referred to as YOLO regardless of version), which is a version of YOLO released in June 2020 that has improved object detection accuracy by more than 10% compared to the existing YOLO, has improved processing speed, and has a lightweight model size. Such a deep learning model enables the detection of harmful marine organisms with high accuracy despite the complex characteristics of the marine environment and allows for the effective recognition of organisms of various shapes and sizes.

[0031] In this invention, objects are detected based on underwater images, and marine organisms with a high probability of entering the water intake are automatically detected in real time and utilized as basic data to determine whether marine organisms have entered. This can be implemented through computer vision-based technology called object detection.

[0032] Object detection is a field of computer vision that identifies desired objects in images by distinguishing them from the background. To accurately detect objects, boundaries must be established within the image and their associations compared with attribute information representing each object; in this process, object detection algorithms, which are artificial intelligence techniques, are applied.

[0033] Object detection using artificial intelligence is based on convolutional neural networks, and CNNs have higher learning efficiency compared to other algorithms because they can learn while maintaining a 3D data format. In this invention, an image-based marine organism detection model was constructed by applying a CNN-based YOLO (You Only Look Once) model, which is known to have the best real-time object detection and class classification performance.

[0034] The detection unit (300) utilizes a Recurrent Neural Network (RNN) and a Long Short-Term Memory (LSTM) to analyze temporal changes in environmental data and predict the likelihood of occurrence of harmful marine organisms. At this time, the prediction accuracy is improved by applying an attention mechanism to focus on important time intervals. This time series analysis enables the prediction of periods and regions with a high probability of occurrence of harmful marine organisms in advance, thereby allowing for preventive measures to be taken. The artificial intelligence models required for sensor fusion and data integration related to the detection of harmful marine organisms are as follows.

[0035] 1) Multimodal Deep Learning

[0036] Data from each sensor is processed individually through a parallel network structure and then integrated at the final layer to comprehensively analyze the status of harmful marine organisms. Collaborative learning techniques are employed to learn correlations between various data points, thereby improving model performance. This fusion approach overcomes the limitations of single-sensor data and leverages the strengths of each sensor to derive more accurate analysis results.

[0037] 2) Graph Neural Network (GNN)

[0038] By modeling the interactions between monitoring sites as graphs, the diffusion patterns of harmful marine organisms are analyzed, and predictions considering spatial relationships are performed. Through graph neural networks, complex interactions in the marine environment can be analyzed, and the movement and diffusion paths of harmful marine organisms can be precisely predicted.

[0039] 3) Generative Adversarial Network (GAN)

[0040] Data diversity is enhanced by constructing an accurately labeled dataset with the help of experts and generating synthetic data using Generative Adversarial Networks (GANs). This improves the model's generalization ability and enables it to maintain high performance in various situations. Data augmentation compensates for insufficient data and prevents overfitting issues that may occur during model training.

[0041] 4) Loss Function and Optimization Algorithm

[0042] Custom loss functions such as Focal Loss are used to address class imbalance, and AdamW and Lookahead Optimizers are employed to improve training speed and stability. These optimization algorithms accelerate the model's convergence and ensure stable training even in complex ocean environments.

[0043] 5) GRAD-CAM

[0044] GRAD-CAM visualizes key aspects of the model's decision-making process and utilizes SHAP values ​​to quantitatively evaluate how each input feature contributes to the prediction. This enhances the reliability of the model's prediction results and supports users in understanding and utilizing the findings.

[0045] The state analysis unit (400) fuses marine environment data detected and classified by the detection unit (300) to analyze the state of harmful marine organisms.

[0046] The prediction unit (500) predicts the location and spread path of harmful marine organisms by analyzing temporal and spatial data from marine environment data using ConvLSTM and Transformer models. Specifically, the prediction unit (500) predicts the spread pattern of harmful marine organisms by simultaneously considering spatial and temporal features using ConvLSTM, and improves prediction accuracy by effectively processing long time series data through Transformer models. This model integrates data such as ocean currents, temperature, and nutrients to precisely predict the spread path of harmful organisms, thereby enabling the planning of preventive measures in advance.

[0047] The prediction unit (500) simulates the behavior of individual organisms through agent-based modeling and analyzes collective diffusion patterns.

[0048] The prediction unit (500) numerically simulates biological processes using differential equations through mathematical modeling and evaluates the impact of the spread of harmful marine organisms under various scenarios. These simulations provide important information for policymakers or researchers to formulate optimal response strategies.

[0049] The visualization unit (600) visualizes the location and diffusion path predicted by the prediction unit (500) and the degree of influence caused by diffusion, and transmits them to the user terminal.

[0050] Here, the impact of the spread of harmful marine organisms refers to changes in ocean currents, water temperature, salinity, pH, dissolved oxygen, and nutrients resulting from the spread of harmful marine organisms, and such changes in ocean currents, water temperature, salinity, pH, dissolved oxygen, and nutrients can cause casualties or property damage. For example, changes in dissolved oxygen can cause fish to die in aquaculture farms, and changes in water temperature can cause jellyfish to proliferate, which can lead to casualties caused by jellyfish.

[0051] Through this, users can clearly identify the occurrence and spread of harmful marine organisms and quickly determine corresponding response strategies. Spread pathways and the impact of such spread can be visualized in map form, and the visualized results can be utilized for establishing response strategies and making policy decisions.

[0052] The appearance determination unit (700) uses an anomaly detection model to detect abnormal data that deviates from the normal pattern of the location and diffusion path predicted by the prediction unit (500) and determines whether harmful marine organisms have appeared and the degree of severity. Here, an autoencoder may be used as the anomaly detection model. According to one embodiment, the appearance determination unit (700) can determine whether harmful marine organisms have appeared and the degree of severity by detecting abnormal data where the location and diffusion path predicted by the prediction unit (500) deviate from the preset location and preset pattern.

[0053] Here, the pre-set locations and pre-set patterns are the patterns of appearance locations and diffusion paths of harmful marine organisms in previous years, and the anomalous data may be data in which the similarity to these pre-set locations and pre-set patterns falls outside a certain range.

[0054] In addition, severity is a numerical representation of the stages based on cases where the predicted population of harmful marine organisms exceeds a certain level compared to the population of previous years.

[0055] The real-time notification unit (800) issues a real-time notification to the user terminal via alarm data based on abnormal data determined by the appearance determination unit (700). The alarm data is notified through a multi-stage alarm system. The multi-stage alarm system is constructed based on environmental variables and the density detection amount of the target harmful marine organism, enabling early detection of the occurrence of harmful marine organisms and prompt action to take necessary countermeasures. When an alarm is issued, the user receives an immediate notification and, if necessary, can execute a response strategy in real time.

[0056] The recommendation unit (900) recommends an optimal response strategy based on the predicted spread path and alarm data. At this time, physical response is possible through an automated system such as an autonomous robot or drone, and the response plan can be visualized on a dashboard.

[0057] The sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system described above has the following characteristics.

[0058] 1) Provision of APIs and Interfaces: Through the data sharing platform, APIs are provided to share data and collaborate with other research institutions or government agencies, and web or mobile app interfaces that are easily accessible to users can be developed. This provides an environment where researchers and policymakers can utilize data and collaborate in real time.

[0059] 2) Model Update and Retraining: The model is continuously updated using real-time input data through online learning, and an automated retraining pipeline is established to automatically handle everything from data collection to model deployment. This continuous improvement process maintains the model's up-to-date status and enables rapid adaptation to the changing marine environment.

[0060] 3) Integration of Cloud and Edge Computing: By integrating cloud and edge computing, it enables the real-time processing and analysis of large-scale data. Cloud infrastructure supports high-performance computation for data storage and analysis, while edge computing facilitates real-time inference and decision-making in the field, thereby enhancing system responsiveness. Through this, it provides an integrated solution capable of processing data collected from marine sites in real time and utilizing the results immediately.

[0061] FIG. 2 is a flowchart illustrating the construction and learning process of an object detection algorithm according to an embodiment of the present invention. FIG. 3 is a graph showing the relationship between metric values ​​for an epoch according to an embodiment of the present invention, and FIG. 4 is a graph showing the relationship between Validation Loss for an epoch according to an embodiment of the present invention.

[0062] Referring to FIG. 2, the object detection algorithm construction and training process involves preparing an image training dataset of the desired detection target (S10), loading a YOLO model (S20), and training a deep learning model (S30). After S30, a loss function is calculated (S40) to determine whether the model weights have been updated (S50). After S50, if the model weights have been updated, the final weights are calculated as the final result and saved (S60). If the model weights have not been updated in S50, S30 is performed.

[0063] Real-time automatic detection is possible by combining the final weights calculated in S60 with underwater imaging equipment. In the future, to continuously improve detection performance, a new image learning dataset will be collected and preprocessed to repeatedly perform additional artificial intelligence learning, thereby maintaining an optimized algorithm.

[0064] Meanwhile, in the present invention, for model training and model validation, a training dataset of approximately 152,000 augmented images consisting of three classes of fish, jellyfish, and salps was trained on YOLOv5. 80% of the total images were configured as a training dataset for training, 10% as a validation dataset for verifying the training, and the remaining 10% as a test dataset for final verification.

[0065] The initial training parameters were set to run for a maximum of 10,000 epochs, and an Early Stopping instruction was configured to terminate training if no performance improvement is observed for 50 or more iterations. Additionally, the Batch Size, representing the maximum number of images the model reads at once, was set to 144 to account for GPU performance, and Stochastic Gradient Descent (SGD) was applied as the optimizer.

[0066] The model was trained a total of 247 times and terminated early. The model's average precision on the validation data was calculated to be 0.931, the recall 0.881, and the mAP (mean Average Precision, a representative metric for evaluating the performance of an object detection model) 0.948.

[0067] In addition, the mAP for each class was 0.97 for fish, 0.97 for jellyfish, and 0.91 for salps, with values ​​exceeding 0.9 (90%) for all classes, confirming excellent performance. Among these, salps showed a recall of 0.840, which is slightly lower than that of fish and jellyfish. This is attributed to the fact that the AI ​​model experienced some difficulty in identification due to the relatively small amount of training data and the fact that the individuals did not have a standardized form but existed in groups in chains, as floating debris, or in clumped shapes (see Figures 3, 4, and 6).

[0068] FIG. 5 is a flowchart of a sensor fusion and deep learning-based method for detecting and predicting the spread of harmful marine organisms according to an embodiment of the present invention.

[0069] Referring to FIG. 5, the sensor fusion and deep learning-based method for detecting and predicting the spread of harmful marine organisms may include S100 to S800.

[0070] First, at least one sensor (100a, 100b, 100c) collects marine environment data (S100). Here, the marine environment data may be at least one of ocean currents, water temperature, salinity, pH, dissolved oxygen, and nutrients, and may also be collected from a satellite. At least one sensor (100a, 100b, 100c) may be an environmental sensor, an image sensor, etc. The image sensor may collect biological data. By utilizing satellite, aerial, and underwater images, anomalies in the marine monitoring area can be detected and fluctuations in the biological community can be identified. By fusing this data, the condition of harmful marine organisms can be comprehensively evaluated, and changes in the marine ecosystem can be monitored in real time from the collected data.

[0071] After S100, the preprocessing unit (200) converts the collected marine environment data into a form suitable for analysis through normalization, filtering, and dimensionality reduction (S200). Through normalization and standardization, the data range for each sensor of the collected marine environment data is consistently adjusted, and the data quality is improved through noise removal and filtering. Furthermore, by utilizing dimensionality reduction techniques such as PCA (Principal Component Analysis) or t-SNE, the complexity of the data can be reduced and it can be converted into a form suitable for analysis. In this process, meaningful features of the data are extracted to increase the efficiency of model training and to enable a clear understanding of the correlations between the data.

[0072] After S200, the detection unit (300) detects and classifies harmful marine organisms from the marine environment data converted by the preprocessing unit (200) using multimodal data and a deep learning-based model (S300). Specifically, the detection unit (300) classifies harmful marine organisms using Convolutional Neural Network (CNN) models such as ResNet, Inception, and EfficientNet, and detects harmful marine organisms within an image and designates the detection location using object detection algorithms such as YOLOv5 and Faster R-CNN. These deep learning models enable the detection of harmful marine organisms with high accuracy despite the complex characteristics of the marine environment and allow for the effective recognition of organisms of various shapes and sizes.

[0073] After S300, the state analysis unit (400) fuses the detected and classified marine environment data to analyze the state of harmful marine organisms (S400).

[0074] After S400, the prediction unit (500) uses ConvLSTM and a Transformer model to analyze temporal and spatial data from marine environment data to predict the location and diffusion path of harmful marine organisms (S500). Specifically, the prediction unit (500) uses ConvLSTM to simultaneously consider spatial and temporal characteristics to predict the diffusion pattern of harmful marine organisms, and uses a Transformer model to effectively process long time-series data to increase prediction accuracy. This model integrates data such as ocean currents, temperature, and nutrients to precisely predict the diffusion path of harmful organisms, thereby allowing preventive measures to be planned in advance. Although S500 is performed after S400 for convenience, it can be performed after S100 independently of S200 to S400.

[0075] Meanwhile, after S500, the visualization unit (600) may further include a step of visualizing the predicted location, the diffusion path, and the degree of impact caused by diffusion and transmitting them to a user terminal. Through this, the user can clearly identify the occurrence and diffusion status of harmful marine organisms and quickly determine a response strategy accordingly. The diffusion path and the degree of impact caused by diffusion can be visualized in a map format, and the visualized results can be utilized for establishing response strategies and policy decisions.

[0076] Furthermore, after S500, the procedure may further include a step in which an appearance determination unit (700) detects abnormal data in which the location and diffusion path predicted by the prediction unit (500) deviate from a preset location and preset pattern, and determines whether harmful marine organisms have appeared and the severity thereof, and a step in which a real-time notification unit (800) issues a real-time notification to a user terminal through alarm data according to the abnormal data.

[0077] Here, the pre-set locations and pre-set patterns are the patterns of appearance locations and diffusion paths of harmful marine organisms in previous years, and the anomalous data may be data in which the similarity to these pre-set locations and pre-set patterns falls outside a certain range.

[0078] In addition, severity is a numerical representation of the stages based on cases where the predicted population of harmful marine organisms exceeds a certain level compared to the population of previous years.

[0079] Here, an autoencoder can be used as the anomaly detection model. Alert data is notified through a multi-stage alert system. This multi-stage alert system is constructed based on environmental variables and the detected density of target harmful marine organisms, enabling the early detection of such occurrences and the rapid implementation of necessary response measures. Upon issuance of an alert, users receive immediate notifications and can execute response strategies in real time if necessary.

[0080] Meanwhile, after S500, the recommendation unit (900) may further include a step of recommending an optimal response strategy based on the predicted diffusion path and alarm data. At this time, physical response is possible through an automated system such as an autonomous robot or drone, and the response plan can be visualized on a dashboard.

[0081] The embodiments of the present invention described above have been explained with reference to the embodiments illustrated in the drawings for the sake of understanding, but this is merely illustrative and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the appended claims. Explanation of the symbols

[0082] 100: Sensor unit 200: Preprocessing unit 300: Detection Unit 400: Status Analysis Unit 500: Prediction Section 600: Visualization Section 700: Appearance Judgment Unit 800: Real-time Notification Unit 900: Recommendation Department 1000: Harmful Marine Organism Detection and Spread Prediction System

Claims

Claim 1 A sensor unit (100) equipped with an environmental sensor and an image sensor for collecting marine environment data; a preprocessing unit (200) that converts the marine environment data collected by the environmental sensor and the image sensor, respectively, into a form suitable for analysis through normalization, filtering, and dimensionality reduction; a detection unit (300) that detects and classifies harmful marine organisms from the marine environment data converted by the preprocessing unit (200) using multimodal data and a deep learning-based model; and a state analysis unit (400) that fuses the marine environment data detected and classified by the detection unit (300) to analyze the state of the harmful marine organisms; wherein the marine environment data collected by the sensor unit (100) includes pH, dissolved oxygen, nutrients, ocean currents, water temperature, and salinity data collected from the environmental sensor, and satellite, aerial, and underwater image data collected from the image sensor; and the state analysis unit (400) includes the types of harmful marine organisms detected and classified by the detection unit (300) and the marine environment data collected from the environmental sensor, such as pH, dissolved oxygen, and nutrients. The system further includes a prediction unit (500) that fuses salt, ocean current, water temperature, and salinity data to comprehensively analyze the state of the harmful marine organism, and predicts the location and diffusion path of the harmful marine organism by analyzing temporal and spatial data from the marine environment data of the sensor unit (100) using ConvLSTM and Transformer models, simulates the behavior of individual organisms and analyzes collective diffusion patterns through agent-based modeling, numerically simulates biological processes using differential equations through mathematical modeling, and evaluates the impact of the diffusion of the harmful marine organism according to multiple scenarios; and a visualization unit (600) that visualizes the location and diffusion path of the harmful marine organism predicted by the prediction unit (500) and the degree of impact caused by diffusion and transmits them to a user terminal.A sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system further comprising: an appearance determination unit (700) that detects abnormal data through an autoencoder where the location and diffusion path predicted by the prediction unit (500) deviate from a preset location and preset pattern, and determines the appearance and severity of the harmful marine organism; a real-time notification unit (800) that issues a real-time notification to a user terminal through alarm data according to the abnormal data determined by the appearance determination unit (700); and a recommendation unit (900) that recommends a response strategy based on the diffusion path of the harmful marine organism, the appearance and severity of the harmful marine organism, and the alarm data. Claim 2 delete Claim 3 A sensor fusion and deep learning-based harmful marine organism detection and diffusion prediction system according to claim 1, wherein the diffusion path of the harmful marine organism and the degree of impact caused by diffusion are visualized in a map format. Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 The method comprises: a step in which a sensor unit (100) collects marine environment data using an environmental sensor and an image sensor; a step in which a preprocessing unit (200) converts the collected marine environment data into a form suitable for analysis through normalization, filtering, and dimensionality reduction; a step in which a detection unit (300) detects and classifies harmful marine organisms from the marine environment data converted by the preprocessing unit (200) using multimodal data and a deep learning-based model; and a step in which a state analysis unit (400) fuses the detected and classified marine environment data to analyze the state of the harmful marine organisms; wherein the marine environment data collected by the sensor unit (100) includes pH, dissolved oxygen, nutrients, ocean currents, water temperature, and salinity data collected from the environmental sensor, and satellite, aerial, and underwater image data collected from the image sensor; and the step in which the state analysis unit (400) analyzes the state of the harmful marine organisms includes the type of harmful marine organism detected and classified by the detection unit (300) and the marine environment data collected from the environmental sensor, such as pH and dissolved oxygen The method further includes the step of comprehensively analyzing the state of the harmful marine organism by fusing oxygen, nutrient, ocean current, water temperature, and salinity data, and the prediction unit (500) predicting the location and diffusion path of the harmful marine organism by analyzing temporal and spatial data from the marine environment data of the sensor unit (100) using ConvLSTM and Transformer models, simulating the behavior of individual organisms and analyzing collective diffusion patterns through agent-based modeling, numerically modeling biological processes using differential equations through mathematical modeling, and evaluating the impact of the diffusion of the harmful marine organism according to multiple scenarios; and the step of the visualization unit (600) visualizing the predicted location and diffusion path of the harmful marine organism and the degree of impact caused by diffusion and transmitting them to a user terminal.A method for detecting and predicting the spread of harmful marine organisms based on sensor fusion and deep learning, further comprising: a step in which an appearance determination unit (700) detects abnormal data through an autoencoder in which the location and spread path predicted by the prediction unit (500) deviate from a preset location and preset pattern, and determines whether the harmful marine organism has appeared and its severity; a step in which a real-time notification unit (800) issues a real-time notification to a user terminal through alarm data according to the abnormal data; and a step in which a recommendation unit (900) recommends a response strategy based on the spread path of the harmful marine organism, whether the harmful marine organism has appeared and its severity, and the alarm data. Claim 8 delete Claim 9 delete Claim 10 delete

Images