A real-time monitoring and modeling system and method for marine environment based on multimodal fusion

Abstract

The present invention provides a real-time monitoring and modeling system and method for marine environment based on multimodal fusion. The system includes: a data acquisition module for collecting marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generating multimodal data after preprocessing; a multimodal fusion module for registering the multimodal data in terms of time and space and calculating the weights of various data sources in real time, and fusing the registered multimodal data based on the weights to obtain fused marine environment data; a real-time three-dimensional modeling module for constructing a real-time three-dimensional model of the marine environment by using three-dimensional modeling technology based on the fused marine environment data, and dynamically updating the real-time three-dimensional model of the marine environment according to new multimodal data. By integrating satellite remote sensing, buoy sensors, and public crowdsourcing data, and using a dynamic weight allocation algorithm, real-time three-dimensional modeling of multi-source data of space-air-ground-sea is realized, so as to comprehensively and accurately reflect the marine environment status.

Description

Technical Field

[0001] The present invention relates to the technical field of marine monitoring, and particularly relates to a real-time monitoring and modeling system and method for marine environment based on multi-modal fusion. Background Art

[0002] The marine environment is complex and changeable. Accurate and real-time monitoring of it is of crucial significance for many aspects such as marine resource development, marine ecological protection, and safety of maritime transportation. Traditional marine environment monitoring means mainly include satellite remote sensing monitoring, buoy sensor monitoring, etc. Satellite remote sensing can obtain information on large areas of the ocean region from a macroscopic scale, such as sea surface temperature, sea color, etc.; buoy sensors can measure parameters such as seawater temperature, salinity, dissolved oxygen, etc. in real time at specific locations. However, a single monitoring means has limitations. The resolution of satellite remote sensing data is limited, and it is difficult to accurately capture local small-scale marine environment changes; the monitoring range of buoy sensors is limited by their deployment locations and cannot comprehensively cover the vast ocean region. In addition, with the improvement of the public's awareness of marine protection, public crowdsourcing data (such as marine observation information provided by divers, fishermen, etc.) contains rich local marine environment details but has not been fully and effectively utilized. Therefore, how to integrate multi-source data to achieve comprehensive, real-time, and accurate monitoring and modeling of the marine environment has become an urgent problem to be solved. Summary of the Invention

[0003] The present invention provides a real-time monitoring and modeling system and method for marine environment based on multi-modal fusion. By integrating satellite remote sensing, buoy sensor, and public crowdsourcing data, and using a dynamic weight allocation algorithm, real-time three-dimensional modeling of multi-source data from space-air-ground-sea is realized, so as to comprehensively and accurately reflect the marine environment conditions.

[0004] The present invention provides a real-time monitoring and modeling system for marine environment based on multi-modal fusion, including:

[0005] A data acquisition module, configured to collect marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multi-modal data after preprocessing;

[0006] A multi-modal fusion module, configured to register the multi-modal data in terms of time and space, and calculate the weights of various data sources in real time, and fuse the registered multi-modal data based on the weights to obtain fused marine environment data;

[0007] A real-time three-dimensional modeling module, configured to construct a real-time three-dimensional model of the marine environment by using three-dimensional modeling technology based on the fused marine environment data, and dynamically update the real-time three-dimensional model of the marine environment according to new multi-modal data.

[0008] Preferably, in a marine environment real-time monitoring modeling system based on multimodal fusion, the data acquisition module includes:

[0009] The satellite remote sensing data acquisition unit is used to receive the ocean remote sensing image data sent by the satellite and send it to the data preprocessing unit, and analyze the preprocessed ocean remote sensing image data to generate ocean satellite remote sensing data;

[0010] The buoy sensor data acquisition unit is used to establish communication connections with buoy sensors distributed throughout the ocean, collect seawater data measured by various sensors on the buoy in real time, and generate buoy sensor data;

[0011] A public crowdsourcing data collection unit, used to collect ocean-related data on the Internet of Things based on big data technology to obtain public crowdsourcing data;

[0012] The data preprocessing unit is used to process the ocean remote sensing image data, buoy sensor data and public crowdsourcing data respectively based on corresponding preprocessing rules to generate multimodal data.

[0013] Preferably, in a marine environment real-time monitoring modeling system based on multimodal fusion, the data preprocessing unit includes:

[0014] The satellite remote sensing data preprocessing subunit is used to perform radiation correction, atmospheric correction and geometric correction on the ocean remote sensing image to obtain preprocessed ocean remote sensing image data;

[0015] The buoy sensor data preprocessing subunit is used to perform denoising and filtering on the seawater data collected by the buoy sensor;

[0016] The public crowdsourcing data preprocessing subunit is used to verify the authenticity and classify the ocean-related data uploaded by the public on the Internet of Things to obtain public crowdsourcing data.

[0017] Preferably, in a marine environment real-time monitoring modeling system based on multimodal fusion, the public crowdsourcing data preprocessing subunit includes:

[0018] The key information judgment subunit is used to obtain the release time, release location and upload copy information of the publicly uploaded ocean-related data;

[0019] Performing semantic recognition and keyword extraction on the uploaded document information, and judging whether the uploaded document contains the ocean geographic location information of the ocean-related data uploaded by the work according to the semantic results and the keyword extraction results;

[0020] The first data verification subunit is configured to, if the uploaded text contains the marine geographical location information of the marine-related data uploaded for work, predict the actual observation time interval of the marine-related data based on the distance difference between the published locations of the marine geographical location information and in combination with the publication time;

[0021] Based on the actual observation time interval and the marine geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics;

[0022] Correspondingly, in combination with the semantic recognition result and the text keywords, determine the main phenomenon described by the user;

[0023] According to the main phenomenon described by the user, extract the key features of the phenomenon from the publicly uploaded marine-related data. Based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded marine-related data is real data;

[0024] The second data verification subunit is configured to, if the uploaded text does not contain the marine geographical location information of the marine-related data uploaded for work, extract the marine geological features from the publicly uploaded marine-related data, perform relevant marine retrievals based on the marine geological features, and determine whether there is a suspected target ocean in the preset marine information database;

[0025] If not, determine that the publicly uploaded marine-related data is fake data;

[0026] If so, obtain the key features of the phenomenon and the corresponding phenomenon occurrence area features, and based on the local environmental data of the geographical location corresponding to the suspected target ocean, in combination with the marine biological activity characteristics and the marine activity characteristics, infer whether the phenomenon described by the publicly uploaded marine-related data may exist;

[0027] If so, obtain the geographical location information corresponding to the suspected ocean and send it to the first data verification subunit for secondary determination;

[0028] Otherwise, determine that the publicly uploaded marine-related data is fake data.

[0029] Preferably, in a real-time monitoring and modeling system for the marine environment based on multi-modal fusion, the multi-modal fusion module includes:

[0030] The data registration unit is configured to register the multi-modal data in terms of time and space based on the geographical information system technology, determine the corresponding relationship between the time and position coordinates among different multi-modal data, and obtain the registered multi-modal data;

[0031] The dynamic weight allocation unit is configured to establish a multi-factor decision-making model based on the analytic hierarchy process and the fuzzy comprehensive evaluation method to obtain the actual needs of the user;

[0032] Analyze the actual requirements to obtain the user's focus on ocean monitoring. Based on the focus on ocean monitoring, determine the user's preferred ocean factors. Based on the user's preferred ocean factors, use a multi-factor decision-making model to adjust the data weights of different multi-modal data to obtain the current weight allocation result of the multi-modal data;

[0033] A multi-modal data fusion unit for fusing the registered multi-modal data according to the current weight allocation result.

[0034] Preferably, in a real-time ocean environment monitoring and modeling system based on multi-modal fusion, the real-time three-dimensional modeling module includes:

[0035] A three-dimensional model construction unit for constructing a real-time three-dimensional model of the ocean environment based on the fused multi-modal data using three-dimensional modeling technology;

[0036] A model update and visualization unit for receiving new multi-source data in real time, recalculating weights according to the dynamic weight allocation algorithm, and performing data fusion.

[0037] Preferably, in a real-time ocean environment monitoring and modeling system based on multi-modal fusion, the three-dimensional model construction unit includes:

[0038] A model layering sub-unit for dividing the ocean three-dimensional model into multiple layers based on the ocean standard model, and obtaining the ocean characterization data corresponding to each layer, as well as the change characteristics of marine biological species and the seawater decomposition data characteristics at the boundaries of each layer;

[0039] A model construction sub-unit for classifying the fused multi-modal data obtained based on the ocean characterization data corresponding to each layer to obtain multi-modal data sets corresponding to each layer respectively;

[0040] Adjust the ocean standard model based on the multi-modal data sets to determine the ocean three-dimensional model of each layer corresponding to the current ocean, and obtain a primary ocean model;

[0041] At the same time, based on the change characteristics of marine biological species and the seawater decomposition data characteristics at the boundaries of each layer, determine the decomposition characteristics of each layer of the ocean three-dimensional model corresponding to the current ocean;

[0042] Fuse each layer corresponding to the primary ocean model based on the decomposition characteristics of each layer of the current ocean to obtain the real-time three-dimensional model of the current corresponding ocean environment.

[0043] The present invention provides a real-time ocean environment monitoring and modeling method based on multi-modal fusion, including:

[0044] Step 1: Collect ocean satellite remote sensing data, buoy sensor data, and public crowdsourcing data. After preprocessing, generate multimodal data;

[0045] Step 2: Register the multimodal data in terms of time and space, and calculate the weights of various data sources in real time. Based on the weights, fuse the registered multimodal data to obtain fused ocean environment data;

[0046] Step 3: Based on the fused ocean environment data, use three-dimensional modeling technology to construct a real-time three-dimensional model of the ocean environment, and dynamically update the real-time three-dimensional model of the ocean environment according to the new multimodal data.

[0047] Preferably, in a method for real-time monitoring and modeling of the ocean environment based on multimodal fusion, verifying the authenticity of public crowdsourcing data includes:

[0048] Step A: Obtain the release time, release location, and uploaded text information of the ocean-related data uploaded publicly;

[0049] Perform semantic recognition and keyword extraction on the uploaded text information. According to the semantic result and keyword extraction result, determine whether the uploaded text contains the ocean geographical location information of the ocean-related data uploaded publicly;

[0050] Step B: If the uploaded text contains the ocean geographical location information of the ocean-related data uploaded publicly, then based on the distance difference between the ocean geographical location information and the released location, combined with the release time, predict the actual observation time interval of the ocean-related data;

[0051] Based on the actual observation time interval and the ocean geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics;

[0052] Correspondingly, combine the semantic recognition result and the text keywords to determine the main phenomenon described by the user;

[0053] According to the main phenomenon described by the user, extract the key features of the phenomenon from the publicly uploaded ocean-related data. Based on comparing the key features of the phenomenon with the multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded ocean-related data is real data;

[0054] Step C: If the uploaded text does not contain the ocean geographical location information of the ocean-related data uploaded publicly, then extract the ocean geological features from the publicly uploaded ocean-related data. Based on the ocean geological features, conduct relevant ocean retrievals to determine whether there is a suspected target ocean in the preset ocean information database;

[0055] If not, determine that the publicly uploaded ocean-related data is false data;

[0056] If it exists, obtain the key features of the phenomenon and the corresponding regional features of the phenomenon occurrence area. Based on the local environmental data of the geographical location corresponding to the suspected target ocean, combined with the characteristics of marine biological activities and marine activities, speculate whether the phenomenon described in the publicly uploaded ocean-related data may exist;

[0057] If so, obtain the geographical location information corresponding to the suspected ocean and send it to step B for secondary determination;

[0058] Otherwise, determine that the publicly uploaded ocean-related data is false data.

[0059] Preferably, in a real-time monitoring and modeling method for marine environment based on multi-modal fusion, step 3 includes:

[0060] Divide the three-dimensional ocean model into multiple levels based on the ocean standard model, and obtain the corresponding ocean characterization data for each level, as well as the characteristics of the change in marine biological species and the characteristics of seawater decomposition data at the boundaries of each level;

[0061] Classify the obtained fused multi-modal data based on the ocean characterization data corresponding to each level to obtain multi-modal data sets respectively determined for each level;

[0062] Adjust the ocean standard model based on the multi-modal data set to determine the three-dimensional ocean model for each level corresponding to the current ocean, and obtain a primary ocean model;

[0063] At the same time, based on the characteristics of the change in marine biological species and the characteristics of seawater decomposition data at the boundaries of each level, and the three-dimensional ocean model for each level corresponding to the current ocean, determine the decomposition characteristics of each level of the current ocean;

[0064] Fuse the corresponding levels of the primary ocean model based on the decomposition characteristics of each level of the current ocean to obtain a real-time three-dimensional model of the current corresponding marine environment.

[0065] Compared with the prior art, the present invention has at least the following beneficial effects:

[0066] The present invention integrates satellite remote sensing, buoy sensors and public crowdsourcing data, collects information from multiple dimensions of air, space, land and sea, fills the blind spots of traditional monitoring means, and the diversified data collection method enables the system to quickly capture the dynamic changes of the marine environment. Then, through the registration of multi-modal data in time and space, it is ensured that data from different sources can be compared and analyzed in the same spatio-temporal framework. The dynamic weight allocation algorithm can dynamically adjust the model according to different monitoring indicators according to the needs of marine research. For example, when monitoring the sea surface temperature, a higher weight is given to satellite remote sensing data; when monitoring the local marine biodiversity, the weight of public crowdsourcing data is increased. Based on the weights, the registered data is fused, giving full play to the advantages of each data source, effectively making up for the limitations of a single data source, significantly improving the accuracy of marine environment monitoring, making the monitoring results more in line with the actual marine environment conditions. Finally, a real-time three-dimensional model is constructed based on the fused marine environment data, transforming the abstract marine environment data into an intuitive three-dimensional visualization scene. The model covers multiple levels such as the sea surface, the interior of the water body and the seabed topography, vividly showing the three-dimensional spatial distribution characteristics of the marine environment, enabling a quick grasp of the overall picture of the marine environment, providing an intuitive and efficient information display method for marine scientific research and management decision-making, and receiving new multi-modal data in real time, recalculating weights and fusing data according to the dynamic weight allocation algorithm, and updating the three-dimensional model in a timely manner, enabling the model to closely track the real-time changes of the marine environment, providing a powerful tool for the dynamic monitoring and scientific management of the marine environment.

[0067] Other features and advantages of the present invention will be described in the following specification, and in part will be obvious from the specification, or can be learned by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures specifically pointed out in this application document.

[0068] The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments. Description of the Drawings

[0069] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention. In the accompanying drawings:

[0070] Figure 1 is a structural diagram of a real-time monitoring and modeling system for marine environment based on multi-modal fusion in an embodiment of the present invention;

[0071] Figure 2 is a structural diagram of a data acquisition module of a real-time monitoring and modeling system for marine environment based on multi-modal fusion in an embodiment of the present invention;

[0072] Figure 3The structural diagram of the multi-modal fusion module of a real-time marine environment monitoring and modeling system based on multi-modal fusion in an embodiment of the present invention;

[0073] Figure 4 The structural diagram of the real-time 3D modeling module of a real-time marine environment monitoring and modeling system based on multi-modal fusion in an embodiment of the present invention;

[0074] Figure 5 The flowchart of a real-time marine environment monitoring and modeling method based on multi-modal fusion in an embodiment of the present invention. Detailed implementation manners

[0075] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not used to limit the present invention.

[0076] Embodiment 1:

[0077] The present invention provides a real-time marine environment monitoring and modeling system based on multi-modal fusion, as Figure 1 shown, including:

[0078] A data acquisition module, configured to collect marine satellite remote sensing data, buoy sensor data, and public crowdsourcing data, and generate multi-modal data after preprocessing;

[0079] A multi-modal fusion module, configured to register multi-modal data in terms of time and space, and calculate the weights of various data sources in real time, and fuse the registered multi-modal data based on the weights to obtain fused marine environment data;

[0080] A real-time 3D modeling module, configured to construct a real-time 3D model of the marine environment by using 3D modeling technology based on the fused marine environment data, and dynamically update the real-time 3D model of the marine environment according to new multi-modal data.

[0081] The beneficial effects of the above technical solution are as follows: the present invention integrates satellite remote sensing, buoy sensors and public crowdsourcing data, collects information from multiple dimensions such as air, sky, land and sea, fills the blind spots of traditional monitoring methods, and the diversified data collection method enables the system to quickly capture the dynamic changes of the marine environment. Then, by aligning the multimodal data in time and space, it ensures that data from different sources can be compared and analyzed in the same time and space framework. The dynamic weight allocation algorithm can dynamically adjust the model according to the needs of marine research based on different monitoring indicators. For example, when monitoring the ocean surface temperature, a higher weight is given to satellite remote sensing data; when monitoring local marine biodiversity, the weight of public crowdsourcing data is increased, and the aligned data is fused based on the weight, giving full play to the advantages of each data source and effectively making up for the The limitations of a single data source are overcome, and the accuracy of marine environmental monitoring is significantly improved, making the monitoring results more in line with the actual marine environmental conditions. Finally, a real-time three-dimensional model is constructed based on the fusion of marine environmental data, converting abstract marine environmental data into an intuitive three-dimensional visualization scene. The model covers multiple levels such as the ocean surface, the interior of the water body, and the seabed topography, vividly displaying the three-dimensional spatial distribution characteristics of the marine environment, and can quickly grasp the overall picture of the marine environment, providing an intuitive and efficient information display method for marine scientific research and management decisions. It also receives new multimodal data in real time, recalculates weights and fuses data based on a dynamic weight allocation algorithm, and updates the three-dimensional model in a timely manner, so that the model can closely track real-time changes in the marine environment, providing a powerful tool for dynamic monitoring and scientific management of the marine environment.

[0082] Embodiment 2:

[0083] Based on Example 1, the data acquisition module, such as Figure 2 As shown, including:

[0084] A satellite remote sensing data acquisition unit, used for receiving ocean remote sensing image data sent by satellites, analyzing the ocean remote sensing image data, and generating ocean satellite remote sensing data;

[0085] The buoy sensor data acquisition unit is used to establish communication connections with buoy sensors distributed throughout the ocean, collect seawater data measured by various sensors on the buoy in real time, and generate buoy sensor data;

[0086] A public crowdsourcing data collection unit, used to collect ocean-related data on the Internet of Things based on big data technology to obtain public crowdsourcing data;

[0087] The data preprocessing unit is used to process the ocean satellite remote sensing data, buoy sensor data and public crowdsourcing data respectively based on corresponding preprocessing rules to generate multimodal data.

[0088] In this embodiment, the ocean satellite remote sensing data includes ocean surface temperature, ocean color, sea surface height, and information on sea surface biological activities.

[0089] In this embodiment, the seawater data includes temperature, salinity, dissolved oxygen, flow velocity, flow direction, etc.

[0090] In this embodiment, the ocean-related data includes photos of marine organisms taken, descriptions of seawater color changes, records of local ocean anomalies, etc.

[0091] Beneficial effects of the above technical solution: The present invention collects information from multiple dimensions of air, space, land, and sea, filling the blind spots of traditional monitoring means. The diversified data collection method can comprehensively describe the ocean environment from multiple dimensions, make up for the limitations of a single data source, and achieve more accurate and comprehensive monitoring and modeling of the ocean environment.

[0092] Embodiment 3:

[0093] Based on Embodiment 2, the data preprocessing unit includes:

[0094] The satellite remote sensing data preprocessing subunit is used to perform radiometric correction, atmospheric correction, and geometric correction on the ocean remote sensing images to obtain the preprocessed ocean remote sensing image data;

[0095] The buoy sensor data preprocessing subunit is used to perform denoising and filtering on the data collected by the buoy sensor;

[0096] The public crowdsourcing data preprocessing subunit is used to verify the authenticity and classify and sort the ocean-related data uploaded by the public on the Internet of Things to obtain the public crowdsourcing data

[0097] Beneficial effects of the above technical solution: The present invention performs radiometric correction, atmospheric correction, and geometric correction on the satellite remote sensing images to eliminate errors caused by factors such as sensor characteristics and atmospheric interference, and improve the accuracy and comparability of the image data. At the same time, it performs denoising and filtering on the data collected by the buoy sensor to remove abnormal data caused by sensor noise, electromagnetic interference, etc. For example, the Kalman filtering algorithm is used to denoise time series data such as seawater temperature and salinity. And it verifies the authenticity and classifies and sorts the ocean-related data uploaded by the public to ensure the accuracy and availability of the public crowdsourcing data.

[0098] Embodiment 4:

[0099] Based on Embodiment 3, the public crowdsourcing data preprocessing subunit includes:

[0100] The key information judgment subunit is used to obtain the release time, release location, and upload copy information of the ocean-related data uploaded by the public;

[0101] Perform semantic recognition and keyword extraction on the uploaded text information. Based on the semantic results and keyword extraction results, determine whether the uploaded text contains the marine geographical location information of the marine-related data uploaded for work;

[0102] The first data verification subunit is used to, if the uploaded text contains the marine geographical location information of the marine-related data uploaded for work, predict the actual observation time interval of the marine-related data based on the distance difference between the marine geographical location information and the published location, in combination with the publication time;

[0103] Based on the actual observation time interval and the marine geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics;

[0104] Correspondingly, combine the semantic recognition results and the text keywords to determine the main described phenomenon of the user;

[0105] According to the main described phenomenon of the user, extract the key features of the phenomenon from the publicly uploaded marine-related data. Based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded marine-related data is real data;

[0106] The second data verification subunit is used to, if the uploaded text does not contain the marine geographical location information of the marine-related data uploaded for work, extract the marine geological features from the publicly uploaded marine-related data, conduct relevant marine retrievals based on the marine geological features, and determine whether there is a suspected target ocean in the preset marine information database;

[0107] If not, determine that the publicly uploaded marine-related data is fake data;

[0108] If so, obtain the key features of the phenomenon and the corresponding phenomenon occurrence area features, and based on the local environmental data of the geographical location corresponding to the suspected target ocean, in combination with the marine biological activity characteristics and marine activity characteristics, speculate whether the phenomenon described by the publicly uploaded marine-related data may exist;

[0109] If so, obtain the geographical location information corresponding to the suspected ocean and send it to the first data verification subunit for determination;

[0110] Otherwise, determine that the publicly uploaded marine-related data is fake data.

[0111] Advantages of the above technical solution: The present invention conducts authenticity verification and classification and sorting on the publicly uploaded marine-related data, ensuring the accuracy and usability of the public crowdsourcing data.

[0112] Example 5:

[0113] Based on Embodiment 1, the multimodal fusion module, such as Figure 3 shown, includes:

[0114] A data registration unit, configured to register multimodal data in terms of time and space based on geographic information system technology, determine the corresponding relationship between time and position coordinates among different multimodal data, and obtain registered multimodal data;

[0115] A dynamic weight allocation unit, configured to establish a multi-factor decision-making model based on the analytic hierarchy process and the fuzzy comprehensive evaluation method, and obtain the actual needs of the user;

[0116] Analyze the actual needs to obtain the focus of the user's ocean monitoring, and based on the focus of the ocean monitoring, determine the user-preferred ocean factors. Based on the user-preferred ocean factors, use the multi-factor decision-making model to adjust the data weights of different multimodal data, and obtain the current weight allocation result of the multimodal data;

[0117] A multimodal data fusion unit, configured to fuse the registered multimodal data according to the current weight allocation result.

[0118] In this embodiment, for numerical data (such as seawater temperature, salinity, etc.), the weighted average method is used for fusion; for image data (such as satellite remote sensing images and ocean photos taken by the public), a feature-based fusion method is used to extract key features (such as color features, texture features, etc.) in the images for fusion processing to generate fused ocean environment data.

[0119] Beneficial effects of the above technical solution: According to the present invention, the data registration unit performs registration on multi-modal data in terms of time and space based on the geographic information system technology, clarifies the corresponding relationship between time and position coordinates among different multi-modal data, and generates registered multi-modal data, enabling data from various sources such as satellite remote sensing, buoy sensors, and public crowdsourcing to be analyzed within a unified spatio-temporal framework, eliminating data biases caused by spatio-temporal inconsistencies. For example, the large-area ocean surface temperature data obtained by a satellite at a specific time is accurately associated with the seawater temperature data measured by a buoy sensor at a local position at the same moment, providing a reliable basis for subsequent fusion and analysis. Subsequently, the dynamic weight allocation unit constructs a multi-factor decision-making model by using the analytic hierarchy process and the fuzzy comprehensive evaluation method, and by analyzing the actual needs of users, clarifies the focus of users' ocean monitoring, and then determines the ocean factors preferred by users, and can flexibly adjust the data proportion of different data sources according to the needs of different users. For example, scientific researchers may be more concerned about biodiversity in the marine ecosystem. Through model adjustment, the weight of the part of public crowdsourcing data regarding marine biological observations will increase; while the maritime department focuses more on ocean meteorology and ocean current data when monitoring maritime traffic safety, the model will correspondingly increase the weights of data such as sea surface height, wind speed and direction obtained by satellite remote sensing and the flow velocity and direction measured by buoy sensors, so that the monitoring results output by the system are more in line with the actual needs of users. Then, according to the ocean factors preferred by users, the data weights of different multi-modal data are adjusted by using the multi-factor decision-making model to obtain the current weight allocation result. For example, when monitoring the ocean surface temperature, the weight of satellite remote sensing data is set to 0.6, the weight of buoy sensor data is set to 0.3, and the weight of public crowdsourcing data is set to 0.1; when monitoring marine biodiversity, the weight of satellite remote sensing data is set to 0.2, the weight of buoy sensor data is set to 0.2, and the weight of public crowdsourcing data is set to 0.6. Finally, the multi-modal data fusion unit fuses the registered multi-modal data according to this result, ensuring that in the data fusion process, more important and user-demand-compliant data can be fully utilized, avoiding fusion result biases caused by unreasonable data weights, and enhancing the accuracy and reliability of ocean environment monitoring results.

[0120] Embodiment 6:

[0121] On the basis of Embodiment 1, the real-time three-dimensional modeling module, as Figure 4 shown, includes:

[0122] A three-dimensional model construction unit for constructing a real-time three-dimensional model of the ocean environment based on the fused ocean environment data by using three-dimensional modeling technology;

[0123] A model update and visualization unit for receiving new multi-source data in real time, recalculating weights according to the dynamic weight allocation algorithm, and performing data fusion.

[0124] Beneficial effects of the above technical solution: Based on the fusion of marine environmental data to construct a real-time three-dimensional model, the abstract marine environmental data is transformed into an intuitive three-dimensional visualization scene. The model covers multiple levels such as the ocean surface, the interior of the water body, and the seabed topography, vividly demonstrating the three-dimensional spatial distribution characteristics of the marine environment, enabling a quick grasp of the overall picture of the marine environment, providing an intuitive and efficient information display method for marine scientific research and management decision-making, and receiving new multimodal data in real time, recalculating weights and fusing data according to the dynamic weight allocation algorithm, and updating the three-dimensional model in a timely manner, so that the model can closely track the real-time changes of the marine environment, providing a powerful tool for the dynamic monitoring and scientific management of the marine environment.

[0125] Example 7:

[0126] Based on Example 6, the three-dimensional model construction unit includes:

[0127] A model layering subunit, configured to divide the marine three-dimensional model into multiple levels based on the marine standard model, and obtain the marine characterization data corresponding to each level, as well as the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each level;

[0128] A model construction subunit, configured to classify the fused multimodal data obtained based on the marine characterization data corresponding to each level, and obtain multimodal data sets respectively determined for each level;

[0129] Adjust the marine standard model based on the multimodal data set, determine the marine three-dimensional model of each level corresponding to the current ocean, and obtain a primary marine model;

[0130] At the same time, based on the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each level, and the marine three-dimensional model of each level corresponding to the current ocean, determine the decomposition characteristics of each level of the current ocean;

[0131] Fuse each level corresponding to the primary marine model based on the decomposition characteristics of each level of the current ocean to obtain a real-time three-dimensional model of the marine environment corresponding to the current situation.

[0132] Beneficial effects of the above technical solution: The present invention is based on the marine standard model to establish each level of the current ocean respectively and then fuse multiple levels, effectively accelerating the establishment speed of the real-time three-dimensional model of the marine environment and realizing the rapid establishment of the real-time three-dimensional model of the marine environment.

[0133] Example 8:

[0134] The present invention provides a method for real-time monitoring and modeling of a marine environment based on multimodal fusion, as Figure 5 shown, including:

[0135] Step 1: Collect ocean satellite remote sensing data, buoy sensor data, and public crowdsourcing data. After preprocessing, generate multi-modal data;

[0136] Step 2: Register the multi-modal data in terms of time and space, and calculate the weights of various data sources in real time. Based on the weights, fuse the registered multi-modal data to obtain fused ocean environment data;

[0137] Step 3: Based on the fused ocean environment data, use 3D modeling technology to construct a real-time 3D model of the ocean environment, and dynamically update the real-time 3D model of the ocean environment according to the new multi-modal data.

[0138] Beneficial effects of the above technical solution: The present invention integrates satellite remote sensing, buoy sensors, and public crowdsourcing data, collects information from multiple dimensions of air, space, land, and sea, fills the blind spots of traditional monitoring means. The diversified data collection method enables the system to quickly capture the dynamic changes of the ocean environment. Then, by registering the multi-modal data in time and space, it ensures that data from different sources can be compared and analyzed in the same spatio-temporal framework. The dynamic weight allocation algorithm can dynamically adjust the model according to different monitoring indicators and ocean research needs. For example, when monitoring the ocean surface temperature, a higher weight is given to satellite remote sensing data; when monitoring the local marine biodiversity, the weight of public crowdsourcing data is increased. Based on the weights, the registered data is fused, giving full play to the advantages of each data source, effectively making up for the limitations of a single data source, significantly improving the accuracy of ocean environment monitoring, making the monitoring results more in line with the actual ocean environment conditions. Finally, based on the fused ocean environment data, a real-time 3D model is constructed, transforming the abstract ocean environment data into an intuitive 3D visualization scene. The model covers multiple levels such as the ocean surface, the water body interior, and the seabed topography, vividly showing the 3D spatial distribution characteristics of the ocean environment, enabling a quick grasp of the overall picture of the ocean environment, providing an intuitive and efficient information display method for ocean scientific research and management decision-making. And it can receive new multi-modal data in real time, recalculate the weights and fuse the data according to the dynamic weight allocation algorithm, and update the 3D model in time, enabling the model to closely track the real-time changes of the ocean environment, providing a powerful tool for the dynamic monitoring and scientific management of the ocean environment.

[0139] Example 9:

[0140] Based on Example 8, a real-time monitoring and modeling method for ocean environment based on multi-modal fusion includes verifying the authenticity of public crowdsourcing data, including:

[0141] Step A: Obtain the release time, release location, and uploaded text information of ocean-related data uploaded publicly;

[0142] Perform semantic recognition and keyword extraction on the uploaded text information. Based on the semantic results and keyword extraction results, determine whether the uploaded text contains the marine geographical location information of the marine-related data uploaded for work;

[0143] Step B: If the uploaded text contains the marine geographical location information of the marine-related data uploaded for work, then based on the distance difference between the marine geographical location information and the published location, combined with the publication time, predict the actual observation time interval of the marine-related data;

[0144] Based on the actual observation time interval and the marine geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics;

[0145] Correspondingly, combine the semantic recognition results and the text keywords to determine the main phenomenon described by the user;

[0146] According to the main phenomenon described by the user, extract the key features of the phenomenon from the publicly uploaded marine-related data. Based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded marine-related data is real data;

[0147] Step C: If the uploaded text does not contain the marine geographical location information of the marine-related data uploaded for work, then extract the marine geological features from the publicly uploaded marine-related data. Based on the marine geological features, conduct relevant marine retrievals to determine whether there is a suspected target ocean in the preset marine information database;

[0148] If not, determine that the publicly uploaded marine-related data is fake data;

[0149] If there is, then obtain the key features of the phenomenon and the corresponding regional features of the phenomenon occurrence. Based on the local environmental data of the geographical location corresponding to the suspected target ocean, combined with the marine biological activity characteristics and marine activity characteristics, speculate whether the phenomenon described by the publicly uploaded marine-related data may exist;

[0150] If so, obtain the geographical location information corresponding to the suspected ocean and send it to Step B for secondary determination;

[0151] Otherwise, determine that the publicly uploaded marine-related data is fake data.

[0152] Beneficial effects of the above technical solution: By obtaining the release time, location, and uploaded text information of the data uploaded by the public, the present invention performs semantic recognition and keyword extraction on the text, determines whether it contains marine geographical location information, quickly filters out the data containing key geographical information, and clarifies the direction for subsequent more in-depth verification work. For example, among the data uploaded by the public, accurately identify those records that clearly mention the observation location, concentrate the limited verification resources on these potentially valid data, improve the data screening efficiency, and avoid unnecessary subsequent processing of a large amount of irrelevant or invalid data; and when the uploaded text contains marine geographical location information, based on the distance difference between the location information and the release location and the release time, predict the actual observation time interval, which helps to combine the data uploaded by the public with information such as satellite remote sensing images at the corresponding time and location. For example, by reasonably inferring the observation time, multiple target remote sensing images of the corresponding area and their sea surface change characteristics during that time period can be accurately obtained. At the same time, by combining text analysis to determine the main phenomenon described by the user, and extracting the key characteristics of the phenomenon for comparison with the sea surface change characteristics, through the method of multi-information fusion and comparison, the authenticity of the data can be verified from multiple dimensions, greatly improving the accuracy of data verification and reducing misjudgments caused by data mismatch or false descriptions; when the uploaded text does not contain marine geographical location information, extract the marine geological characteristics of the data, and retrieve the suspected target ocean in the preset marine information database. If there is a suspected target ocean, combine the local environmental data, marine organisms, and activity characteristics to speculate whether the described phenomenon may exist, which can uncover data that may be related to real marine phenomena through other characteristics even though the geographical location is not clearly stated. For example, some data describing the behavior of unique marine organisms or abnormal marine phenomena, even if not marked with a location, may discover potential real observation records through geological feature matching and phenomenon rationality speculation. If a suspected target ocean exists, send it to step B for secondary determination to further ensure the authenticity of the data, avoid missing valuable data, and the authenticity determination mechanism for true and false data can effectively eliminate false, incorrect, or misleading data uploaded by the public, ensure that the public crowdsourcing data entering the marine environmental monitoring system has high quality, ensure the accuracy and availability of the public crowdsourcing data, and exclude pseudo data to avoid its interference with subsequent data analysis, model construction, and decision-making, and improve the reliability and scientific nature of the entire marine environmental monitoring system.

[0153] Example 10:

[0154] Based on the embodiment 8, step 3 includes:

[0155] Divide the ocean three-dimensional model into multiple layers based on the ocean standard model, and obtain the corresponding ocean characterization data for each layer, as well as the change characteristics of marine biological species and the sea water decomposition data characteristics at the boundaries of each layer;

[0156] Classify the obtained fused multi-modal data based on the ocean characterization data corresponding to each layer, and obtain the multi-modal data sets corresponding to each layer respectively;

[0157] Adjust the ocean standard model based on the multi-modal data sets, determine the three-dimensional ocean models of each layer corresponding to the current ocean, and obtain the primary ocean model;

[0158] At the same time, based on the characteristics of the changes in the types of marine organisms and the characteristics of seawater decomposition data at the boundaries of each layer, determine the decomposition characteristics of each layer of the current ocean for the three-dimensional ocean models of each layer corresponding to the current ocean;

[0159] Fuse each layer corresponding to the primary ocean model based on the decomposition characteristics of each layer of the current ocean to obtain the real-time three-dimensional model of the current corresponding ocean environment.

[0160] Advantages of the above technical solution: Based on the ocean standard model, this invention separately establishes each layer of the current ocean and then fuses multiple layers, effectively accelerating the establishment speed of the real-time three-dimensional model of the ocean environment and realizing the rapid establishment of the real-time three-dimensional model of the ocean environment.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A real-time monitoring and modeling system for marine environment based on multimodal fusion, characterized in that, Including: A data acquisition module, which is used to collect ocean satellite remote sensing data, buoy sensor data and public crowdsourcing data, and after preprocessing, generate multimodal data; A multimodal fusion module, which is used to register multimodal data in terms of time and space, and calculate the weights of various data sources in real time, and fuse the registered multimodal data based on the weights to obtain fused ocean environment data; A real-time three-dimensional modeling module, which is used to construct a real-time three-dimensional model of the ocean environment based on the fused ocean environment data by using three-dimensional modeling technology, and dynamically update the real-time three-dimensional model of the ocean environment according to new multimodal data; The system further includes: a public crowdsourcing data preprocessing subunit, including: A key information judgment subunit, which is used to obtain the release time, release location and upload copy information of the ocean-related data uploaded publicly; Perform semantic recognition and keyword extraction on the uploaded copy information, and judge whether the uploaded copy contains the ocean geographical location information of the ocean-related data uploaded for work according to the semantic result and keyword extraction result; A first data verification subunit, which is used to, if the uploaded copy contains the ocean geographical location information of the ocean-related data uploaded for work, predict the actual observation time interval of the ocean-related data based on the distance difference between the ocean geographical location information and the released location, in combination with the release time; Obtain corresponding multiple target remote sensing images and their sea surface change characteristics based on the actual observation time interval and the ocean geographical location information; Correspondingly, combine the semantic recognition result and the copy keywords to determine the main phenomenon described by the user; Extract the key features of the phenomenon from the publicly uploaded ocean-related data according to the main phenomenon described by the user, and based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded ocean-related data is real data; A second data verification subunit, which is used to, if the uploaded copy does not contain the ocean geographical location information of the ocean-related data uploaded for work, extract the ocean geological features from the publicly uploaded ocean-related data, and perform relevant ocean retrieval based on the ocean geological features to determine whether there is a suspected target ocean in the preset ocean information database; If not, determine that the publicly uploaded ocean-related data is false data; If so, obtain the key features of the phenomenon and the corresponding phenomenon occurrence area features, and based on the local environment data of the geographical location corresponding to the suspected target ocean, in combination with the marine biological activity characteristics and marine activity characteristics, infer whether the phenomenon described by the publicly uploaded ocean-related data may exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to the first data verification subunit for secondary determination; Otherwise, determine that the publicly uploaded ocean-related data is false data.

2. The real-time monitoring and modeling system for marine environment based on multimodal fusion according to claim 1, characterized in that, The data acquisition module includes: A satellite remote sensing data acquisition unit, which is used to receive the ocean remote sensing image data sent by the satellite and send it to the data preprocessing unit, and parse the preprocessed ocean remote sensing image data to generate ocean satellite remote sensing data; A buoy sensor data acquisition unit, which is used to establish a communication connection with buoy sensors distributed everywhere in the ocean, collect in real time the seawater data measured by various sensors on the buoy, and generate buoy sensor data; A public crowdsourcing data acquisition unit, which is used to collect ocean-related data on the Internet of Things based on big data technology to obtain public crowdsourcing data; A data preprocessing unit, which is used to process ocean remote sensing image data, buoy sensor data, and public crowdsourcing data respectively based on corresponding preprocessing rules to generate multimodal data.

3. The real-time monitoring and modeling system for marine environment based on multimodal fusion according to claim 2, characterized in that, The data preprocessing unit includes: A satellite remote sensing data preprocessing subunit, which is used to perform radiometric correction, atmospheric correction, and geometric correction on ocean remote sensing images to obtain preprocessed ocean remote sensing image data; A buoy sensor data preprocessing subunit, which is used to denoise and filter the seawater data collected by buoy sensors; A public crowdsourcing data preprocessing subunit, which is used to verify the authenticity and classify and sort the ocean-related data uploaded by the public on the Internet of Things to obtain public crowdsourcing data.

4. The real-time monitoring and modeling system for marine environment based on multi-modal fusion according to claim 1, characterized in that, The multimodal fusion module includes: A data registration unit, which is used to register multimodal data in terms of time and space based on geographic information system technology, determine the corresponding relationship between time and position coordinates among different multimodal data, and obtain registered multimodal data; A dynamic weight allocation unit, which is used to establish a multi-factor decision-making model based on the analytic hierarchy process and the fuzzy comprehensive evaluation method to obtain the actual needs of users; Analyze the actual needs to obtain the user's focus on ocean monitoring, and based on the user's focus on ocean monitoring, determine the user's preferred ocean factors. Based on the user's preferred ocean factors, use the multi-factor decision-making model to adjust the data weights of different multimodal data to obtain the current weight allocation result of multimodal data; A multimodal data fusion unit, which is used to fuse the registered multimodal data according to the current weight allocation result.

5. The real-time monitoring and modeling system for marine environment based on multi-modal fusion according to claim 1, wherein, The real-time three-dimensional modeling module includes: A three-dimensional model construction unit, which is used to construct a real-time three-dimensional model of the ocean environment based on the fused multimodal data using three-dimensional modeling technology; A model update and visualization unit, which is used to receive new multi-source data in real time, recalculate weights according to the dynamic weight allocation algorithm, and perform data fusion.

6. The real-time monitoring and modeling system for marine environment based on multi-modal fusion according to claim 5, wherein, The three-dimensional model construction unit includes: A model layering subunit, which is used to divide the ocean three-dimensional model into multiple layers based on the ocean standard model, and obtain the corresponding ocean characterization data for each layer, as well as the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer; A model construction subunit, which is used to classify the fused multimodal data obtained based on the corresponding ocean characterization data for each layer to obtain the multimodal data sets corresponding to each layer respectively; Adjust the ocean standard model based on the multimodal data sets to determine the ocean three-dimensional model for each layer corresponding to the current ocean, and obtain a primary ocean model; At the same time, based on the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer, and the ocean three-dimensional model for each layer corresponding to the current ocean, determine the decomposition characteristics of each layer of the current ocean; Fuse each layer corresponding to the primary ocean model based on the decomposition characteristics of each layer of the current ocean to obtain a real-time three-dimensional model of the current corresponding ocean environment.

7. A real-time monitoring and modeling method for marine environment based on multimodal fusion, characterized in that, Including: Step 1: Collect ocean satellite remote sensing data, buoy sensor data, and public crowdsourcing data. After preprocessing, generate multimodal data; Step 2: Register the multimodal data in terms of time and space, and calculate the weights of various data sources in real time. Based on the weights, fuse the registered multimodal data to obtain fused ocean environment data; Step 3: Based on the fused ocean environment data, use three-dimensional modeling technology to construct a real-time three-dimensional model of the ocean environment, and dynamically update the real-time three-dimensional model of the ocean environment according to the new multimodal data; The method further includes: Verify the authenticity of the public crowdsourcing data, including: Step A: Obtain the release time, release location, and upload copy information of the ocean-related data uploaded publicly; Perform semantic recognition and keyword extraction on the uploaded copy information. According to the semantic result and keyword extraction result, determine whether the uploaded copy contains the ocean geographical location information of the ocean-related data uploaded for work; Step B: If the uploaded copy contains the ocean geographical location information of the ocean-related data uploaded for work, then based on the distance difference between the ocean geographical location information and the released location, combined with the release time, predict the actual observation time interval of the ocean-related data; Based on the actual observation time interval and the ocean geographical location information, obtain the corresponding multiple target remote sensing images and their sea surface change characteristics; Correspondingly, combine the semantic recognition result and the copy keywords to determine the main phenomenon described by the user; According to the main phenomenon described by the user, extract the key features of the phenomenon from the publicly uploaded ocean-related data. Based on comparing the key features of the phenomenon with multiple sea surface change characteristics respectively, when the similarity between any sea surface change characteristic and the key feature of the phenomenon is greater than or equal to the preset value, determine that the publicly uploaded ocean-related data is real data; Step C: If the uploaded copy does not contain the ocean geographical location information of the ocean-related data uploaded for work, then extract the ocean geological features from the publicly uploaded ocean-related data, conduct relevant ocean retrievals based on the ocean geological features, and determine whether there is a suspected target ocean in the preset ocean information database; If not, determine that the publicly uploaded ocean-related data is fake data; If so, obtain the key features of the phenomenon and its corresponding phenomenon occurrence area features. Based on the local environment data corresponding to the geographical location of the suspected target ocean, combined with the characteristics of marine biological activities and ocean activities, speculate whether the phenomenon described by the publicly uploaded ocean-related data may exist; If so, obtain the geographical location information corresponding to the suspected ocean and send it to Step B for secondary determination; Otherwise, determine that the publicly uploaded ocean-related data is fake data.

8. A real-time monitoring and modeling method for marine environment based on multi-modal fusion according to claim 7, characterized in that, Step 3 includes: Based on the ocean standard model, divide the ocean three-dimensional model into multiple layers, and obtain the ocean characterization data corresponding to each layer, as well as the change characteristics of marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer; Based on the ocean representation data corresponding to each layer, classify the obtained fused multi-modal data to obtain multi-modal data sets corresponding to each layer respectively; Based on the multi-modal data sets, adjust the ocean standard model to determine the ocean three-dimensional models of each layer corresponding to the current ocean, and obtain the primary ocean model; At the same time, based on the characteristics of the changes in marine biological species and the characteristics of seawater decomposition data at the boundaries of each layer, and the ocean three-dimensional models of each layer corresponding to the current ocean, determine the decomposition characteristics of each layer of the current ocean; Based on the decomposition characteristics of each layer of the current ocean, fuse each layer corresponding to the primary ocean model to obtain the real-time three-dimensional model of the current corresponding ocean environment.

Images